# Presentation: 213582

## Requirements and Services: The Application Point of View

- Jennifer M. Schopf
- Argonne National Lab
- (Steven Newhouse, OMII)

## The Never EndingRoad Trip

- Jennifer M. Schopf
- Argonne National Lab
- (Steven Newhouse, OMII)

## What If...

- We had a Grid based around scientists, not scientists using what they had to from the Grid
- Tools fit into a users normal environment with little or no adaptation in work style
- There were simple, reliable, persistent services that were easy to use
- So why aren’t we here already?

## The Need:

- Several projects developing or delivering middleware without clear guidance
- Users and uses have grown and changed
- Middleware has grown and changed
- (warning needs to be spread of Steven’s move and that fact that I’m spending a year in the UK ....)
- Vital need for information exchange at this juncture

## And Our Guest Today Is...

- Many current projects, we’ve focused on those in the UK due to OMII mission (and lack of US understanding into these)
- Current application developers with some Grid or Web Services experience
- Those with software that might be of broader use or interest
- Those who have expressed dissatisfaction with current tools

## So Far

- Oxford Security Workshop
- Networking for Non-Networkers Workshop (NNFN)
- T. Cooper-Chadwich, Southampton, gYacht
- S. Cox, Southampton, GeoDise
- M. Daw, Manchester, AG
- M. Giles, Oxford, gViz
- C. Goble, Manchester, MyGrid & Integrative Biology Project
- S. Lloyd, Oxford, eDiamond
- J. MacLaren, Manchester, UoM Broker
- A. Martin, Oxford, ClimatePreciction.NET
- M. McKeown, Manchester, OGSI:lite and WSRF:lite
- S. Pickles, Manchester, TeraGyroid & GRENADE
- A. Porter, Manchester, Reality Grid
- A. Rector, Manchester, CLEF
- M. Rider, Manchester, eViz

## Still To Come:

- Edinburgh
- Glasgow
- University College London
- Imperial College
- Southampton (again)
- ... and others by request

## The Results So Far

- Need for Training
- Security
- Functionality
- Jobs
- Data
- What isn’t mentioned
- What tools should look like
- Infrastructure/Operations

## Training

- Grid vision still needs to be sold –
- “What do these tool give me over SSH, scp?”
- “What if I don’t want to stop using my magnifying glass to read x-rays?”
- Still need basic common practices to be written: for user, developers *and* admins
- Web service basics
- Firewalls
- Builds and packaging
- No surprise: Communication is still a large unsolved problem in Grid computing

## Security: What’s Old News

- If it isn’t easy users aren’t interested
- All users hate firewalls, all system administrators love them
- Anonymizing data is hard
- Still need a lot of information sharing:
- How fire walls interact with Grid/Web services
- Security audits

## Security: What’s Surprising Us

- User focus on need for _data integrity_ not authentication/authorization
- Time and again this was mentioned
- Delegation seems to be the next big question
- GT2-style delegation needed in a services world
- No one has an agreed upon solution yet

## Questions?

- Need for Training
- Security
- Functionality
- Jobs
- Data
- What isn’t mentioned
- What tools should look like
- Infrastructure/Operations

## Functionality

- Prediction of Pete Beckman (TeraGrid):
- All users want is SSH, scp, and top

## Functionality

- All users talk about is job submission and file transfer capabilities
- When asked about trouble spots they also want tools to tell how tools are progressing
- Occasionally would like an application specific service, like a viz tool
- Many other tools are on the horizon, but they simply don’t seem to be on the 6 month horizon for the users we’ve spoken with to date

## Job Submission: No Surprises

- Want simple, dependable “run my application” interfaces
- This was identified at GF1!
- Only resource discovery is “small”
- How many nodes have a matlab license?
- NOT: which cluster should I use?

## Job Submission: Urgent Needs

- Tools to understand errors while a job is running- something stopped, where and why?
- TeraGyroids’ use of SSH for debugging
- Need for Global Job Unique ID
- What to do when a job fails-
- Resubmit or ignore?
- Workflow issues
- Steering

## File Transfer

- People seem pretty happy with GridFTP
- Some reliability (RFT) would fill out rest of use
- This needs 3rd party transfer (delegation)
- Some projects starting to work with provenance issues or access to databases
- Still issues with performance and making sure background infrastructure is all as it should be – more later

## What (These)Users Aren’t Talking About...

- Notification – except for job progress tracking
- Registries or resource discovery
- QoS, reservations, brokering, advanced scheduling techniques
- Job migration, checkpointing
- Accounting and pricing (but we’re talking with users, not admins so far)
- Data replication, migration
- Instruments

## ...And Why We Think This Is

- A gap still exists between the computer science research and tool building community and the average user
- Large difference between short term needs and long term planning-
- Most users are still trying for basic functionality and dealing with today’s hurdles
- Most researchers are looking at the greener pastures a few years out

## Questions So Far?

- Need for Training
- Security
- Functionality
- Jobs
- Data
- What isn’t mentioned
- What tools should look like
- Infrastructure/Operations

## User’s View of Tools

- Users have strong opinions on tools – what a surprise!
- Mostly known problems, but the prioritization of certain aspects wasn’t known

## Tools Should Be

- Vertical solutions
- End to end use cases, not horizontal pieces that don’t work together
- Simple
- One job, one tool – think unix!
- Work easily for the 80% case, and rest is possible if needed
- Ease of use/installs
- Bundle all together so you have entire use together
- Don’t reinstall things I already have

## Sometimes There Is No “Best” Tool

- Same workspace may have many solutions, each with slightly different attributes, all valuable
- BPEL vs SCUFL
- Cluster monitors
- Users should not be forced to use a tool from another environment if there is no underlying scientific need for it
- However, we still need to try to avoid replication of tools

## Composable Functionality

- Lego blocks of basic functions
- “Shims” to fit between where needed
- API mismatches
- Data translation
- Interfaces to legacy code
- SOAP Lab (wraps command line to look like SOAP)

## Interfaces

- Need simple APIs at the user level
- eg. SAGA-RG (but then we knew this)
- User API might sit a layer above standard tool APIs to mitigate upgrade effects
- eg. HiCog
- The API a user sees and the API the infrastructure not only can but should be different – different goals

## Environments

- Tools need to fit in with existing “user comfort zone”
- Biologists like Perl
- CFD folks like MatLab
- HEP (EGEE) are used to Python
- This is a sys admin’s and tool developer’s nightmare – but for usability it’s a must

## Questions?

- Need for Training
- Security
- Functionality
- Jobs
- Data
- What isn’t mentioned
- What tools should look like
- Infrastructure/Operations

## Builds and Upgrades

- Need for a reproducible build
- Hands off process, works every time
- Verification tools
- Need better understanding of effects of upgrades, or else users don’t want it
- What will change, what will be affected
- Tools are being used “off label”
- Tool for usecase A in common use for usecase B
- Scalability becomes an issue
- New/different functions needed

## Understanding System Stability

- Need for basic tools to verify functionality and performance
- “I can’t transfer my files today”
- What’s broken?
- What changed?
- How do I fix it?
- Why couldn’t someone find this before me?
- Strong needs for quality assurance tests on all platforms – clusters, networks, AG, etc.

## System Tests

- Often system tests don’t look like current applications
- Tests for firewall functionality don’t include checks for all ports in current use
- System benchmarks don’t look like “my” application
- Ping tests aren’t enough to assure that a GridFTP transfer will work
- Need for better testing, verification- for the user, and even by the user!

## WebMD for Applications

- Basic diagnostics needed for users
- Q - “I’m trying to transfer a 1 gig file between A and B and can’t”
- A- “Is your cert ok? Here’s how to check”	Y/N, if Y...
- A – “Is the route between you’re hosts up? Here’s how to run traceroute for your system...”

## Topics of Conversation

- Need for Training
- Security
- Functionality
- Jobs
- Data
- What isn’t mentioned
- What tools should look like
- Infrastructure/Operations

## Our Conclusions... So Far

- Delegation
- Job tracking
- Easing tool use
- Wrappers
- Composability
- Verification and instability analysis
- Diagnosis

## The Point

- We can’t say it any more simply
- Grid tool developers must continue to talk and interact with application scientists – without them, we are nothing

## How To Find Us:

- Jennifer M. Schopf
- jms@mcs.anl.gov
- Will be in Edinburgh
- Steven Newhouse
- s.newhouse@omii.ac.uk