# Presentation: 158487

## Grid Futures- Where Are We Going Next?A Pragmatic Viewpoint

- Jennifer M. Schopf
- Argonne National Laboratory
- UK National eScience Centre

## Grids

- You’ve all just spent the last 4 days learning about hands on approaches to Grids
- By now you should know a little about several different ways to use Grid technology
- But....

## Things heard about Grids...

- “Isn’t the Grid just a funding construct?” (SC ’01)
- "Grid computing has been more hype than reality,“ - Hewlett-Packard CEO Carly Fiorina, 10/03
- “[Customers] don't need the Globus Toolkit to do high-performance compute clusters" - Charles Fitzgerald, a Microsoft general manager, Information Week 1/05
- “ ’Grid’ is simply a marketing term” –EU policy meeting 2/06

- Obviously, much more is needed in the Grid space to address these concerns
- Anyone who says “Grids are done” hasn’t spoken to users recently
- There’s a lot more work to do, but we do have some nice examples of where Grids are helping research and science

## What *I* think will happen next

- Technology
- Social-political
- Application-oriented interactions
- For each topic, I’ll walk through an example, talk about current vs future, and give my estimate of when it might happen

## Technology

- Virtualization
- Ease of Use
- Windows for Grids
- Security

## Virtualization

- Vision of the Grid:
- Plug in and get the services you need
- Just like electricity
- Doesn’t matter what resource is supplying it, or where it is, just use the “juice”
- Concrete example, a use might ask..
- “Run my job, finish by lunch”
- “Get a data set that has these attributes”
- “Tell me when that simulation will finish”

## Where are we today?

- “Run my job, finish by lunch” becomes
- Run my job on this exact machine
- With these data files transferred in using this protocol
- I think it will take 2 hours, the queues have been slow lately, so I should make sure I send this off by 9am, or earlier if I want to be safe in having results for 2

## Where are we today?

- “Get a data set that has these attributes” becomes
- Given a set of attributes, give me a set of logical file names
- Given those, map them to physical file names
- Given physical placements of the file, figure out which one is easiest to access
- Copy the file to my machine

## Where are we today (cont)

- General agreement we have basic functionality
- Tell me what this set of resources look like
- Run this job on that resource
- Transfer this file
- Globus (among others) does give these basic building blocks (mostly)
- General agreement general functionality isn’t enough by far

## Virtualization is happening.. slowly

- General concept of service-oriented grids is being accepted
- Service Level Agreements (SLAs) are coming into place- this is a first step
- Higher-level tools to separate user from basic resources are more common
- When will this be resolved?
- Not this year...

## Ease of Use

- Users will only come when they have decent tools
- Simple enough for “easy” use
- Robust enough for “stupid” use
- Still allow work-arounds for “hard-core” use
- Users are hampered by software that doesn’t do what they need it to
- Globus is NOT an end-to-end solution

## What is needed

- Closer ties between tool builders and user
- Tool builders still creating “cool” solutions to problems that don’t exist
- Users still not communicating what they need – or ignoring “not built here” solutions when available
- When will this be resolved?
- Current focus of several funding calls now
- First steps – but a long way to travel

## Windows

- Much of the available grid software doesn’t run on windows
- Do you really NEED it to?
- Maybe, maybe not
- Often for political reasons, not technical
- When will this be resolved?
- When it’s needed technically

## Grid Security

- Without security we can’t have a Grid
- EVERYTHING needs to be secure-
- Who can run on a machine
- File transfers
- What data does someone have access to	 (program data, system data)
- Who can access which services?

## Security vs. Usability

- Users want security but don’t want to deal with it
- If security is hard- it won’t be used
- Most security (including Grid Security Infrastructure (GSI)) is based on public key infrastructure (PKI)
- Users have files (public and private keys) that must be secure, use reasonable passwords, etc.

## What about...

- Multiple certificates?
- Group access?
- Dynamic policy changes?
- Scalability?
- Overheads
- Etc., etc., etc...

## Security is an open question

- Until security is made easier to use, it won’t be used
- Until security is made easier to manage at the group level, it won’t be used
- Without security no one will really use the Grid

## Current work – there’s a lot of it!

- Grid-Shibboleth interactions
- Tying PKI into Shib, which is used on many higher education campuses
- VOMS
- Community-level group membership system
- Allows the VO to centrally manage user roles
- Single-sign on
- Currently available, but always ways to make things easier
- When will this be resolved?
- This is an area that will ALWAYS have active research and development

## What *I* think will happen next

- Technology
- Social-political
- Communication
- Interoperability between grids
- Standards groups
- Application-oriented interactions

## Socio-political Issues

- Hardest problems are often not technical ones
- Multiple administration domains means multiple policies
- Multiple countries means multiple communication styles
- Decisions are often made on non-technical basis

## Communication is hard

- Too many people in the mix
- Not everyone is informed of status updates
- Often hallway conversation becomes what people believe
- Too often assumptions are not verified
- Many communication styles can lead to misunderstandings

## What to do?

- Ongoing efforts to continue better communication are needed to build a global community
- Constructive criticism, reporting of errors, etc – just saying “Globus Sucks” simply isn’t helpful 
- When will this be resolved?
- Better than it used to be – although the rumor mill is still there

## Need for Standards

- Need for standard APIs and protocols to allow easier
- Access to data sources
- Registration of data
- Archiving tools
- Standards for what information is available
- Standards for what that information means
- Standards for communication of errors
- This is in part what inspired Globus’s move to Web services!

## Standards

- What’s the real goal behind standards?
- Interoperabilty!
- Without standard interfaces, languages, schemas, etc we cannot have multiple implementations that work together
- However, standards are hard

## Why Are Standards Hard?

- Agreement between many partners
- This is often socio-political, not technical
- Standardizing too soon versus too late
- Need to be very exact in order to only have one interpretation of a standard
- Need to make sure you take performance into account
- Need to have take-up by the major players
- When will this be resolved?
- This will continue to be an ongoing issue

## What Does It Take ForTwo Grids To Interoperate?

- What do you mean by interoperate?
- Share data?
- Share job submissions?
- Share accounting data?
- Share accounts?
- Have a common broker?
- Have a common software stack?
- Have common environments or testing?

## One Approach: GIN

- Grid Interop Now
- Go beyond pair-wise interactions
- Looking at 4 basic ways grids can interact
- Common CA acceptance
- File transfers (using GridFTP)
- Job submission language (using JSDL)
- Publish basic resource data (defined by attribute, not schema)
- 18+ Grids involved
- Demo for SC

## Will GIN succeed?

- There will be a demo at SC
- But will there be users?
- When will this be resolved?
- Pair-wise interoperability is happening now, and expanding
- *Real* interoperability will happen when users demand it

- Technology
- Social-Political
- Application-Oriented Interactions
- Grids Become Data Grids
- Interfaces
- User communities

## At First, There Were Cycles...

- Originally, application users (scientists) wanted to run big jobs
- These were the obvious candidates coming from HPC
- These were the people willing to put up with poor interfaces, buggy software, no support
- These were some of the people writing the software
- Obvious use case when Grid were first being looked at

## ...and then came the data

- Replicating very large data sets
- Accessing distributed databases
- Understanding data provenance
- Finding files (or parts of files) with certain attributes

## How is Data Different From Compute?

- More things can go wrong
- In a compute only world you have your executable, and it runs on a given resource
- Data is everywhere, many links between you and it
- Many additional dependencies

## Where does the data come from?

- UK funding councils are beginning to demand that projects make their data publicly available
- Annotated, curated, freely available
- Where are the tools to help with this?
- How do you maintain this data once your project ends?

## When will the dataproblem be resolved?

- Some applications are already doing this quite happily
- Additional work will continue for next several years to
- Increase usability
- Tie together additional services
- Curation issue
- This is where much of the growth of Grids is likely to be

## Interfaces

- Different groups of users want different interfaces
- Biologists seem to prefer Perl
- Physicists seem to prefer Python
- Engineers like matlab
- Some communities want Java or command line
- Some want portals

## The Power of Portals

- The power of portals
- Low learning curve
- Can be made domain specific
- Can hide “all the certificate stuff” from user
- Toolkits for Grid Portals
- PURSE, OGCE, GAMA, GridSphere, GridSite, etc.
- But, we must also understand the limitations of portals
- An 80/20 solution
- Power-users easily get annoyed
- Difficult for “tinkering-centric” research
- (slide compliments Von Welch)

## Spectrum of Use

- _Command-line_
- Can run any command, compile any executable
- Available on most resources
- Lots of details to keep track of
- 20%

- _Portals_
- Set of pre-defined functionality
- Must be built for specific applications
- Simple interactions
- 80%

## Where Are We With Portals?

- Many application groups are building what they need
- Work is in tuning it for a community
- There will always be functionality that you can only get through command line
- When will this be resolved?
- Some applications are already doing this quite happily
- Additional work will continue

## User Communities

- Traditionally, we started with the physicists
- Hard core users (heroic users)
- Large computational problems
- Already had strong national and international collaborations
- This is growing and changing as understanding of how the resources can be used to further science and research are better understood

## The eScience Centres

**CeSC (Cambridge**)

**Notes:**
Current UK e-Infrastructure.
This shows the National Centre, 8 regional centres and 2 laboratories in blue
	That is the original set up at the start of UK e-Science August 2001
	NeSC is jointly run by Edinburgh & Glasgow Universities
In 2003 several smaller centres were added (vermilion)
(1st call out)  The e-Science Institute is run by the National e-Science Centre.  It runs a programme of events and hosts visiting international researchers.  It was established in 2001.
The Open Middleware Infrastructure Institute was established in 2004, to provide support and direction for Grid middleware developed in the UK.  It is based at the University of Southampton.
The Grid Operations Centre will be established in 2004.
(2nd call out) The Digital Curation Centre was established in 2004 by the Universities of Edinburgh and Glasgow, the UK Online Library Network at the University of Bath, and the Central Laboratories at Daresbury and Rutherford.  It’s job is to provide advice on curating scientific data and on preserving digital media, formats, and access software.
(3rd call out)  Edinburgh is one of the 4 founders of the Globus Alliance (Sept 2003) which will take responsibility for the future of the Globus Toolkit.  The other founders are:  Chicago University (Argonne National Lab), University of Southern California, Los Angeles (Information Sciences Institute) and the PDC, Stockholm, Sweden
(4th call out)  The EU EGEE project (Enabling Grids for E-Science in Europe) is establishing a common framework for Grids in Europe.  The UK e-Science programme has several connections with EGEE.  NeSC leads the training component for the whole of Europe.

## The High Energy Physicists

**CeSC (Cambridge**)

- EGEE

- ATLAS
- CMS
- D0
- Star
- QCD
- Lattice Grid
- GridPP

**Notes:**
Current UK e-Infrastructure.
This shows the National Centre, 8 regional centres and 2 laboratories in blue
	That is the original set up at the start of UK e-Science August 2001
	NeSC is jointly run by Edinburgh & Glasgow Universities
In 2003 several smaller centres were added (vermilion)
(1st call out)  The e-Science Institute is run by the National e-Science Centre.  It runs a programme of events and hosts visiting international researchers.  It was established in 2001.
The Open Middleware Infrastructure Institute was established in 2004, to provide support and direction for Grid middleware developed in the UK.  It is based at the University of Southampton.
The Grid Operations Centre will be established in 2004.
(2nd call out) The Digital Curation Centre was established in 2004 by the Universities of Edinburgh and Glasgow, the UK Online Library Network at the University of Bath, and the Central Laboratories at Daresbury and Rutherford.  It’s job is to provide advice on curating scientific data and on preserving digital media, formats, and access software.
(3rd call out)  Edinburgh is one of the 4 founders of the Globus Alliance (Sept 2003) which will take responsibility for the future of the Globus Toolkit.  The other founders are:  Chicago University (Argonne National Lab), University of Southern California, Los Angeles (Information Sciences Institute) and the PDC, Stockholm, Sweden
(4th call out)  The EU EGEE project (Enabling Grids for E-Science in Europe) is establishing a common framework for Grids in Europe.  The UK e-Science programme has several connections with EGEE.  NeSC leads the training component for the whole of Europe.

## Bio-Medical Community

- CancerGrid
- eDiamond
- myGrid
- Integrative Biology
- Mouse Atlas

**Notes:**
Current UK e-Infrastructure.
This shows the National Centre, 8 regional centres and 2 laboratories in blue
	That is the original set up at the start of UK e-Science August 2001
	NeSC is jointly run by Edinburgh & Glasgow Universities
In 2003 several smaller centres were added (vermilion)
(1st call out)  The e-Science Institute is run by the National e-Science Centre.  It runs a programme of events and hosts visiting international researchers.  It was established in 2001.
The Open Middleware Infrastructure Institute was established in 2004, to provide support and direction for Grid middleware developed in the UK.  It is based at the University of Southampton.
The Grid Operations Centre will be established in 2004.
(2nd call out) The Digital Curation Centre was established in 2004 by the Universities of Edinburgh and Glasgow, the UK Online Library Network at the University of Bath, and the Central Laboratories at Daresbury and Rutherford.  It’s job is to provide advice on curating scientific data and on preserving digital media, formats, and access software.
(3rd call out)  Edinburgh is one of the 4 founders of the Globus Alliance (Sept 2003) which will take responsibility for the future of the Globus Toolkit.  The other founders are:  Chicago University (Argonne National Lab), University of Southern California, Los Angeles (Information Sciences Institute) and the PDC, Stockholm, Sweden
(4th call out)  The EU EGEE project (Enabling Grids for E-Science in Europe) is establishing a common framework for Grids in Europe.  The UK e-Science programme has several connections with EGEE.  NeSC leads the training component for the whole of Europe.

## Support Services

**CeSC (Cambridge**)

- Grid Operations
- Support
- Centre

**Notes:**
Current UK e-Infrastructure.
This shows the National Centre, 8 regional centres and 2 laboratories in blue
	That is the original set up at the start of UK e-Science August 2001
	NeSC is jointly run by Edinburgh & Glasgow Universities
In 2003 several smaller centres were added (vermilion)
(1st call out)  The e-Science Institute is run by the National e-Science Centre.  It runs a programme of events and hosts visiting international researchers.  It was established in 2001.
The Open Middleware Infrastructure Institute was established in 2004, to provide support and direction for Grid middleware developed in the UK.  It is based at the University of Southampton.
The Grid Operations Centre will be established in 2004.
(2nd call out) The Digital Curation Centre was established in 2004 by the Universities of Edinburgh and Glasgow, the UK Online Library Network at the University of Bath, and the Central Laboratories at Daresbury and Rutherford.  It’s job is to provide advice on curating scientific data and on preserving digital media, formats, and access software.
(3rd call out)  Edinburgh is one of the 4 founders of the Globus Alliance (Sept 2003) which will take responsibility for the future of the Globus Toolkit.  The other founders are:  Chicago University (Argonne National Lab), University of Southern California, Los Angeles (Information Sciences Institute) and the PDC, Stockholm, Sweden
(4th call out)  The EU EGEE project (Enabling Grids for E-Science in Europe) is establishing a common framework for Grids in Europe.  The UK e-Science programme has several connections with EGEE.  NeSC leads the training component for the whole of Europe.

## Support Services

**CeSC (Cambridge**)

- Grid Operations
- Support
- Centre

**Notes:**
Current UK e-Infrastructure.
This shows the National Centre, 8 regional centres and 2 laboratories in blue
	That is the original set up at the start of UK e-Science August 2001
	NeSC is jointly run by Edinburgh & Glasgow Universities
In 2003 several smaller centres were added (vermilion)
(1st call out)  The e-Science Institute is run by the National e-Science Centre.  It runs a programme of events and hosts visiting international researchers.  It was established in 2001.
The Open Middleware Infrastructure Institute was established in 2004, to provide support and direction for Grid middleware developed in the UK.  It is based at the University of Southampton.
The Grid Operations Centre will be established in 2004.
(2nd call out) The Digital Curation Centre was established in 2004 by the Universities of Edinburgh and Glasgow, the UK Online Library Network at the University of Bath, and the Central Laboratories at Daresbury and Rutherford.  It’s job is to provide advice on curating scientific data and on preserving digital media, formats, and access software.
(3rd call out)  Edinburgh is one of the 4 founders of the Globus Alliance (Sept 2003) which will take responsibility for the future of the Globus Toolkit.  The other founders are:  Chicago University (Argonne National Lab), University of Southern California, Los Angeles (Information Sciences Institute) and the PDC, Stockholm, Sweden
(4th call out)  The EU EGEE project (Enabling Grids for E-Science in Europe) is establishing a common framework for Grids in Europe.  The UK e-Science programme has several connections with EGEE.  NeSC leads the training component for the whole of Europe.

## Support Services

**CeSC (Cambridge**)

- e-Science Institute

- Grid Operations
- Support
- Centre

- National eScience Centre

**Notes:**
Current UK e-Infrastructure.
This shows the National Centre, 8 regional centres and 2 laboratories in blue
	That is the original set up at the start of UK e-Science August 2001
	NeSC is jointly run by Edinburgh & Glasgow Universities
In 2003 several smaller centres were added (vermilion)
(1st call out)  The e-Science Institute is run by the National e-Science Centre.  It runs a programme of events and hosts visiting international researchers.  It was established in 2001.
The Open Middleware Infrastructure Institute was established in 2004, to provide support and direction for Grid middleware developed in the UK.  It is based at the University of Southampton.
The Grid Operations Centre will be established in 2004.
(2nd call out) The Digital Curation Centre was established in 2004 by the Universities of Edinburgh and Glasgow, the UK Online Library Network at the University of Bath, and the Central Laboratories at Daresbury and Rutherford.  It’s job is to provide advice on curating scientific data and on preserving digital media, formats, and access software.
(3rd call out)  Edinburgh is one of the 4 founders of the Globus Alliance (Sept 2003) which will take responsibility for the future of the Globus Toolkit.  The other founders are:  Chicago University (Argonne National Lab), University of Southern California, Los Angeles (Information Sciences Institute) and the PDC, Stockholm, Sweden
(4th call out)  The EU EGEE project (Enabling Grids for E-Science in Europe) is establishing a common framework for Grids in Europe.  The UK e-Science programme has several connections with EGEE.  NeSC leads the training component for the whole of Europe.

## New Application-Focused Centres

**CeSC (Cambridge**)

- Arts and Humanities
- e-Science Support Centre

- NationalCentre fore-SocialScience

- National Institute
- for
- Environmentale-Science

**Notes:**
Current UK e-Infrastructure.
This shows the National Centre, 8 regional centres and 2 laboratories in blue
	That is the original set up at the start of UK e-Science August 2001
	NeSC is jointly run by Edinburgh & Glasgow Universities
In 2003 several smaller centres were added (vermilion)
(1st call out)  The e-Science Institute is run by the National e-Science Centre.  It runs a programme of events and hosts visiting international researchers.  It was established in 2001.
The Open Middleware Infrastructure Institute was established in 2004, to provide support and direction for Grid middleware developed in the UK.  It is based at the University of Southampton.
The Grid Operations Centre will be established in 2004.
(2nd call out) The Digital Curation Centre was established in 2004 by the Universities of Edinburgh and Glasgow, the UK Online Library Network at the University of Bath, and the Central Laboratories at Daresbury and Rutherford.  It’s job is to provide advice on curating scientific data and on preserving digital media, formats, and access software.
(3rd call out)  Edinburgh is one of the 4 founders of the Globus Alliance (Sept 2003) which will take responsibility for the future of the Globus Toolkit.  The other founders are:  Chicago University (Argonne National Lab), University of Southern California, Los Angeles (Information Sciences Institute) and the PDC, Stockholm, Sweden
(4th call out)  The EU EGEE project (Enabling Grids for E-Science in Europe) is establishing a common framework for Grids in Europe.  The UK e-Science programme has several connections with EGEE.  NeSC leads the training component for the whole of Europe.

## The e-Science Centres

**CeSC (Cambridge**)

- e-Science Institute

- Open Middleware Infrastructure Institute

- EGEE

- Grid Operations
- Support
- Centre

- Open Middleware Infrastructure Institute

- Arts and Humanities e-Science Support Centre

- NationalCentre fore-SocialScience

- Open Middleware Infrastructure Institute- UK

- National Institute
- for
- Environmentale-Science

- National eScience Centre

**Notes:**
Current UK e-Infrastructure.
This shows the National Centre, 8 regional centres and 2 laboratories in blue
	That is the original set up at the start of UK e-Science August 2001
	NeSC is jointly run by Edinburgh & Glasgow Universities
In 2003 several smaller centres were added (vermilion)
(1st call out)  The e-Science Institute is run by the National e-Science Centre.  It runs a programme of events and hosts visiting international researchers.  It was established in 2001.
The Open Middleware Infrastructure Institute was established in 2004, to provide support and direction for Grid middleware developed in the UK.  It is based at the University of Southampton.
The Grid Operations Centre will be established in 2004.
(2nd call out) The Digital Curation Centre was established in 2004 by the Universities of Edinburgh and Glasgow, the UK Online Library Network at the University of Bath, and the Central Laboratories at Daresbury and Rutherford.  It’s job is to provide advice on curating scientific data and on preserving digital media, formats, and access software.
(3rd call out)  Edinburgh is one of the 4 founders of the Globus Alliance (Sept 2003) which will take responsibility for the future of the Globus Toolkit.  The other founders are:  Chicago University (Argonne National Lab), University of Southern California, Los Angeles (Information Sciences Institute) and the PDC, Stockholm, Sweden
(4th call out)  The EU EGEE project (Enabling Grids for E-Science in Europe) is establishing a common framework for Grids in Europe.  The UK e-Science programme has several connections with EGEE.  NeSC leads the training component for the whole of Europe.

## Where are applications going

- Growing and growing
- Not every application is well suited to Grids, those that are
- Collaborate
- Share data or resources
- When will this be resolved?
- A lot of work still to be done, especially in “non-traditional” communities
- Efforts like the training school (and NCeSS, AHCeSS) make a difference as well

## Demand for Production-Level Services

- Reliable services
- Dependable help
- Straight-forward solutions
- From the UK Requirements trip (summer 2004), biggest request was for more robust systems and easier deployment

## So where are we?

- Software is MUCH easier to install and run
- OMII and Globus both took this to heart
- Service testing is much more common
- GT4 underwent literally billions of service calls before its release
- OMII is part of the NMI testing service
- Troubleshooting is still an open problem
- Documentation is always a problem
- When will this be resolved?
- Production-quality grids are starting to appear this year- but is it one you have access to?

## Summary

- Grids are here, and the problems they solve aren’t going away
- The problems they create aren’t going away either, although there’s not a lot more focus especially for
- Data-oriented approaches
- Easy-of-use
- Production-level software and services

## For More Information

- Jennifer M. Schopf
- jms@mcs.anl.gov
- www.mcs.anl.gov/~jms
- Support from NeSC/JISC, NSF, DOE
- This talk
- 	www.mcs.anl.gov/~jms/Talks (not there yet)
- National eScience Centre
- 	www.nesc.ac.uk