# Presentation: 659134

## Globus Toolkit 4:Futures and Open Issues

- Jennifer M. Schopf
- UK National eScience Centre
- Argonne National Lab

## What is a Grid

- Resource sharing
- Computers, storage, sensors, networks, ...
- Sharing always conditional: issues of trust, policy, negotiation, payment, ...
- Coordinated problem solving
- Beyond client-server: distributed data analysis, computation, collaboration, ...
- Dynamic, multi-institutional virtual orgs
- Community overlays on classic org structures
- Large or small, static or dynamic

## Not A New Idea

- Late 70’s – Networked operating systems
- Late 80’s – Distributed operating system
- Early 90’s – Heterogeneous computing
- Mid 90’s - Metacomputing
- Then the “Grid” – Foster and Kesselman, 1999
- Also called parallel distributed computing

## Why is this hard/different?

- Lack of central control
- Where things run
- When they run
- Shared resources
- Contention, variability
- Communication
- Different sites implies different sys admins, users, institutional goals, and often “strong personalities”

## So why do it?

- Computations that need to be done with a time limit
- Data that can’t fit on one site
- Data owned by multiple sites
- Applications that need to be run bigger, faster, more

## What Is the Globus Toolkit?

- The Globus Toolkit is a collection of solutions to problems that frequently come up when trying to build collaborative distributed applications
- Heterogeneity
- Focus on *simplifying heterogenity* for application developers
- Working towards more “vertical solutions” in future versions.
- Standards
- Capitalize on and encourage use of existing standards (IETF, W3C, OASIS, GGF).
- Reference implementations of new/proposed standards in these organizations.

## With Grid Computing – Forget Homogeneity!

- Trying to force homogeneity on users is futile. Everyone has their own preferences, sometimes even *dogma*.
- The Internet provides the model...

## Evolution of the Grid

- Increased functionality,
- standardization

- Time

- Globus Toolkit

- Open Grid
- Services Arch

- GGF: OGSI, WSRF, ...
- (leveraging OASIS, W3C, IETF)
- Multiple implementations,
- including Globus Toolkit

- Defacto standards
- GGF: GridFTP, GSI
- (leveraging IETF)

- Custom
- solutions

- X.509,
- LDAP,
- FTP, ...

- Web services

- App-specific
- Services

## Globus is Service-Oriented Infrastructure Technology

- Software for service-oriented infrastructure
- Service enable new & existing resources
- E.g., GRAM on computer, GridFTP on storage system, custom application service
- Uniform abstractions & mechanisms
- Tools to build applications that exploit service-oriented infrastructure
- Registries, security, data management, ...
- Open source & open standards
- Each empowers the other
- eg – monitoring across different protocols is hard
- Enabler of a rich tool & service ecosystem

**Notes:**
“Service enable”: enabling access, monitoring, management, etc., in common ways. 

NOT just throwing some WSDL on something. Review history: need commonality at the low levels. Web services help. WSRF is good. Need more: agreements, resource modeling, common management. EXAMPLE: Common monitoring – LDAP and GridFTP and GRAM all had subtly different ways of monitoring. Common monitoring capability that spans workflows very hard. Thus WSRF/WSN. EXAMPLE: Want to talk about QoS negoitation in the same way for all resources.

## Globus Toolkit V4.0

- Major release planned April 29th 2005
- Second internal release candidate cut yesterday – still on good track for this date
- Fifteen months of design, development, and testing
- 1.8M lines of code
- Major contributions from five institutions
- Hundreds of millions of service calls executed over weeks of continuous operation
- Significant improvements over GT3 code base in all dimensions

## Our Goals for GT4

- Usability, reliability, scalability, ...
- Web service components have quality equal or superior to pre-WS components
- Documentation at acceptable quality level
- Consistency with latest standards (WS-*, WSRF, WS-N, etc.) and Apache platform
- WS-I Basic (Security) Profile compliant
- New components, platforms, languages
- And links to larger Globus ecosystem

## Slide 12

## GT4 Components and Performance

- Globus Toolkit Components
- Core
- Security
- Data Management
- Resource Management
- Monitoring
- Performance in the broadest sense of the word.
- How fast
- How many
- How stable
- (How easy)
- www-unix.globus.org/toolkit/docs/development/4.0-drafts/perf_overview.html

## GT4 Web Services Core

- Supports both Globus services (GRAM, RFT, Delegation, etc.) & user-developed services
- Redesign to enhance scalability, modularity, performance, usability
- Leverages existing WS standards
- WS-I Basic Profile: WSDL, SOAP, etc.
- WS-Security, WS-Addressing
- Adds support for emerging WS standards
- WS-Resource Framework, WS-Notification
- Java, Python, & C hosting environments

## GT4 Web Services Core

**GT4 Web Services Core**

## Open Source/Open Standards

- WSRF developed in collaboration with IBM
- Currently in OASIS process
- Contributions to Apache for
- WS-Security
- WS-Addressing
- Axis
- Apollo (WSRF)
- Hermes (WS-Notification)

## Java Core Performance

- We’ve been working hard to increase basic messaging performance
- Factor of four improvement over GT3 so far
- Reliability
- Core can scale to a very large number of resources (>10,000)

## Java Core Messaging Performance

## GT4 Security Highlights

- Standards based support for message level and transport level security
- Transport level is default due to performance
- Standards based authorization (SAML) via Community Authorization Service (CAS) or callouts
- Stand-alone delegation service
- More authentication options
- MyProxy, simpleCA, ...

## GT4’s Use of Security Standards 

## GT4 Security

- Users

## Security Performance

- We’ve measured performance for both WS and transport security mechanisms
- See next slide for graph
- Transport security is significantly faster than WS security
- We made transport security (i.e. https) our default
- We’re working on making it even faster by using connection caching

## Slide 23

## GT4 Data Management

- Stage large data to/from nodes
- Replicate data for performance & reliability
- Locate data of interest
- Provide access to diverse data sources
- File systems, parallel file systems, hierarchical storage (GridFTP)
- Databases (OGSA-DAI)

## GT4 Data Functions

- Find your data: **Replica Location Service**
- Managing ~40M files in production settings
- Move/access your data: **GridFTP**, **RFT**
- High-performance striped data movement
- Couple data & execution management
- GRAM uses GridFTP and RFT for staging

## GridFTP in GT4

- 100% Globus code
- No licensing issues
- Stable, extensible
- IPv6 Support
- XIO for different transports
- Striping  multi-Gb/sec wide area transport
- Pluggable
- Front-end: e.g., future WS control channel
- Back-end: e.g., HPSS, cluster file systems
- Transfer: e.g., UDP, NetBLT transport

**Notes:**
Future: integration of provisioning
E.g., storage reservation

## GridFTP Performance

- TeraGrid Striping results
- 30Gbs network, 32 IBM ia64 nodes
- Ran varying number of stripes
- Ran both memory-to-memory and disk-to-disk

## Memory to MemoryStriping Performance

- High linear scalability (slope near 1)
- 27 Gbs on a 30 Gbs link (90% utilization) with 32 nodes

## Disk to Disk Striping Performance

- Limited by the storage system
- Achieved 17.5 Gbs

## And in conversation...

- “We think we have hit the limit of python code. The GridFTP C libraries are delivering data so fast to the buffers that the python client code cannot keep up in doing the fseek, fwrite, and then re-register the data callback. We are going to have to code our "transfer agents" entirely in C for S5.Not a bad problem to have...”
- Scott Koranda, Dept of Physics, University of Minnesota

## Reliable File Transfer:Third Party Transfer

**Reliable File Transfer:****Third Party Transfer**

- RFT Service

- RFT Client

- SOAP Messages

- Notifications(Optional)

- GridFTP Server

- GridFTP Server

- Fire-and-forget transfer
- Web services interface
- Many files & directories
- Integrated failure recovery

## RFT Performance Stats

- Current maximum request size is approx 20,000 entries with a default 64MB heap size.
- Infinite transfer - LAN
- ~120,000 transfers (servers were killed by mistake)
- Was a good test. Found a corner  case where postgres was not able to perform ~ 3 update queries / sec and was using up CPU
- Infinite transfer – WAN
-  ~67000 transfers (killed because of the same reason as above)
- Sloan Digital Sky Survey DR3 archive move
- 900+K files, 6 TB
- Killed the transfer several times for recoverability testing
- No human intervention has been required to date

## Replica Location Service

- Identify location of files via logical to physical name map
- Distributed indexing of names, fault tolerant update protocols
- GT4 version scalable & stable
- Managing ~40 million files across ~10 sites

- Index

- Index

| Local DB | Update send (secs) | Bloom filter  (secs)  | Bloom filter (bits) |
| --- | --- | --- | --- |
| 10K | <1 | 2 | 1 M |
| 1 M | 2 | 24 | 10 M |
| 5 M | 7 | 175 | 50 M |

## LIGO Use of RLS

- Some hands-on numbers
- Produce 1 TB per day
- 8 sites
- > 3 million entries in the RLS
- > 30 million files
- “This replication of data using RLS and GridFTP is enabling more gravitational wave data analysts across the world to do more science more efficiently then ever before. Globus RLS and GridFTP are in the critical path for LIGO data analysis.”

## Data Replication Service (tech preview)

- Pull “missing” files to local site

- Site B

- List of required Files

*Reliable File Transfer**Service*

*Data Replication Service*

*GridFTP*

*Local*
*Replica**Catalog*

*Replica*
*Location**Index*

- Site A

*Data Replication Service*

*Reliable File Transfer Service*

*Local*
*Replica*
*Catalog*

*Replica**Location**Index*

*GridFTP*

## OGSA-DAI

- Flexible & Composable Middleware
- Data access
- Relational & XML Databases, semi-structured files
- Data integration
- Multiple data delivery mechanisms, data translation
- Extensible & Efficient framework
- Request documents contain multiple tasks
- A task = execution of an activity
- Group work to enable efficient operation
- Extensible set of activities
- > 30 predefined, framework for writing your own
- Moves computation to data
- Pipelined and streaming evaluation
- Concurrent task evaluation

## OGSA-DAI

- Current Release = Release 5 = in GT4
- Added Installation wizards & indexed files
- >1100 registered users we know about
- Running on 3 message passing infrastructures
- Release 6 – May 2005
- Improved client side API
- Explicit control of sequential & parallel tasks
- Dynamic reconfigurability
- WS-DAI reference implementation

## Execution Management (GRAM)

- Common WS interface to schedulers
- Unix, Condor, LSF, PBS, SGE, ...
- More generally: interface for process execution management
- Lay down execution environment
- Stage data
- Monitor & manage lifecycle
- Kill it, clean up
- A basis for application-driven provisioning

## GT4 GRAM

- 2nd-generation WS implementation
-  optimized for performance, stability, scalability
- Streamlined critical path
- Use only what you need
- Flexible credential management
- Credential cache & delegation service
- GridFTP & RFT used for data operations
- Data staging & streaming output
- Eliminates redundant GASS code
- Single and multi-job support

**Notes:**
See: Karl C. on GRAM
See: Kate Keahey on dynamic workspaces and VMs
Future: towards richer execution management
Virtual machine integration, support for dynamic accounts, quality of service, provisioning

## GT4 GRAM Structure:WSRF/WSN Poster Child

- GRAM
- services

- GT4 Java Container

- GRAM
- services

- Delegation

- RFT File
- Transfer

*Transfer*
*request*

- GridFTP

- Remote
- storage
- element(s)

- Local
- scheduler

- User
- job

- Compute element

- GridFTP

- sudo

- GRAM
- adapter

*FTP*
*control*

*Local job*
*control*

*Delegate*

*FTP data*

**Client**

*Job functions*

*Delegate*

- Service host(s) and compute element(s)

**GT4 GRAM Structure:****WSRF/WSN Poster Child**

## Some of our Goals

- “GRAM should add little to no overhead compared to an underlying batch system”
- Submit as many jobs to GRAM as is possible to the underlying scheduler
- Goal - 10,000 jobs to a batch scheduler
- Goal – efficiently fill the process table for fork scheduler
- Submit/process jobs as fast to GRAM as is possible to the underlying scheduler
- Goal - 1 per second
- We are not there yet...
- A range of limiting factors at play

##  Design Decisions

- Efforts and features towards the goal
- Allow job brokers the freedom to optimize
- E.g. Condor-G is smarter than globusrun
- Protocol steps made optional and shareable
- Reduced cost for GRAM service on host
- Single WSRF host environment
- Better job status monitoring mechanisms
- More scalable/reliable file handling
- GridFTP and RFT instead of globus-url-copy
- Removal of non-scalable GASS caching
- GT4 tests performing better than GT3 did
- But more work to do

## GRAM 3.9.4 performance

- Throughput
- Test: Simple job to fork scheduler (/bin/date); no staging, streaming, or cleanup
- ~77 jobs/min sustained
- ~60 jobs/minute with delegation
- Long Running test
- Ran 500,000+ sequential jobs over 23 days
- These included staging, delegation, fork job manager

## Gram Performance (2)

- Concurrency
- Job submits to Condor scheduler (long running sleep job); no staging, streaming, or cleanup; no delegation
- Current limit is 32,000 jobs due to a Linux directory limit
- using multiple sub-directories will resolve this, look for this in 4.2

## Monitoring and Discovery

- “Every service should be monitorable and discoverable using common mechanisms”
- WSRF/WSN provides those mechanisms
- A common aggregator framework for collecting information from services, thus:
- Index Service: Registry supporting Xpath queries, with caching
- Trigger Service: perform action on condition
- Deep integration with Globus containers & services: every GT4 service is discoverable
- GRAM, RFT, GridFTP, CAS, ...

## GT4 Monitoring & Discovery

**GT4 ****Monitoring & Discovery**

- GRAM

- User

- MDS-Index

- GT4 Cont.

- RFT

- MDS-Index

- GT4 Container

- MDS-Index

- GridFTP

- adapter

- Registration &
- WSRF/WSN Access

- Custom protocols
- for non-WSRF entities

- Clients
- (e.g., WebMDS)

- Automated
- registration
- in container

- WS-ServiceGroup

**Notes:**
WebMDS: configurable via XSLT style sheets

## MDS4 Extensibility

- Aggregator framework provides
- Registration management
- Collection of information from Grid Resources
- Plug in interface for data access, collection ,query, ...
- WebMDS framework provides for customized display
- XSLT transformations

## MDS4 in 3.9.5Index Query Performance

- Small queries – 10 minute averages
- Message size 7.5 KB
- Requests processed: 11262
- Average round-trip time in milliseconds: 16
- Medium queries – 10 minute averages
- Message Size 32KB
- Queries processed: 6232
- Average round-trip time in milliseconds: 29

## Long Running Test

- Ran 14 days (killed by accident during other testing)
- Over 94 million requests processed,
- 76 requests/sec average
- 13 millisecond average Query RTT
- Has also had diperf tests run against it (next slide)

## Slide 50

## GT4 Documentationis Much Improved!

**GT4 Documentation****is ****Much Improved!**

## The Globus Ecosystem

**Globus components** address core issues relating to resource access, monitoring, discovery, security, data movement, etc.
- GT4 being the latest version
- A larger **Globus ecosystem** of open source and proprietary components provide complementary components
- A growing list of components
- These components can be combined to produce **solutions** to Grid problems
- We’re building a list of such solutions

## Many Tools Build on, or Can Contribute to, GT4-Based Grids

- Condor-G, DAGman
- MPICH-G2
- GRMS
- Nimrod-G
- Ninf-G
- Open Grid Computing Env.
- Commodity Grid Toolkit
- GriPhyN Virtual Data System
- Virtual Data Toolkit
- GridXpert Synergy

- Platform Globus Toolkit
- VOMS
- PERMIS
- GT4IDE
- Sun Grid Engine
- PBS scheduler
- LSF scheduler
- GridBus
- TeraGrid CTSS
- NEES
- IBM Grid Toolbox
- ...

## 2005 and Beyond

- We have a solid Web services base
- We now want to build, on that base, a open source service-oriented infrastructure
- Virtualization
- New services for provisioning, data management, security, VO management
- End-user tools for application development
- Etc., etc.

## Globus and its User Community

- How can “we” best support “you”?
- We try to provide the best software we can
- We use bugzilla & other community tools
- We work to grow the set of contributors
- How can “you” best support “us”?
- Become a contributor: of software, bug fixes, answers to questions, documentation
- Provide us with success stories that can justify continued Globus development
- Promote Globus within your communities

## Working with GT4

- Download and use the software, and provide feedback
- Join **gt4friends@globus.org** mail list
- Review, critique, add to documentation
- Globus Doc Project: **http://gdp.globus.org**
- Tell us about your GT4-related tool, service, or application

## So...

- GT4 is a significant step forward in the quality, functionality and standards compliance of GT.
- Beta release available for immediate use, final April 29th
- Downloads and docs at:
- www.globustoolkit.org

- 2nd Edition
- www.mkp.com/grid2

## But...

## Things heard about Grids...

- “Isn’t the Grid just a funding construct?” (SC ’01)
- "Grid computing has been more hype than reality,“ - Hewlett-Packard CEO Carly Fiorina, 10/03
- “[Customers] don't need the Globus Toolkit to do high-performance compute clusters" - Charles Fitzgerald, a Microsoft general manager, Information Week 1/05
- “We tried to install Globus and found out that it was too hard to do.  So we decided to just write our own....”

## Where are allthe (happy) users?

- In July ‘04 I spoke with 25 UK user groups, and on occasion it got ugly
- _[www.](http://www.nesc.ac.uk/technical_papers/UKeS-2004-08.pdf)__[nesc](http://www.nesc.ac.uk/technical_papers/UKeS-2004-08.pdf)__[.ac.](http://www.nesc.ac.uk/technical_papers/UKeS-2004-08.pdf)__[uk](http://www.nesc.ac.uk/technical_papers/UKeS-2004-08.pdf)__[/technical_papers/](http://www.nesc.ac.uk/technical_papers/UKeS-2004-08.pdf)__[UKeS](http://www.nesc.ac.uk/technical_papers/UKeS-2004-08.pdf)__[-2004-08.](http://www.nesc.ac.uk/technical_papers/UKeS-2004-08.pdf)__[pdf](http://www.nesc.ac.uk/technical_papers/UKeS-2004-08.pdf)_
- Many users have been told to use the Grid to get funding, not because they actually want to
- There are a few well known successes (LHC, CACTUS, and a couple others) but this isn’t widespread enough to be considered more than a one-off

## “We expected using Gridsto be a lot of work...

- Parallel computing showed us that they “If you build it they will come” scenario just won’t work
- Until debuggers, fast compilers, languages, libraries, etc. the users didn’t want to use parallel machines
- Many hundreds, even thousand, of hours went into re-writing codes for parallel machines

## ...but how much is acceptable?”

- There is the impression (right or wrong) that only heroic efforts will allow you to use a Grid
- Some re-writing of code required
- Access to resources isn’t easy even once code is changed

## Where are we today?

- What a user would like:
- “Run my job, finish by lunch”
- “Get a data set that has these attributes”
- “Tell me when that simulation will finish”
- Where are we today:
- Specify exact machines, data files, explicit data transfers, etc
- Little (or no) dynamic information or prediction

## Where are we today (cont)

- General agreement we have basic functionality
- Tell me what this set of resources look like
- Run this job on that resource
- Transfer this file
- Globus (among others) does give these basic building blocks (mostly)
- General agreement general functionality isn’t enough by far

## How do we move forward?

- Users will only come when they have decent tools
- Simple enough for “easy” use
- Robust enough for “stupid” use
- Still allow work arounds for “hard-core” use
- Users are hampered by software that doesn’t do what they need it to
- Globus is NOT an end-to-end solution

## 2) Why doesn’t Grid softwaredo what we need it to yet?

- “Globus doesn’t provide end-to-end solutions”
- Globus Ecosystem Tutorial
- Globus is building blocks –still missing “vertical” solutions
- Mismatch between developer vision of use and users vision of use
- Many tools are used “off label”

## “Off Label” Use

- Tool built to do A is used for B
- This is good since a user has something to use
- This is bad since the tool is being in a way that wasn’t envisaged
- Arch concerns
- Scaling concerns
- Etc
- But without use of the tool, there’s no way to know how it will be used!

## What is a usage scenario?

- Information from the user about a specific use case
- What’s the right level of detail?
- What’s a “general” use case?
- Note: much application built software is one-off, but we need general tools that can adapt
- Who does this?
- Application scientists and computer scientists speak different languages (eg. C. Pancake)

## Will Grid softwareever meet user’s needs?

- Without better _communication_ between developers and users, the Grid cannot succeed
- Grids are about people, not just technology

## 3. Need for Standards:Information as a Case Study

- Open question – how should I store the information about a Grid?
- Globus Monitoring and Discovery Service (MDS)
- A tool that does streaming data like R-GMA?
- A cluster tool over many sites like Ganglia?
- A certification tool like Inca from the TG project?
- A Grid-wide data base?
- All of these are right for some of the data, no one is right for all uses

## Why are so many tools bad?

- Large number of tools isn’t bad
- Large number of tools that have no way to interoperate is!
- Grid3 has 8 different tools in use
- LHC has an equal number (at least!)

## Need for Standard Interfaces

- Need for standard APIs and protocols to allow easier
- Access to data sources
- Registration of data
- Archiving tools
- Standards for what information is available
- Standards for what that information means
- Standards for communication of errors
- This is in part what inspired Globus’s move to web services!

## Standards are a necessity,not a luxury

- Without standards of all kinds – protocols, APIs, languages - all the information in the world won’t do us any good
- Open question about right process for standardization
- GGF, OASIS, IETF
- Need for standards vs standardizing too soon
- Need for standard vs time lag for agreement on standards

## 4. How do we make Grids secure?

- Without security we can’t have a Grid
- EVERYTHING needs to be secure-
- Who can run on a machine
- File transfers
- What data does someone have access to	 (program data, system data)
- Who can access which services?

## Security vs. Usability

- Users want security but don’t want to deal with it
- If security is hard- it won’t be used
- Most security (including Grid Security Infrastructure (GSI)) is based on public key infrastructure (PKI)
- Users have files (public and private keys) that must be secure, use reasonable passwords, etc.

## What about...

- Multiple certificates?
- Group access?
- Dynamic policy changes?
- Scalability?
- Overheads
- Etc., etc., etc...

## Security is an open question

- Until security is made easier to use, it won’t be used
- Until security is made easier to manage at the group level, it won’t be used
- Without security no one will really use the Grid

## 5. Socio-political Issues

- Hardest problems are often not technical ones
- Multiple administration domains means multiple policies
- Multiple countries means multiple communication styles
- Decisions are often made on non-technical basis

## Communication is hard

- Too many people in the mix
- Not everyone is informed of status updates
- Often hallway conversation becomes what people believe
- Too often assumptions are not verified
- Many communication styles can lead to misunderstandings

## What to do?

- Ongoing efforts to continue better communication are needed to build a global community
- When in doubt – ask someone of directly!
- And please – constructive criticism, reporting of errors, etc – just saying “Globus Sucks” simply isn’t helpful 

## 6+: Other open problems

- What performance is acceptable?
- What do we do about variance?
- What about easier testbed setup?
- Where are the benefits to encourage sharing on the Grid?
- Where are the benefits for the sys admins – users get a plus, PIs get a plus but what about them?
- How do we educate the funding agencies about the need for hardened software, documentation, and support?
- What cost models are needed by the Grid?
- Economic Grids are only the first step
- And many more...

## Progress

- Significant improvements in security infrastructure
- Basic functionality is much closer
- More funding aid for support
- Need for better-defined use cases and simpler deployment has been strengthened, as has the need for basic information and basic information services

## Where are theperformance metrics for success?

- No more “Grid” papers, just a footnote that states “This work was achieved using the Grid”
- Supercomputer centers don’t give a user the choice of using their machines or the Grid, that line doesn’t exist (TG does this now!)
- SuperComputing demos can be run at any time of the year

## Conclusion

- Many interesting problems are left – both in terms of research and deployment issues
- Much work is being done to help address these open issues
- Next year’s open issues will be different yet

## References

- This talk:
- _[www.mcs.anl.gov/~jms/Talks](http://www.mcs.anl.gov/~jms/Talks)_ (not there yet)
- Globus Alliance:
- _[www.globus.org](http://www.globus.org/)_
- Globus Performance
- _[http://www-unix.globus.org/toolkit/docs/development/4.0-drafts/perf_overview.html](http://www-unix.globus.org/toolkit/docs/development/4.0-drafts/perf_overview.html)_
- Journal paper version open questions (dated):
- _[www.](http://www.mcs.anl.gov/Pubs/jmspubs.html)__[mcs](http://www.mcs.anl.gov/Pubs/jmspubs.html)__[.](http://www.mcs.anl.gov/Pubs/jmspubs.html)__[anl](http://www.mcs.anl.gov/Pubs/jmspubs.html)__[.](http://www.mcs.anl.gov/Pubs/jmspubs.html)__[gov](http://www.mcs.anl.gov/Pubs/jmspubs.html)__[/Pubs/](http://www.mcs.anl.gov/Pubs/jmspubs.html)__[jmspubs](http://www.mcs.anl.gov/Pubs/jmspubs.html)__[.html](http://www.mcs.anl.gov/Pubs/jmspubs.html)_
- Conversations with 25 UK User groups
- _[http://www.](http://www.nesc.ac.uk/technical_papers/UKeS-2004-08.pdf)__[nesc](http://www.nesc.ac.uk/technical_papers/UKeS-2004-08.pdf)__[.ac.](http://www.nesc.ac.uk/technical_papers/UKeS-2004-08.pdf)__[uk](http://www.nesc.ac.uk/technical_papers/UKeS-2004-08.pdf)__[/technical_papers/](http://www.nesc.ac.uk/technical_papers/UKeS-2004-08.pdf)__[UKeS](http://www.nesc.ac.uk/technical_papers/UKeS-2004-08.pdf)__[-2004-08.](http://www.nesc.ac.uk/technical_papers/UKeS-2004-08.pdf)__[pdf](http://www.nesc.ac.uk/technical_papers/UKeS-2004-08.pdf)_

## Contact Information

- Jennifer M. Schopf
- jms@mcs.anl.gov
- _[www.mcs.anl.gov/~jms](http://www.mcs.anl.gov/~jms)_
- Support from DOE, NSF, Microsoft, NeSC, JISC

- Slides on how globus works

## How Globus Works

**Globus** is a distributed open source community with many contributors & users
- CVS, documentation, bugzilla, email lists
- Modular structure allows many to contribute
**Globus Alliance Board** provides governance when needed
- Meritocracy: individuals who demonstrate ongoing contributions & commitment
- Primarily: what to include, when to release
**Globus Alliance** is an informal partnership of organizations led by Board members

**Notes:**
Board members may command an army...

## Evolution of the Globus Alliance

- Argonne/U.Chicago (Childers, Foster): 1995
- USC/ISI (Kesselman): 1995
- Edinburgh (Atkinson, Parsons): 2003
- Swedish PDC (Johnsson, Mulmo): 2003
- NCSA (Welch): 2004
- Univa (Czajkowski, Tuecke): 2004
*   **Other contributors will surely be added*

## From eScience to eBusiness

- Since ~2001, growing interest in Globus for commercial use
- Enterprises, IT vendors, ISVs asking Globus leaders to address commercial needs
- But hard to do in a research laboratory
- In response, we have created two new organizations
- Globus Consortium
- Univa

## Globus Consortium(www.globusconsortium.com)

- Nonprofit organization funded by companies to advance Globus Toolkit for enterprise use
- Initial sponsor members: HP, IBM, Intel, Sun
- Initial contributors: Nortel, Univa
- First two projects already identified
- Member-driven software quality improvements
- Contributions to job submission standards
- Other projects to be defined, e.g.
- Develop new features key to enterprise use
- Education & outreach

## Provider of commercial support, services, & products around open source Globus
Commercial distribution of GT4 & beyond
Integration with enterprise systems
Committed to open source & open standards
Founded by Tuecke, Foster, Kesselman
Tuecke left Argonne to be CEO
Foster, Kesselman remain at Argonne, ISI
Experienced management team 
Rich Miller, Vas Vasiliadis, Paul Davé, Bob Mandel

- Commercial distribution of GT4 & beyond
- Integration with enterprise systems
- Committed to open source & open standards
- Founded by Tuecke, Foster, Kesselman
- Tuecke left Argonne to be CEO
- Foster, Kesselman remain at Argonne, ISI
- Experienced management team
- Rich Miller, Vas Vasiliadis, Paul Davé, Bob Mandel

## Globus and its User Community

- How can “we” best support “you”?
- We try to provide the best software we can
- We use bugzilla & other community tools
- We work to grow the set of contributors
- How can “you” best support “us”?
- Become a contributor: of software, bug fixes, answers to questions, documentation
- Provide us with success stories that can justify continued Globus development
- Promote Globus within your communities

## Working with GT4

- Download and use the software, and provide feedback
- Join **gt4friends@globus.org** mail list
- Review, critique, add to documentation
- Globus Doc Project: **http://gdp.globus.org**
- Tell us about your GT4-related tool, service, or application

## So...

- GT4 is a significant step forward in the quality, functionality and standards compliance of GT.
- Beta release available for immediate use, final April 29th
- Downloads and docs at:
- www.globustoolkit.org

- 2nd Edition
- www.mkp.com/grid2

- Slides on open issues

## 6. What about performance?

- It’s not enough to use the Grid, it has to perform – otherwise, why bother?
- First prototypes rarely consider perf.
- MDS1–centralized LDAP
- MDS2–decentralized LDAP
- MDS3–decentralized Grid service
- MDS4-decentralized Web service
- Often performance is simply not known

## Performance of GIS Information Servers vs. Number of Users

- Zhang, Freschl, and Schopf, A Performance Study of Monitoring and Information Services for Distributed Systems, submitted to HPDC 2003.

## What we found

- Performance can be a matter of deployment
- Effect of background load
- Effect of network bandwidth
- Performance can be affected by underlying infrastructure
- LDAP/JAVA strengths and weaknesses
- Performance can be improved using standard CS techniques
- Caching; multi-threading; etc.

## Moral:

- Performance should be analyzed early and often
- Prototypes should be recognized as such and thrown out
- Without performance, no reason to use a Grid

## 7. What do we do about variance?

- Resources on the Grid change with time
- Bandwidth
- CPU load
- Disk space
- Memory usage
- Queue sizes

## Variance – technical problem

- How do you tell if something is slow versus broken?
- How do you make a prediction?

## Variance – socio-political

- Users want the same application to take roughly the same amount of time every time you run it
- Our experience – a longer running time that’s more predictable is preferred to a high variance, high risk situation

## Moral:

- Variance is here to live with, we need techniques to take advantage of it

## 8. How do we set up a Grid testbed?

- Bill Johnson, LBNL, talks about this often, based on IPG experience
- Get the sys admins involved
- Have a standard set-up
- Make this a priority at the start of a project
- Accounting – open issue
- Cross-site scheduling – open issue
- Policies across sites – open issue

## Example: Installing Globus

- GT2- how do you know it’s installed ok?
- Now have test scripts
- _[http://www-unix.globus.org/toolkit/testing/](http://www-unix.globus.org/toolkit/testing/)_
- But it would be better to have something automatic
- GT3 – how many configuration files do I need to work with?

## Moral:

- Users are building testbeds, but this is still hard
- Need to have rule of thumb published for assistance with this

## 4. How do we understand information once we get it?

- Assume we have access to information about the Grid – can we use it?
- Grid3 has 8 different tools in use, that give conflicting answers
- A monitoring system says “the load on machine X is Y”
- A scheduler wants to evaluate this data
- No common language for this to be communicated
- Some effort now to come up with a common schema (GLUE schema, work with CIM in GGF) but this only touched the surface, no agreement for moving forward