# Presentation: 933187

## Globus Toolkit 4:Current Status and Futures

- Jennifer M. Schopf
- UK National eScience Centre
- Argonne National Lab

## What is a Grid

- Resource sharing
- Computers, storage, sensors, networks, ...
- Sharing always conditional: issues of trust, policy, negotiation, payment, ...
- Coordinated problem solving
- Beyond client-server: distributed data analysis, computation, collaboration, ...
- Dynamic, multi-institutional virtual orgs
- Community overlays on classic org structures
- Large or small, static or dynamic

## Not A New Idea

- Late 70’s – Networked operating systems
- Late 80’s – Distributed operating system
- Early 90’s – Heterogeneous computing
- Mid 90’s - Metacomputing
- Then the “Grid” – Foster and Kesselman, 1999
- Also called parallel distributed computing

## Why is this hard/different?

- Lack of central control
- Where things run
- When they run
- Shared resources
- Contention, variability
- Communication
- Different sites implies different sys admins, users, institutional goals, and often “strong personalities”

## So why do it?

- Computations that need to be done with a time limit
- Data that can’t fit on one site
- Data owned by multiple sites
- Applications that need to be run bigger, faster, more

## What Is the Globus Toolkit?

- The Globus Toolkit is a collection of solutions to problems that frequently come up when trying to build collaborative distributed applications
- Heterogeneity
- Focus on *simplifying heterogenity* for application developers
- Working towards more “vertical solutions” in future versions.
- Standards
- Capitalize on and encourage use of existing standards (IETF, W3C, OASIS, GGF).
- Reference implementations of new/proposed standards in these organizations.

## With Grid Computing – Forget Homogeneity!

- Trying to force homogeneity on users is futile. Everyone has their own preferences, sometimes even *dogma*.
- The Internet provides the model...

## Evolution of the Grid

- Increased functionality,
- standardization

- Time

- Globus Toolkit

- Open Grid
- Services Arch

- GGF: OGSI, WSRF, ...
- (leveraging OASIS, W3C, IETF)
- Multiple implementations,
- including Globus Toolkit

- Defacto standards
- GGF: GridFTP, GSI
- (leveraging IETF)

- Custom
- solutions

- X.509,
- LDAP,
- FTP, ...

- Web services

- App-specific
- Services

## Globus is Service-Oriented Infrastructure Technology

- Software for service-oriented infrastructure
- Service enable new & existing resources
- E.g., GRAM on computer, GridFTP on storage system, custom application service
- Uniform abstractions & mechanisms
- Tools to build applications that exploit service-oriented infrastructure
- Registries, security, data management, ...
- Open source & open standards
- Each empowers the other
- eg – monitoring across different protocols is hard
- Enabler of a rich tool & service ecosystem

**Notes:**
“Service enable”: enabling access, monitoring, management, etc., in common ways. 

NOT just throwing some WSDL on something. Review history: need commonality at the low levels. Web services help. WSRF is good. Need more: agreements, resource modeling, common management. EXAMPLE: Common monitoring – LDAP and GridFTP and GRAM all had subtly different ways of monitoring. Common monitoring capability that spans workflows very hard. Thus WSRF/WSN. EXAMPLE: Want to talk about QoS negoitation in the same way for all resources.

## Globus Toolkit V4.0

- Major release planned April 29th 2005
- Second internal release candidate cut Monday – still on good track for this date
- Fifteen months of design, development, and testing
- 1.8M lines of code
- Major contributions from five institutions
- Hundreds of millions of service calls executed over weeks of continuous operation
- Significant improvements over GT3 code base in all dimensions

## Our Goals for GT4

- Usability, reliability, scalability, ...
- Web service components have quality equal or superior to pre-WS components
- Documentation at acceptable quality level
- Consistency with latest standards (WS-*, WSRF, WS-N, etc.) and Apache platform
- WS-I Basic (Security) Profile compliant
- New components, platforms, languages
- And links to larger Globus ecosystem

## Slide 12

## GT4 Components and Performance

- Globus Toolkit Components
- Core
- Security
- Data Management
- Resource Management
- Monitoring
- Performance in the broadest sense of the word.
- How fast
- How many
- How stable
- (How easy)
- www-unix.globus.org/toolkit/docs/development/4.0-drafts/perf_overview.html

## GT4 Web Services Core

- Supports both Globus services (GRAM, RFT, Delegation, etc.) & user-developed services
- Redesign to enhance scalability, modularity, performance, usability
- Leverages existing WS standards
- WS-I Basic Profile: WSDL, SOAP, etc.
- WS-Security, WS-Addressing
- Adds support for emerging WS standards
- WS-Resource Framework, WS-Notification
- Java, Python, & C hosting environments

## GT4 Web Services Core

**GT4 Web Services Core**

## Open Source/Open Standards

- WSRF developed in collaboration with IBM
- Currently in OASIS process
- Contributions to Apache for
- WS-Security
- WS-Addressing
- Axis
- Apollo (WSRF)
- Hermes (WS-Notification)

## Java Core Performance

- We’ve been working hard to increase basic messaging performance
- Factor of four improvement over GT3 so far
- Reliability
- Core can scale to a very large number of resources (>10,000)

## Java Core Messaging Performance

## Why Grid Security is Hard

- Resources being used may be valuable & the problems being solved sensitive
- Resources are often located in distinct administrative domains
- Each resource has own policies & procedures
- Set of resources used by a single computation may be large, dynamic, and unpredictable
- Not just client/server, requires delegation
- It must be broadly available & applicable
- Standard, well-tested, well-understood protocols; integrated with wide variety of tools

## Basic Grid Security Mechanisms

- Basic Grid authentication and authorization mechanisms come in two flavors.
- Pre-Web services
- Web services
- Both are included in the Globus Toolkit, and both provide vital security features.
- Grid-wide identities implemented as PKI certificates
- Transport-level and message-level authentication
- Ability to delegate credentials to agents
- Ability to map between Grid & local identities
- Local security administration & enforcement
- Single sign-on support implemented as “proxies”
- A “plug in” framework for authorization decisions

## Message-level

- Provides message protection for SOAP messages
- Support for WS-Security standard and WS-SecureConversation specification
- Complies with WS-Interoperability Basic Security Profile
- Currently has poor performance
- Unclear if this is an implementation issue ot a specification issue

## Transport-level Security

- Authentication via TLS protocol
- (ietf spec)
- Support for X.509 proxy certificates
- Default in GT4 due to performance concerns, driven by user requirements

## GSI Functions

- Four distinct functions: message protection, authentication, delegation, and authorization
- Implementations of different standards are used to provide each of these functions:
- TLS (transport-level) or WS-Security and WS-SecureConversation (message level) used as message protection mech. in combination with SOAP
- X.509 End Entity Certificates or Username and Password are used as authentication credentials
- X.509 Proxy Certificates and WS-Trust are used for delegation
- SAML assertions are used for authorization

## GT4’s Use of Security Standards 

## Basic Grid Security Mechanisms

- Basic security mechanisms are provided as libraries/classes and APIs.
- Integrated with other GT tools and services
- Integrated with many Grid community tools and services (and applications & systems)
- A few stand-alone tools are also included.

## A Cautionary Note

- Grid security mechanisms are tedious to set up.
- If exposed to users, hand-holding is usually required.
- These mechanisms can be *hidden entirely* from end users, but still used behind the scenes.
- These mechanisms exist for good reasons.
- Many useful things can be done without Grid security.
- It is unlikely that an ambitious project could go into production operation without security like this.
- Most successful projects end up using Grid security, but using it in ways that end users don’t see much.

## GT4 Security Highlights

- Standards based support for message level and transport level security
- Transport level is default due to performance
- Standards based authorization (SAML) via Community Authorization Service (CAS) or callouts
- Stand-alone delegation service
- More authentication options
- MyProxy, simpleCA, ...

## GT4 Security

- Users

## Security Performance

- We’ve measured performance for both WS and transport security mechanisms
- See next slide for graph
- Transport security is significantly faster than WS security
- We made transport security (i.e. https) our default
- We’re working on making it even faster by using connection caching

## Slide 30

## A Model Architecture for Data Grids

**A Model Architecture for Data Grids**

- Metadata Catalog

- Replica Loc. Svc

**Tape** **Library**

**Disk Cache**

**Attribute Specification**

**Logical Collection and Logical File Name**

**Disk Array**

**Disk Cache**

- Application

- Replica
- Selection

**Multiple Locations**

- NWS

**Selected**
**Replica**

**GridFTP Control Channel**

**Performance**
**Information &**
**Predictions**

**Replica Location 1**

**Replica Location 2**

**Replica Location 3**

- MDS

**GridFTP****Data****Channel**

## GT4 Data Management

- Stage large data to/from nodes
- Replicate data for performance & reliability
- Locate data of interest
- Provide access to diverse data sources
- File systems, parallel file systems, hierarchical storage (GridFTP)
- Databases (OGSA-DAI)

## GT4 Data Functions

- Find your data: **Replica Location Service**
- Managing ~40M files in production settings
- Move/access your data: **GridFTP**, **RFT**
- High-performance striped data movement
- Couple data & execution management
- GRAM uses GridFTP and RFT for staging

## GridFTP in GT4

- 100% Globus code
- No licensing issues
- Stable, extensible
- IPv6 Support
- XIO for different transports
- Striping  multi-Gb/sec wide area transport
- Pluggable
- Front-end: e.g., future WS control channel
- Back-end: e.g., HPSS, cluster file systems
- Transfer: e.g., UDP, NetBLT transport

**Notes:**
Future: integration of provisioning
E.g., storage reservation

## GridFTP Performance

- TeraGrid Striping results
- 30Gbs network, 32 IBM ia64 nodes
- Ran varying number of stripes
- Ran both memory-to-memory and disk-to-disk

## Memory to MemoryStriping Performance

- High linear scalability (slope near 1)
- 27 Gbs on a 30 Gbs link (90% utilization) with 32 nodes

## Disk to Disk Striping Performance

- Limited by the storage system
- Achieved 17.5 Gbs

## And in conversation...

- “We think we have hit the limit of python code. The GridFTP C libraries are delivering data so fast to the buffers that the python client code cannot keep up in doing the fseek, fwrite, and then re-register the data callback. We are going to have to code our "transfer agents" entirely in C for S5.Not a bad problem to have...”
- Scott Koranda, Dept of Physics, University of Minnesota

## Reliable File Transfer:Third Party Transfer

**Reliable File Transfer:****Third Party Transfer**

- RFT Service

- RFT Client

- SOAP Messages

- Notifications(Optional)

- GridFTP Server

- GridFTP Server

- Fire-and-forget transfer
- Web services interface
- Many files & directories
- Integrated failure recovery

## RFT Performance Stats

- Current maximum request size is approx 20,000 entries with a default 64MB heap size.
- Infinite transfer - LAN
- ~120,000 transfers (servers were killed by mistake)
- Was a good test. Found a corner  case where postgres was not able to perform ~ 3 update queries / sec and was using up CPU
- Infinite transfer – WAN
-  ~67000 transfers (killed because of the same reason as above)
- Sloan Digital Sky Survey DR3 archive move
- 900+K files, 6 TB
- Killed the transfer several times for recoverability testing
- No human intervention has been required to date

## Replica Location Service

- Identify location of files via logical to physical name map
- Distributed indexing of names, fault tolerant update protocols
- GT4 version scalable & stable
- Managing ~40 million files across ~10 sites

- Index

- Index

| Local DB | Update send (secs) | Bloom filter  (secs)  | Bloom filter (bits) |
| --- | --- | --- | --- |
| 10K | <1 | 2 | 1 M |
| 1 M | 2 | 24 | 10 M |
| 5 M | 7 | 175 | 50 M |

## LIGO Use of RLS

- Some hands-on numbers
- Produce 1 TB per day
- 8 sites
- > 3 million entries in the RLS
- > 30 million files
- “This replication of data using RLS and GridFTP is enabling more gravitational wave data analysts across the world to do more science more efficiently then ever before. Globus RLS and GridFTP are in the critical path for LIGO data analysis.”

## Data Replication Service (tech preview)

- Pull “missing” files to local site

- Site B

- List of required Files

*Reliable File Transfer**Service*

*Data Replication Service*

*GridFTP*

*Local*
*Replica**Catalog*

*Replica*
*Location**Index*

- Site A

*Data Replication Service*

*Reliable File Transfer Service*

*Local*
*Replica*
*Catalog*

*Replica**Location**Index*

*GridFTP*

## OGSA-DAI

- Flexible & Composable Middleware
- Data access
- Relational & XML Databases, semi-structured files
- Data integration
- Multiple data delivery mechanisms, data translation
- Extensible & Efficient framework
- Request documents contain multiple tasks
- A task = execution of an activity
- Group work to enable efficient operation
- Extensible set of activities
- > 30 predefined, framework for writing your own
- Moves computation to data
- Pipelined and streaming evaluation
- Concurrent task evaluation

## OGSA-DAI

- Current Release = Release 5 = in GT4
- Added Installation wizards & indexed files
- >1100 registered users we know about
- Running on 3 message passing infrastructures
- Release 6 – May 2005
- Improved client side API
- Explicit control of sequential & parallel tasks
- Dynamic reconfigurability
- WS-DAI reference implementation

## The Resource Management Challenge

- Enabling secure, controlled remote access to heterogeneous computational resources and management of remote computation
- Authentication and authorization
- Resource discovery & characterization
- Reservation and allocation
- Computation monitoring and control
- Addressed by a set of protocols & services
- GRAM protocol as a basic building block
- Resource brokering & co-allocation services
- GSI for security, MDS for discovery

## Execution Management (GRAM)

- Common WS interface to schedulers
- Unix, Condor, LSF, PBS, SGE, ...
- More generally: interface for process execution management
- Lay down execution environment
- Stage data
- Monitor & manage lifecycle
- Kill it, clean up
- A basis for application-driven provisioning

## GT4 GRAM

- 2nd-generation WS implementation
-  optimized for performance, stability, scalability
- Streamlined critical path
- Use only what you need
- Flexible credential management
- Credential cache & delegation service
- GridFTP & RFT used for data operations
- Data staging & streaming output
- Eliminates redundant GASS code
- Single and multi-job support

**Notes:**
See: Karl C. on GRAM
See: Kate Keahey on dynamic workspaces and VMs
Future: towards richer execution management
Virtual machine integration, support for dynamic accounts, quality of service, provisioning

## GT4 GRAM Structure:WSRF/WSN Poster Child

- GRAM
- services

- GT4 Java Container

- GRAM
- services

- Delegation

- RFT File
- Transfer

*Transfer*
*request*

- GridFTP

- Remote
- storage
- element(s)

- Local
- scheduler

- User
- job

- Compute element

- GridFTP

- sudo

- GRAM
- adapter

*FTP*
*control*

*Local job*
*control*

*Delegate*

*FTP data*

**Client**

*Job functions*

*Delegate*

- Service host(s) and compute element(s)

**GT4 GRAM Structure:****WSRF/WSN Poster Child**

## Some of our Goals

- “GRAM should add little to no overhead compared to an underlying batch system”
- Submit as many jobs to GRAM as is possible to the underlying scheduler
- Goal - 10,000 jobs to a batch scheduler
- Goal – efficiently fill the process table for fork scheduler
- Submit/process jobs as fast to GRAM as is possible to the underlying scheduler
- Goal - 1 per second
- We are not there yet...
- A range of limiting factors at play

##  Design Decisions

- Efforts and features towards the goal
- Allow job brokers the freedom to optimize
- E.g. Condor-G is smarter than globusrun
- Protocol steps made optional and shareable
- Reduced cost for GRAM service on host
- Single WSRF host environment
- Better job status monitoring mechanisms
- More scalable/reliable file handling
- GridFTP and RFT instead of globus-url-copy
- Removal of non-scalable GASS caching
- GT4 tests performing better than GT3 did
- But more work to do

## GRAM 3.9.4 performance

- Throughput
- Test: Simple job to fork scheduler (/bin/date); no staging, streaming, or cleanup
- ~77 jobs/min sustained
- ~60 jobs/minute with delegation
- Long Running test
- Ran 500,000+ sequential jobs over 23 days
- These included staging, delegation, fork job manager

## Gram Performance (2)

- Concurrency
- Job submits to Condor scheduler (long running sleep job); no staging, streaming, or cleanup; no delegation
- Current limit is 32,000 jobs due to a Linux directory limit
- using multiple sub-directories will resolve this, look for this in 4.2

## Monitoring and Discovery Challenges

- Grid Information Service
- Requirements and characteristics
- Uniform, flexible access to information
- Scalable, efficient access to dynamic data
- Access to multiple information sources
- Decentralized maintenance
- Secure information provision

## Monitoring and Discovery

- “Every service should be monitorable and discoverable using common mechanisms”
- WSRF/WSN provides those mechanisms
- A common aggregator framework for collecting information from services, thus:
- Index Service: Registry supporting Xpath queries, with caching
- Trigger Service: perform action on condition
- Deep integration with Globus containers & services: every GT4 service is discoverable
- GRAM, RFT, GridFTP, CAS, ...

## GT4 Monitoring & Discovery

**GT4 ****Monitoring & Discovery**

- GRAM

- User

- MDS-Index

- GT4 Cont.

- RFT

- MDS-Index

- GT4 Container

- MDS-Index

- GridFTP

- adapter

- Registration &
- WSRF/WSN Access

- Custom protocols
- for non-WSRF entities

- Clients
- (e.g., WebMDS)

- Automated
- registration
- in container

- WS-ServiceGroup

**Notes:**
WebMDS: configurable via XSLT style sheets

## MDS4 Extensibility

- Aggregator framework provides
- Registration management
- Collection of information from Grid Resources
- Plug in interface for data access, collection ,query, ...
- WebMDS framework provides for customized display
- XSLT transformations

## MDS4 in 3.9.5Index Query Performance

- Examined Index with 1 or 2 clients for 10, 30, and 100 entries
- 10 entries
- 1 client: 1453 requests/min (23/sec), Avg query time 40ms
- 2 clients: 2765 requests /min (44 /sec), Avg query time 44ms
- 30 entries
- 1 client: 923 requests/min (14/sec), Avg query time 64ms
- 2 clients: 1335 requests/min (20/sec), Avg query time 93 ms
- 100 entries
- 1 client: 312 requests/min (4/sec), Avg query time 190 ms
- 2 clients: 488 requests/min (6/sec), Avg query time 265ms

## Long Running Test

- Ran 14 days (killed by accident during other testing)
- Over 94 million requests processed,
- 76 requests/sec average
- 13 millisecond average Query RTT
- Has also had diperf tests run against it (next slide)

## Slide 60

## GT4 Documentationis Much Improved!

**GT4 Documentation****is ****Much Improved!**

## The Globus Ecosystem

**Globus components** address core issues relating to resource access, monitoring, discovery, security, data movement, etc.
- GT4 being the latest version
- A larger **Globus ecosystem** of open source and proprietary components provide complementary components
- A growing list of components
- These components can be combined to produce **solutions** to Grid problems
- We’re building a list of such solutions

## Many Tools Build on, or Can Contribute to, GT4-Based Grids

- Condor-G, DAGman
- MPICH-G2
- GRMS
- Nimrod-G
- Ninf-G
- Open Grid Computing Env.
- Commodity Grid Toolkit
- GriPhyN Virtual Data System
- Virtual Data Toolkit
- GridXpert Synergy

- Platform Globus Toolkit
- VOMS
- PERMIS
- GT4IDE
- Sun Grid Engine
- PBS scheduler
- LSF scheduler
- GridBus
- TeraGrid CTSS
- NEES
- IBM Grid Toolbox
- ...

## 2005 and Beyond

- We have a solid Web services base
- We now want to build, on that base, a open source service-oriented infrastructure
- Virtualization
- New services for provisioning, data management, security, VO management
- End-user tools for application development
- Etc., etc.

## Globus and its User Community

- How can “we” best support “you”?
- We try to provide the best software we can
- We use bugzilla & other community tools
- We work to grow the set of contributors
- How can “you” best support “us”?
- Become a contributor: of software, bug fixes, answers to questions, documentation
- Provide us with success stories that can justify continued Globus development
- Promote Globus within your communities

## Working with GT4

- Download and use the software, and provide feedback
- Join **gt4friends@globus.org** mail list
- Review, critique, add to documentation
- Globus Doc Project: **http://gdp.globus.org**
- Tell us about your GT4-related tool, service, or application

## So...

- GT4 is a significant step forward in the quality, functionality and standards compliance of GT.
- Beta release available for immediate use, final April 29th
- Downloads and docs at:
- www.globustoolkit.org

- 2nd Edition
- www.mkp.com/grid2

## References

- Jennifer M. Schopf
- jms@mcs.anl.gov
- _[www.](http://www.mcs.anl.gov/~jms)__[mcs](http://www.mcs.anl.gov/~jms)__[.](http://www.mcs.anl.gov/~jms)__[anl](http://www.mcs.anl.gov/~jms)__[.](http://www.mcs.anl.gov/~jms)__[gov](http://www.mcs.anl.gov/~jms)__[/~jms](http://www.mcs.anl.gov/~jms)_
- Support from DOE, NSF, Microsoft, NeSC, JISC
- This talk:
- _[www.mcs.anl.gov/~jms/Talks](http://www.mcs.anl.gov/~jms/Talks)_ (not there yet)
- Globus Alliance:
- _[www.globus.org](http://www.globus.org/)_
- Globus Performance
- _[www-unix.globus.org/toolkit/docs/development/4.0-drafts/perf_overview.html](http://www-unix.globus.org/toolkit/docs/development/4.0-drafts/perf_overview.html)_

- Slides on how globus works

## How Globus Works

**Globus** is a distributed open source community with many contributors & users
- CVS, documentation, bugzilla, email lists
- Modular structure allows many to contribute
**Globus Alliance Board** provides governance when needed
- Meritocracy: individuals who demonstrate ongoing contributions & commitment
- Primarily: what to include, when to release
**Globus Alliance** is an informal partnership of organizations led by Board members

**Notes:**
Board members may command an army...

## Evolution of the Globus Alliance

- Argonne/U.Chicago (Childers, Foster): 1995
- USC/ISI (Kesselman): 1995
- Edinburgh (Atkinson, Parsons): 2003
- Swedish PDC (Johnsson, Mulmo): 2003
- NCSA (Welch): 2004
- Univa (Czajkowski, Tuecke): 2004
*   **Other contributors will surely be added*

## From eScience to eBusiness

- Since ~2001, growing interest in Globus for commercial use
- Enterprises, IT vendors, ISVs asking Globus leaders to address commercial needs
- But hard to do in a research laboratory
- In response, we have created two new organizations
- Globus Consortium
- Univa

## Globus Consortium(www.globusconsortium.com)

- Nonprofit organization funded by companies to advance Globus Toolkit for enterprise use
- Initial sponsor members: HP, IBM, Intel, Sun
- Initial contributors: Nortel, Univa
- First two projects already identified
- Member-driven software quality improvements
- Contributions to job submission standards
- Other projects to be defined, e.g.
- Develop new features key to enterprise use
- Education & outreach

## Provider of commercial support, services, & products around open source Globus
Commercial distribution of GT4 & beyond
Integration with enterprise systems
Committed to open source & open standards
Founded by Tuecke, Foster, Kesselman
Tuecke left Argonne to be CEO
Foster, Kesselman remain at Argonne, ISI
Experienced management team 
Rich Miller, Vas Vasiliadis, Paul Davé, Bob Mandel

- Commercial distribution of GT4 & beyond
- Integration with enterprise systems
- Committed to open source & open standards
- Founded by Tuecke, Foster, Kesselman
- Tuecke left Argonne to be CEO
- Foster, Kesselman remain at Argonne, ISI
- Experienced management team
- Rich Miller, Vas Vasiliadis, Paul Davé, Bob Mandel

## Globus and its User Community

- How can “we” best support “you”?
- We try to provide the best software we can
- We use bugzilla & other community tools
- We work to grow the set of contributors
- How can “you” best support “us”?
- Become a contributor: of software, bug fixes, answers to questions, documentation
- Provide us with success stories that can justify continued Globus development
- Promote Globus within your communities

## Working with GT4

- Download and use the software, and provide feedback
- Join **gt4friends@globus.org** mail list
- Review, critique, add to documentation
- Globus Doc Project: **http://gdp.globus.org**
- Tell us about your GT4-related tool, service, or application

## So...

- GT4 is a significant step forward in the quality, functionality and standards compliance of GT.
- Beta release available for immediate use, final April 29th
- Downloads and docs at:
- www.globustoolkit.org

- 2nd Edition
- www.mkp.com/grid2