# Presentation: 696291

## The Globus® Toolkit in Cyberinfrastructure Examples and Lessons

- The Globus® Toolkit in Cyberinfrastructure Examples and Lessons

- Lee LimingUniversity of Chicago • Argonne National Laboratory

## Outline

- Overview of e-Science Challenges
- Three Case Studies
- High-performance Data Movement for TeraGrid
- User Registration for the Earth System Grid
- Data Replication for LIGO
- General Lessons and Patterns

- For final slides (minor updates, in color) go to
- http://www.mcs.anl.gov/~liming/solutions/

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## History

- For years, a few whacky computer scientists have been trying to help other scientists use distributed computing.
- Interactive simulation (climate modeling)
- Very large-scale simulation and analysis (galaxy formation, gravity waves, battlefield simulation)
- Engineering (parameter studies, linked component models)
- Experimental data analysis (high-energy physics)
- Image and sensor analysis (astronomy, climate study, ecology)
- Online instrumentation (microscopes, x-ray devices, etc.)
- Remote visualization (climate studies, biology)
- Engineering (large-scale structural testing, chemical engineering)
- In these cases, the scientific problems are big enough that they require people in several organizations to collaborate and share computing resources, data, instruments.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## What Types of Problems?

- Your system administrators can’t agree on a uniform authentication system, but you have to allow your users to authenticate once (using a single password) then use services on all systems, with per-user accounting.
- You need to be able to offload work during peak times to systems at other companies, but the volume of work they’ll accept changes from day-to-day.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## What Types of Problems?

- You and your colleagues have 6000 datasets from the past 50 years of studies that you want to start sharing, but no one is willing to submit the data to a centrally-managed storage system or database.
- You need to run 24 experiments that each use six large-scale physical experimental facilities operating together in real time.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Two Cardinal Rules of the Grid

- You can’t rely on homogeneity.
- Impossible to achieve in the real world.
- STRATEGY - Plan on dealing with diverse systems and use mechanisms to *manage* heterogeneity.
- You can’t rely on trust.
- Severely limits participation.
- STRATEGY - Provide a security model that can *express* complicated social networks.
- STRATEGY - Use full disclosure when making requests (who is requesting, authorizing, and authenticating the request) and give service owners *and* service hosts tools to *enforce* local policies.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Challenging Applications

- The applications that Grid technology is aimed at are *not* easy applications!
- The reason these things haven’t been done before is because people believed it was too hard to bother trying.
- If you’re trying to do these things, you’d better be prepared for it to be challenging.
- Grid technologies are aimed at helping to overcome the challenges.
- They solve some of the most common problems
- They encourage standard solutions that make future interoperability easier
- They were developed as parts of real projects
- In many cases, they benefit from years of lessons from multiple applications
- Ever-improving documentation, installation, configuration, training

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Earth System Grid

- _Goal_: Give climate scientists easier access to the distributed data and resources that they require to perform their research.
- Developed new technologies for (1) creating and operating "filtering servers" capable of performing sophisticated analyses, and (2) delivering results to users.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Collaborative Engineering: NEES

- U.Nevada Reno

- www.neesgrid.org

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Laser Interferometer Gravitational Wave Observatory

- Goal: Observe gravitational waves predicted by theory.
- Three physical detectors in two locations. (Plus GEO detector in Germany.)
- Ten+ data centers for data analysis.
- Collaborators in ~40 institutions on at least three continents.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Cancer Biology

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

**Notes:**
I would mention data integration here: integration of different data types: tissue, (pre)clinical, genomics, proteomics....

## NSF’s TeraGrid*

**UCSD**

**UT**

**UC/ANL**

**NCSA**

**PSC**

**ORNL**

**PU**

**IU**

**A National Science Foundation Investment in Cyberinfrastructure**
**$100M 3-year construction (2001-2004)**
**$150M 5-year operation & enhancement ****(2005-2009)**

- NSF’s TeraGrid*

- TeraGrid DEEP: Integrating NSF’s most powerful computers (60+ TF)
- 2+ PB Online Data Storage
- National data visualization facilities
- World’s most powerful network (national footprint)
- TeraGrid WIDE Science Gateways: Engaging Scientific Communities
- 90+ Community Data Collections
- Growing set of community partnerships spanning the science community.
- Leveraging NSF ITR, NIH, DOE and other science community projects.
- Engaging peer Grid projects such as Open Science Grid in the U.S. as peer Grids in Europe and Asia-Pacific.
- Base TeraGrid Cyberinfrastructure:Persistent, Reliable, National
- Coordinated distributed computing and information environment
- Coherent User Outreach, Training, and Support
- Common, open infrastructure services

- * Slide courtesy of Ray Bair, Argonne National Laboratory

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## What End Users Need

- Secure, reliable, on-demand access to data,
- software, people, and other resources
- (ideally all via a Web Browser!)

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## How it Really Happens

- WebBrowser

- ComputeServer

- DataCatalog

- DataViewerTool

- Certificate
- authority

- ChatTool

- CredentialRepository

- WebPortal

- ComputeServer

*Resources implement standard access & management interfaces*

*Collective services aggregate &/or virtualize resources*

*Users work with client applications*

*Application services organize VOs & enable access to other services*

- Database
- service

- Database
- service

- Database
- service

- SimulationTool

- Camera

- Camera

- TelepresenceMonitor

- RegistrationService

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## How it Really Happens

- Implementations are provided by a mix of
- Application-specific code
- “Off the shelf” tools and services
- Tools and services from the Globus Toolkit
- Tools and services from the Grid community (compatible with GT)
- Glued together by...
- Application development
- System integration

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Globus Philosophy

- Globus was first established as an open source project in 1996
- The Globus Toolkit is open source to:
- Allow for inspection
- for consideration in standardization processes
- Encourage adoption
- in pursuit of ubiquity and interoperability
- Encourage contributions
- harness the expertise of the community
- The Globus Toolkit is distributed under the (BSD-style) Apache License version 2

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## dev.globus 

- Governance model based on Apache Jakarta
- Consensus based decision making
- Globus software is organized as several dozen “Globus Projects”
- Each project has its own “Committers” responsible for their products
- Cross-project coordination through shared interactions and committers meetings
- A “Globus Management Committee”
- Overall guidance and conflict resolution

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## http://dev.globus.org

- http://**dev.globus**.org

- Guidelines(Apache
- Jakarta)
- Infrastructure(CVS, email,bugzilla, Wiki)
- Projects
- Include
**...**

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Open Source != “Free time”

- Globus development is well-funded.
- The open source model facilitates contributions.
- NSF and DOE sponsor Globus development at several institutions via multiple grants, totaling >$5M/yr.
- Non-U.S. science agencies also contribute to Globus development.
- Corporations also sponsor developers.
- NSF explicitly funds Globus improvements.
- CDIGS: Community-Driven Improvements to Globus Software

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Globus Technology Areas

- Core runtime
- Infrastructure for building new services
- Security
- Apply uniform policy across distinct systems
- Execution management
- Provision, deploy, & manage services
- Data management
- Discover, transfer, & access large data
- Monitoring
- Discover & monitor dynamic services

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Globus Software: dev.globus.org

**Projects**

**Security**

**Execution**
**Mgmt**

**Info**
**Services**

**Common**
**Runtime**

**Globus Projects**

**Other**

**MPICH-**
**G2**

**GridWay**

**Data Mgmt**

**Incubator**
**Mgmt**

**Cog WF**

**LRMA**

**GAARDS**

**OGRO**

**GDTE**

**UGP**

**HOC-SA**

**PURSE**

**GridShib**

**Introduce**

**Dyn Acct**

**WEEP**

**Gavia JSC**

**Gavia MS**

**DDM**

**Virt WkSp**

**SGGC**

**Metrics**

**ServMark**

**GridFTP**

**Reliable**
**File**
**Transfer**

**OGSA-DAI**

**GRAM**

**MDS4**

**CAS**

**Data**
**Rep**

**Delegation**

**Replica**
**Location**

**Java ****Runtime**

**C ****Runtime**

**Python ****Runtime**

**GT4**

**C Sec**

**GT4 Docs**

**MEDICUS**

**GSI-**
**OpenSSH**

**MyProxy**

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## What Is the Globus Toolkit?

- The Globus Toolkit is a collection of solutions to problems that frequently come up when trying to build collaborative distributed applications.
- Heterogeneity
- To date (v1.0 - v4.0), the Toolkit has focused on *simplifying heterogenity* for application developers.
- We are increasingly including more “vertical solutions” that implement typical application patterns.
- Security
- The Grid Security Infrastructure (GSI) allows collaborators to share resources without blind trust.
- Standards
- Our goal has been to capitalize on and encourage use of existing standards (IETF, W3C, OASIS, GGF).
- The Toolkit also includes reference implementations of new/proposed standards in these organizations.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## What’s In the Globus Toolkit?

- A Grid development environment
- Develop new OGSA-compliant Web Services
- Develop applications using Java or C/C++ Grid APIs
- Secure applications using basic security mechanisms
- A set of basic Grid services
- Job submission/management
- File transfer (individual, queued)
- Database access
- Data management (replication, metadata)
- Monitoring/Indexing system information
- Tools and Examples
- The prerequisites for many Grid community tools

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Leveraging Existingand Proposed Standards

- SSL/TLS v1 (from OpenSSL) (IETF)
- LDAP v3 (from OpenLDAP) (IETF)
- X.509 Proxy Certificates (IETF)
- GridFTP v1.0 (GGF)
- OGSI v1.0 (GGF)
- WSRF (OASIS)
- And others on the road to standardization:
- DAI, WS-Agreement, WSDL 2.0, WSDM, SAML, XACML

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Areas of Competence

- “Connectivity Layer” Solutions
- Service Management (WS Core)
- Monitoring/Discovery (WS Core)
- Security (GSI and WS-Security)
- Communication (XIO)
- “Resource Layer” Solutions
- Computing / Processing Power (GRAM)
- Data Access/Movement (GridFTP, OGSA-DAI)
- In development: Telecontrol (GTCP)
- “Collective Layer” Solutions
- Data Management (RLS, DRS, RFT, OGSA-DAI)
- Monitoring/Discovery (Index, Trigger, Archiver services)
- Security (CAS, MyProxy)

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## How To Use the Globus Toolkit

- By itself, the Toolkit has surprisingly limited *end user* value.
- There’s very little user interface material there.
- You can’t just give it to end users (scientists, engineers, marketing specialists) and tell them to do something useful!
- The Globus Toolkit is useful to *application developers* and *system integrators*.
- You’ll need to have a specific *application or system* in mind.
- You’ll need to have the right *expertise*.
- You’ll need to set up prerequisite *hardware/software*.
- You’ll need to have a *plan*.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## An “Ecosystem” of Grid Software

- There isn’t a Grid software kit for everybody (yet).
- Varying requirements
- Experimentation and learning
- Reluctance to invest in a “static” solution
- There _are_ many tools that work well together.
- Results of successful projects
- Reusable solutions
- Implication: Integrate it yourself (for now).
- Provides considerable flexibility
- Requires expertise and effort
- Reminder: *These are ambitious applications!*

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Methodology

- Building a Grid system or application is _currently_ an exercise in software *integration*.
- Define user requirements
- Derive system requirements or features
- Survey existing components
- Identify useful components
- Develop components to fit into the gaps
- Integrate the system
- Deploy and test the system
- Maintain the system during its operation
- This should be done iteratively, with many loops and eddies in the flow.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Globus User Community

- Large & diverse
- 10s of national Grids, 100s of applications, 1000s of users; probably much more
- Every continent except Antarctica
- Applications ranging across many sciences
- Dozens (at least) of commercial deployments
- Successful
- Many production systems doing real work
- Many applications producing real results
- Smart, energetic, demanding
- Constant stream of new use cases & tools

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## GlobalCommunity

- GlobalCommunity

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Examples ofProduction Scientific Grids

- Examples ofProduction Scientific Grids

- APAC (Australia)
- China Grid
- China National Grid
- DGrid (Germany)
- EGEE
- NAREGI (Japan)
- Open Science Grid
- Taiwan Grid
- TeraGrid
- ThaiGrid
- UK Nat’l Grid Service

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## The Importance of Community

- All Grid technology is evolving rapidly.
- Web services standards
- Grid interfaces
- Grid implementations
- Grid hosting services (ASP, SSP, etc.)
- Community is important!
- Best practices (GGF, OASIS, etc.)
- Open source (Linux, Axis, Globus, etc.)
- Application of community standards is *vital*.
- Increases leverage
- Mitigates (a bit) effects of rapid evolution
- Paves the way for future integration/partnership

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Case Study 1:High-Performance Data Movement for TeraGrid

## NSF’s TeraGrid*

**UCSD**

**UT**

**UC/ANL**

**NCSA**

**PSC**

**ORNL**

**PU**

**IU**

**A National Science Foundation Investment in Cyberinfrastructure**
**$100M 3-year construction (2001-2004)**
**$150M 5-year operation & enhancement ****(2005-2009)**

- NSF’s TeraGrid*

- TeraGrid DEEP: Integrating NSF’s most powerful computers (60+ TF)
- 2+ PB Online Data Storage
- National data visualization facilities
- World’s most powerful network (national footprint)
- TeraGrid WIDE Science Gateways: Engaging Scientific Communities
- 90+ Community Data Collections
- Growing set of community partnerships spanning the science community.
- Leveraging NSF ITR, NIH, DOE and other science community projects.
- Engaging peer Grid projects such as Open Science Grid in the U.S. as peer Grids in Europe and Asia-Pacific.
- Base TeraGrid Cyberinfrastructure:Persistent, Reliable, National
- Coordinated distributed computing and information environment
- Coherent User Outreach, Training, and Support
- Common, open infrastructure services

- * Slide courtesy of Ray Bair, Argonne National Laboratory

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## TeraGrid User Priorities*

**Remote File Read/Write**

**High-Performance File Transfer**

**Coupled Applications, Co-scheduling**

**Advanced Reservations**

**Grid Portal Toolkits**

**Grid Workflow Tools**

**Batch Metascheduling**

**Global File System**

**Client-Side Computing Tools**

**Batch Scheduled Parameter Sweep Tools**

- Partners in Need
- (breadth of need)

- Overall Score
- (depth of need)

- * Slide courtesy of Charlie Catlett, TeraGrid Project Director

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

**Notes:**
Results of in-depth discussions with 16 TeraGrid user teams during first annual user survey (August 2004).
Ordinate bars=5;
Partners in need shows need within first 24 months

## Data Transfer on TeraGrid

- TeraGrid encourages sites to offer unique resources that address different needs.
- Because of this, users may need to move data from one TeraGrid system to another.
- Data transfer must be made as efficient as possible, especially from a “person time” perspective.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## The Challenge

- Early TeraGrid users were disappointed with data transfer rates between sites.
- 270 Mbps over a 30 Gbps link
- Requires knowledge of server and network configuration
- High-performance tools (e.g., globus-url-copy) not as friendly as low-performance tools (scp)
- For some users, managing transfers was also an issue that consumed too much “person time.”
- CHALLENGE: Provide an easy-to-use tool that provides high-performance data transfer when using default settings, reducing human supervision time as much as possible.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Issues - Technical

- Networking hardware
- Ordinary software tools (scp) use single TCP streams, which can’t consume the bandwidth of a NIC (1 Gbps).
- Most TeraGrid NICs are either 100 Mbps or 1 Gbps, with very few 2 Gbps. These can’t consume bandwidth of a 30 Gbps link.
- REQUIREMENT: The tool must employ both parallelism (within a host) and striping (multiple hosts).
- REQUIREMENT: Hosts must be provisioned.
- Filesystems
- Local filesystem performance (disk I/O, local connectivity) is also a limiting factor.
- REQUIREMENT: Parallel filesystems are needed.
- Complexity
- Specialized storage systems at sites may provide unique interfaces. This increases application/user complexity.
- REQUIREMENT: Provide a uniform interface to storage systems (“virtualization”) wherever possible.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Issues - Social (1)

- Who has information?
- User/application knows about local path and remote path. System administrators know about servers and network configuration/properties.
- REQUIREMENT: The tool should get details about servers and network parameters to be used in transfers from administrators, not users.
- How do expectations get set?
- Users know about published specs, not historical experience (30 Gbps, not 300 Mbps). This raises expectations far beyond reality.
- REQUIREMENT: The solution should include a way to capture historical experience and publish it to users.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Issues - Social (2)

- How many transfers are typical?
- Big files may require restarts to complete. Large numbers of small files must be “shepherded.”
- REQUIREMENT: The tool should provide an automated management capability to reduce user “shepherding.”
- Security (Authentication/Authorization)
- Users typically have different accounts at different sites. (This is true for “application” accounts as well.) It’s hard to keep track of this information.
- REQUIREMENT: The tool should not require users or applications to remember their various local accounts at each site.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Requirements Summary

- The tool must employ both parallelism (within a host) and striping (multiple hosts).
- Hosts must be provisioned.
- Parallel filesystems are needed.
- Provide a uniform interface to storage systems (“virtualization”) wherever possible.
- The tool should get details about servers and network parameters to be used in transfers from administrators, not users.
- The solution should include a way to capture historical experience and publish it to users.
- The tool should provide an automated management capability to reduce user “shepherding.”
- The tool should not require users or applications to remember their various local accounts at each site.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Basic Grid Security Mechanisms

- Basic Grid authentication and authorization mechanisms come in two flavors.
- Pre-Web services
- Web services
- Both are included in the Globus Toolkit, and both provide vital security features.
- Grid-wide identities implemented as PKI certificates
- Transport-level and message-level authentication
- Ability to delegate credentials to agents
- Ability to map between Grid & local identities
- Local security administration & enforcement
- Single sign-on support implemented as “proxies”
- A “plug in” framework for authorization decisions

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Basic Grid Security Mechanisms

- Basic security mechanisms are provided as libraries/classes and APIs.
- Integrated with other GT tools and services
- Integrated with many Grid community tools and services (and applications & systems)
- A few stand-alone tools are also included.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## A Cautionary Note

- Grid security mechanisms are tedious to set up.
- If exposed to users, hand-holding is usually required.
- These mechanisms can be *hidden entirely* from end users, but still used behind the scenes.
- These mechanisms exist for good reasons.
- Many useful things can be done without Grid security.
- It is unlikely that an ambitious project could go into production operation without security like this.
- Most successful projects end up using Grid security, but using it in ways that end users don’t see much.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## GridFTP

- A high-performance, secure data transfer service optimized for high-bandwidth wide-area networks
- FTP with extensions
- Uses basic Grid security (control and data channels)
- Multiple data channels for parallel transfers
- Partial file transfers
- Third-party (direct server-to-server) transfers
- GGF recommendation GFD.20

- Basic Transfer
- One control channel, several parallel data channels

- Third-party
- Transfer
- Control channels to each server, several parallel data channels between servers

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Striped GridFTP

- GridFTP supports a striped (multi-node) configuration.
- Establish control channel with one node
- Coordinate data channels on multiple nodes
- Allows use of many NICs in a single transfer
- Requires shared/parallel filesystem on all nodes.
- On high-performance WANs, aggregate performance is limited by filesystem data rates.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## globus-url-copy

- Command-line client for GridFTP servers
- Text interface
- No “interactive shell” (single command per invocation)
- Many features
- Grid security, including data channel(s)
- HTTP, FTP, GridFTP
- Server-to-server transfers
- Subdirectory transfers and lists of transfers
- Multiple parallel data channels
- TCP tuning parameters
- Retry parameters
- Transfer status output

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## RFT - File Transfer Queuing

- A WSRF service for queuing file transfer requests
- Server-to-server transfers
- Checkpointing for restarts
- Database back-end for failovers
- Allows clients to request transfers and then “disappear”
- No need to manage the transfer
- Status monitoring available if desired

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Proposed Solution

- Allocate a set of nodes at each site to serve collectively as a striped GridFTP server.
- Access to shared filesystems
- Ideally, fast NICs
- Provide a command that offers a simple, scp-like interface.
- Use administrator-provided configuration data to redirect to striped servers when possible and add optimized network parameters based on endpoints.
- Use globus-url-copy and rft clients to perform the transfers.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## TGCP - TeraGrid Copy

- Applies a set of transformation rules to source and destination.
- Local admin supplies the rules.
- Adds host/port and appropriate path information, puts into GridFTP URL format.
- When source/dest sites are identified, add network tuning parameters based on a table maintained by administrators.
- Invoke either g-u-c or rft to perform the transfer. Pass through any command line options.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## TGCP User Interface

- SCP-style source and destination
- host:path
- Two options
- -big - Use striped transfer
- -rft - Manage the transfer
- If source is a directory, RFT will transfer the full contents of the directory.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Scenario #1 -Non-shared Local Filesystem

- If the local GridFTP servers can’t get to the local file, tgcp uses g-u-c as a GridFTP client.
- Parallelism and network tuning parameters help to optimize transfer.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Scenario #2 -High-Performance Transfer

- If the file is accessible to servers at both ends, tgcp can invoke a third party transfer.
- Striping can be performed as well as parallelism.
- Servers typically have high-performance NICs.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Scenario #3 -Managed High-perf. Transfer

- If there are many files to be transferred (e.g., a directory) or the user wants to “fire and forget”, tgcp uses rft.
- Requests stored in a persistent database.
- Failure recovery
- RFT can use channel caching for even better performance with small files.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Deployment and Monitoring

- TGCP team produced a TGCP “package” for sites to deploy locally.
- Documentation prepared for users.
- GT4 GridFTP, g-u-c, and tgcp deployed at all eight TeraGrid sites in “experimental” mode.
- Non-default ports
- New hosts
- Softenv keys for user paths
- Speed runs are conducted roughly weekly with large files, and results shared.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Better File Transfer Rates  (MByte/s)*

- * Slide courtesy of Ray Bair, data produced by Jaebum Kim, NCSA, Sept. 2005

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Results - TeraGrid

- TGCP is available, deployed, and monitored.
- Not quite x10 performance improvement
- ~270 Mbps -> ~1.8 Gbps
- Much simpler to use (don’t need to specify parameters or shepherd as much)
- GT4 GridFTP and RFT are deployed.
- Compatibility with other grid systems
- Better support
- Forged relationship with NMI/GRIDS.
- Teams have worked together on something real.
- Remaining work is in provisioning hosts, improving local filesystem performance.
- TGCP will be able to use whatever the system offers.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## A Few TGCP Experiences

- Software is just the beginning.
- TGCP removed the “tooling” bottleneck; new bottlenecks have appeared.
- Performance is limited (now) by hardware provisioning at each site: servers, NICs, and local filesystems. *Dynamic provisioning may be key to solving this problem.*
- Documentation is critical.
- Many TeraGrid users still don’t know about TGCP because TeraGrid’s documentation is complex.
- Data is an inter-grid problem.
- Many users need high performance data movement between TeraGrid and other large systems (OSG, UK NGS, NERSC, etc.).
- A still more general solution is needed.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Case Study 2:User Registration for the Earth System Grid

## The Earth System Grid

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## ESG Project Goals

- Improve productivity/capability for the simulation and data management team (data producers).
- Improve productivity/capability for the research community in analyzing and visualizing results (data consumers).
- Enable broad multidisciplinary communities to access simulation results (end users).
- The community needs an integrated “cyberinfrastructure” to enable smooth *workflow* for *knowledge development*: compute platforms,  collaboration & collaboratories, data management, access, distribution, and analysis.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## The Challenge

- ESG is a distributed system that genuinely requires Grid-style distributed authentication.
- ESG is used by scientists who don’t need to be bothered with certificates.
- CHALLENGE: Provide Grid security for the system but do it in such a way that end users don’t have to manage certificates themselves.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Issues - Social

- Ease of Use
- ESG users shouldn’t have to manage their own certificates.
- It’s too complicated, intrusive.
- They don’t do it well (securely).
- Support
- Certificate management generates a lot of user support work.
- Use cases
- Most ESG users are data “readers”, not “writers.”
- Data producers and project funders want to know who the users are (registration), but access control among registered users is not a major requirement.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Issues - Technical

- Distributed System
- ESG has four major data centers, each with its own security system.
- Users should not have to keep track of four sets of credentials and know when to use each.
- The ESG web portal needs users’ credentials to perform work on their behalf, so a secure mechanism for doing that is important.
- Integration_x0003_
- ESG uses GridFTP, RLS, OpenDAPg, and GRAM to meet other system requirements, so GSI has to be supported.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## MyProxy

- MyProxy is a remote service that stores user credentials.
- Users can request proxies for local use on any system on the network.
- Web Portals can request user proxies for use with back-end Grid services.
- Grid administrators can pre-load credentials in the server for users to retrieve when needed.
- Greatly simplifies certificate management!

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Simple CA

- A convenient method of setting up a certificate authority (CA).
- The Certificate Authority can then be used to issue certificates for users and services that work with GSI and WS-Security.
- Simple CA is intended for operators of small Grid testing environments and users who are not part of a larger Grid.
- Most production Grids will not accept certificates that are not signed by a well-known CA, so the certificates generated by Simple CA will usually not be sufficient to gain access to production services.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Scenario 1 -User Registration

- The user fills out the registration web page, establishes an ID/password, and the information is stored in a database.
- The administrator is sent email.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Scenario 2 -Administrator Approval

- Administrator visits the registration website and retrieves the registration data.
- If the administrator approves the request, PURSE uses SimpleCA to generate a certificate and stores it in MyProxy.
- The user is sent email.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Scenario 3 - User Login

- The user logs into the application website using the ID/password established during registration.
- The application obtains a proxy using MyProxy.
- The application uses the proxy to authenticate to Grid services.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Sample email messages

**(a) Email confirmation step: message sent to user**
**Date: Thu, 1 Jul 2004 14:25:47 -0600 (MDT) ****From: esgport@ucar.edu ****To: john_smart@ucar.edu****Subject: ESG Registration **
**The Earth System Grid (ESG) Portal received a request for a new user account that uses your email address. Click on the link below to confirm your request (NOTE: you will not be able to login until you receive an email from the portal administrator indicating your request has been approved): **
- _[http://www.](http://www.earthsystemgrid.org/security/confirmRequest.do?token=000000fd-7c62-605c-ffffdea0-766ad9819840)__[earthsystemgrid](http://www.earthsystemgrid.org/security/confirmRequest.do?token=000000fd-7c62-605c-ffffdea0-766ad9819840)__[.](http://www.earthsystemgrid.org/security/confirmRequest.do?token=000000fd-7c62-605c-ffffdea0-766ad9819840)__[org/security/confirmRequest](http://www.earthsystemgrid.org/security/confirmRequest.do?token=000000fd-7c62-605c-ffffdea0-766ad9819840)__[.do?token=000000fd-7c62-605c-ffffdea0-766ad9819840](http://www.earthsystemgrid.org/security/confirmRequest.do?token=000000fd-7c62-605c-ffffdea0-766ad9819840)_
**If you did not request this account, please inform us at esg-admin@earthsystemgrid.org.**
**Thank you,**
**ESG System Administrator**

**(b) Email sent to CA operator for approval**
**From: esgport@ucar.edu ****Date: July 1, 2004 12:17:07 AM MDT ****To: esg-ca@ucar.edu ****Subject: ESG Registration ****A request has been made for user account on the ESG Portal. You may access the details of the request by clicking on the following link. **_[http://www.earthsystemgrid.org/administration/accountRequestData.do?token](http://www.earthsystemgrid.org/administration/accountRequestData.do?token=000000fd-2e0e-5d33-00006ac0-8387f64897be)__[=000000fd-2e0e-5d33-00006ac0-8387f64897be](http://www.earthsystemgrid.org/administration/accountRequestData.do?token=000000fd-2e0e-5d33-00006ac0-8387f64897be)_

**Customizable**

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## RA/CA Form

**Customizable**

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Results - ESG

- Four data centers (LBNL, LLNL, NCAR, ORNL)
- 700 registered users by May 2005, 2500 users in 2006, ~4000 now
- Four major datasets are available, with associated code and metadata
- Datasets added as they are produced
- >200 journal articles published 2005-2006 from analyses of data delivered by the ESG

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Results - Science

- ESG allows ~4000 people to work with climate model datasets.
- PURSE is available from dev.Globus
- Generic version for re-use
- Includes portlet code developed by OGCE
- Allows users to import existing credentials
- Supported by dev.Globus PURSE incubator project, with funding from NSF (CDIGS, OGCE)
- Used in ESG, NVO, SWEGrid
- GAMA is available from SDSC.
- Portlet implementation hosted by GridSphere
- Allows sharing by multiple portal applications
- Currently used by GEON and BIRN projects

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## A Few PURSE Lessons

- It is possible (and desirable) to hide Grid security from users.
- Online repositories are one way to do this.
- Others options include online CAs (e.g., KCA and KX.509).
- Requirements and use cases are important.
- Need to know exactly what the community concerns are: what needs to be protected.
- Need to clearly identify roles.
- Generalizing to PURSE was not trivial.
- New requirements (e.g., credential import)
- Documentation and usability testing
- Community support was essential.
- Addition of JSR-168-compliant portlets by OGCE made a big difference in usability.
- Broader community of supporters.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Case Study 3:Data Replication for LIGO

## Laser Interferometer Gravitational Wave Observatory

- Goal: Observe gravitational waves predicted by theory.
- Three physical detectors in two locations. (Plus GEO detector in Germany.)
- Ten+ data centers for data analysis.
- Collaborators in ~40 institutions on at least three continents.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## LIGO by the Numbers

- LIGO records thousands of channels, generating approx. 1 TB per day of data during a detection run.
- Data is published and data centers subscribe to portions that local users want for analysis or for local storage.
- Data analysis results in derived data.
- Currently ~30% of all LIGO data.
- This also is published and replicated.
- Currently over 30 million files in LDR network.*
- 6+ million unique logical files
- 30+ million physical copies of those files

- * Scott Koranda, UW-Milwaukee, April 2005

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## The Challenge

- Replicate 1 TB/day of data to 10+ international sites.
- Publish/subscribe (pull) model
- Provide scientists with the means to specify and discover data based on application criteria (metadata)
- Provide scientists with the means to locate copies of data

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Issues - Technical

- Efficiency
- Avoid unused bandwidth while data transfers are taking place, esp. on high-bandwidth links (10+ Gbps).
- Avoid idle time on the network between transfers.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Issues - Social (1)

- Workflow
- The publish/subscribe model matches how scientists think about their use of the data.
- Use of metadata is critical.
- Security
- Authenticate endpoints of data transfers to prevent unauthorized data manipulation.
- Heterogeneity
- Can’t tell data centers what storage systems to use.
- Can’t get everyone to do accounts the same way.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Issues - Social (2)

- Maintenance
- LIGO is focused on science, not IT. Ideally, they would not have to build or maintain data management software.
- GOAL: If software must be produced, keep it simple and do it in a way that non-CS people can understand.
- GOAL: Produce a solution that others will adopt and maintain for them.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## RLS - Replica Location Service

- A distributed system for tracking replicated data
- Consistent local state maintained in Local Replica  Catalogs (LRCs)
- Collective state with relaxed consistency maintained  in Replica Location Indices (RLIs)
- Performance features
- Soft state maintenance of RLI state
- Compression of state updates
- Membership and partitioning information maintenance

- Simple Hierarchy
- The most basic
- deployment of RLS

- Fully Connected
- High availability of
- the data at all sites

- Tiered Hierarchy
- For very large
- systems and/or very
- Large collections

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## pyGlobus*

- High-level, object-oriented interface in Python to GT4 Pre-WS APIs.
- GSI security
- GridFTP
- GRAM
- XIO
- GASS
- MyProxy
- RLS

- * pyGlobus contributed to GT4 by the Distributed Computing Department of LBNL.

- Also includes tools and services
- GridFTP server
- GridFTP GUI client
- Other GT4 clients

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Lightweight Data Replicator*

- Ties together 3 basic Grid services:
- Metadata Service
- info about files such as size, md5, GPS time, ...
- sets of interesting metadata propagate
- answers question “What files or data are available?”
- Globus Replica Location Service (RLS)
- catalog service maps filenames to URLs
- also maps filenames to sites
- answers question “Where are the files?”
- GridFTP Service
- server and customized, tightly integrated client
- use to actually replicate files from site to site
- answers question “How do we move the files?”

- * Slide courtesy of Scott Koranda, UW-Milwaukee

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## LDR Architecture

- Each site has its own machinery for pulling data needed to satisfy local users.
- Scheduler daemon queries metadata and replica catalogs to identify missing files, which it records in a prioritized list.
- Transfer daemon manages transfers for files on the list, and updates LRC when files arrive.
- If a transfer fails, the transfer daemon simply re-adds the file to the list for future transfers.
- Daemons and metadata service written in Python.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Results - LIGO*

- LIGO/GEO S4 science run completed March
- Replicated over 30 TB in 30 days
- Mean time between failure now one month (outside of CIT)
- Over 30 million files in LDR network
- 6+ million unique LFNs in RLS network
- 30+ million PFNs for those LFNs
- Performance currently appears to be limited by Python performance.
- Partnership with Globus Alliance members is leading to LDR-like tools being added to Globus Toolkit.

- * Slide courtesy of Scott Koranda, UW-Milwaukee

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Results - GT and e-Science

- Data Replication Service (DRS)
- Reimplementation of LDR’s subscription capabilities.
- Uses WSRF-compliant services written in Java.
- Works with RLS and RFT services in GT4.
- Tech preview in GT4.
- LIGO evaluating as a replacement for LDR’s subscription system.
- Remaining piece is metadata service.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## A Few LDR/DRS Lessons

- Understanding performance requirements is important.
- Performance is complicated. Requirements must be expressed very specifically and must be thoroughly informed by use.
- LIGO’s operational use was specific enough to give the RLS team good requirements.
- Reliability doesn’t have to be sophisticated.
- LDR’s strategy for recovering from a transfer failure is simply to put the transfer request back on the queue.
- Standard interfaces are important.
- LIGO was not successful at getting other projects to rally around LDR’s non-standard interfaces.
- DRS *might* overcome that hurdle by using WSRF interfaces.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## General Lessons and Patterns

## Lesson 1 - It’s All About People

- Collaborations are complicated.
- To support collaboration, IT needs to be flexible and adaptive to human needs.
- Security
- Service provisioning and management
- Resource allocation
- Heterogeneity
- IT should not make assumptions to simplify things and then expect users to behave according to the assumptions.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Lesson 1 (cont.)

- Most IT is currently not very flexible or adaptable.
- Networks are statically configured.
- Compute systems are all different, even after you get your job launched.
- Storage management has a long way to go; e.g., reservation, reconfiguration.
- Grid technology is aimed at this problem.
- Extends existing Web and Web services capabilities
- Add a layer of virtualization to deal with heterogeneity
- Add a layer of management to deal with provisioning and adaptation

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Lesson 2 -Atomicizing is Important

- All three case studies involved assembling components to create a unified solution.
- If components are not atomicized first, they can’t be reassembled to do something new.
- Very few problems are universal. The “typical” problems can be hard to isolate.

- “Sometimes through heroism you can make something work.  However, understanding why it worked, abstracting it, making it a primitive is the key to getting to the next order of magnitude of scale.”       – Robert Calderbank          Princeton, AT&T Labs

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Lesson 3 -Requirements and Use Cases

- When solving “people problems” with IT, it’s critical to understand the problem.
- Requirements - What are the criteria for success?
- Use Cases - Does this really make sense to everyone?
- Tailoring a solution to an existing system or organization is not trivial.
- If things have been designed well (both the solution and the existing system), it can be much, much easier.
- Requirements and use cases can help identify good places to interface.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Lesson 4 -This Isn’t New (But It Really Is)

- We’ve known how to solve these types of problems in general for a while now.
- Abstraction
- Virtualization
- Plug-in interfaces
- Modularity
- Parallelism
- Striping
- Virtual machines
- Encapsulation
- Service interfaces
- Notification
- Soft-state registration

- The trick is to apply them in the right places...
- Security
- Execution
- Storage
- Network
- ...to get the desired behaviors...
- Provisioning
- Authorization
- Management
- ...at the right levels.
- Department
- Institution
- Agency
- Virtual Organization

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Lesson 5 -A Grid is a Set of Capabilities

- Our “solutions” are capabilities.
- Data transfer (TGCP)
- User registration (PURSE)
- Data replication (DRS)
- Because of the way they are designed, they each form part of a Grid in the deepest sense.

- “A Grid is a system that coordinates resources that are not subject to centralized control, using standard, open, general-purpose protocols and interfaces to deliver nontrivial qualities of service.”
-            – Ian Foster              U. of Chicago,              Argonne National Lab

*TeraGrid, for example, recently redefined its software stack **based on a set of ten capability “kits.”*

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Lesson 6 - Metadata is a Different Problem

- Communities that share data need metadata standards.
- This is often overlooked in project planning, but it is essential and non-trivial.
- Metadata is as much a social challenge as a technical one.
- Requires different expertise, including a heavy dose of expertise and experience in the problem domain(s).
- Don’t expect your IT team to solve this one for you.
- Data curation is a job description, not a software feature.

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Pattern 1 - Iterative Design

- Design

- Deploy

- Build

- Apply

- Analyze

- Apply

- Apply

- Deploy

- Apply

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Pattern 2 - One Step at a Time

- TGCP had four main  stages of completion.
- Each stage provided something useful.
- Each stage fixed one key limitation.
- A clean, modular design allows modules to be improved independently.
- Knowing when to refactor (change the overall design) is a valuable skill.

- “Change one thing at a time. And at every step, change the smallest thing possible. Then test everything before proceeding.”
-           – Miron Livny             U. of Wisconsin–             Madison

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007

## Conclusions

- The Grid is still a work in progress.
- It’s only as we try to build Grids that we learn what capabilities need to be supported.
- Abstracting and making those capabilities primitive is hard work.
- A growing suite of tools is available for use.
- http://www.globus.org/grid_software/
- We have examples of successes.
- http://www.globus.org/solutions/

- The Globus Toolkit in Cyberinfrastructure

- VOiG June 2007