# Presentation: 526199

## Grid Computing withthe Globus Toolkit® 2.2

- Jennifer Schopf
- for the
- The Globus ProjectTMArgonne National LaboratoryUSC Information Sciences Institute
- www.globus.org

## Overview

- Introduction to Grids
- Why Grids and Globus
- Some definitions
- The Globus Toolkit Core Services
- Grid security infrastructure
- Resource management
- Information infrastructure
- Data management services
- Recap and conclusions

- Globus Toolkit® 2.2

## Grid Scenarios

- A biochemist exploits 10,000 computers to screen 100,000 compounds in an hour
- 1,000 physicists worldwide pool resources for peta-op analyses of petabytes of data
- An emergency response team couples real time data, weather model, population data
- Engineers at a multinational company collaborate on the design of a new product
- An application service provider offloads excess load to a compute cycle provider

- Globus Toolkit® 2.2

## The Grid Problem

**The Grid Problem**

*   *Resource sharing & coordinated problem solving in dynamic, multi-institutional settings

- Globus Toolkit® 2.2

## The Fundamental Concept

-    Enable communities (“virtual organizations”) to share geographically distributed resources as they pursue common goals—in the absence of central control, omniscience, trust relationships

- Globus Toolkit® 2.2

## Globus Toolkit

- Globus Toolkit is the source of many of the protocols described in “Grid architecture”
- Adopted by almost all major Grid projects worldwide as a source of infrastructure
- Open source, open architecture framework encourages community development
- Active R&D program continues to move technology forward
- Developers at ANL, USC/ISI, NCSA, LBNL, and other institutions

- www.globus.org

- Globus Toolkit® 2.2

## The Globus Toolkit

- Tools enabling resource sharing
- GSI (Grid Security Infrastructure)
- Authentication based on Grid-wide credential
- Single sign-on, delegation
- Authorization
- GRAM (Grid Resource Allocation and Management)
- Tool for remote job and resource management
- MDS (Monitoring and Discovery Service)
- Grid-wide information on the state of resources
- Data Services
- GASS
- GridFTP
- Replica Management
- Protocols and APIs

- Globus Toolkit® 2.2

## Globus Applications and Deployments

- Application projects include
- GriPhyN, PPDG, NEES, EU DataGrid, ESG, Fusion Collaboratory, etc., etc.
- Infrastructure deployments include
- DISCOM, NASA IPG, NSF TeraGrid, DOE Science Grid, EU DataGrid, etc., etc.
- UK Grid Center, U.S. GRIDS Center
- Technology projects include
- Data Grids, Access Grid, Portals, CORBA, MPICH-G2, Condor-G, GrADS, etc., etc.

- Globus Toolkit® 2.2

## Grid Communities & Applications:Data Grids for High Energy Physics

- Image courtesy Harvey Newman, Caltech

- Globus Toolkit® 2.2

## Globus Toolkit v2.2

- GT2.0 released in April 2002
- GridFTP & replica management additions
- Repackaged
- GRAM reliability improvements
- Numerous small changes and bug fixes
- We follow the Linux versioning model
- Even number releases are stable releases
- Odd number releases are “experimental”
- This talk covers v2.2

- Globus Toolkit® 2.2

**Notes:**
Update This is 2.2

## Some Important Definitions

- Resource
- Network protocol
- Application Programmer Interface (API)

- Globus Toolkit® 2.2

**Notes:**
Primary point of these definitions is to establish the relationship between protocol and service. A protocol establishes a specific methods by which two entities can interact with on another. 
Service The service is a process or system providing a facility to the network [RFC 2165]. Protocol defines how user requests service to provide a facility.

Two entities may interact via a protocol, but neither provides a “facility” (and hence service).  [In RFC 2165, facility = capability.]

E.g.,
TCP is a protocol, but not all programs that speak TCP implement services.
GlobusIO uses SSL for authentication, but may not implement a service.

## Resource

- An entity that is to be shared
- E.g., computers, storage, data, software
- Does not have to be a physical entity
- E.g., Condor pool, distributed file system, ...
- Defined in terms of interfaces, not devices
- E.g. scheduler such as LSF and PBS define a compute resource
- Open/close/read/write define access to a distributed file system, e.g. NFS, AFS, DFS

- Globus Toolkit® 2.2

**Notes:**
A resource is the most basic entity in the Grid, it is the thing that is being shared via Grid protocols.  A resource does not have to be a physical entity.  In some instances it may be logical: like a Condor pool, or a local resource manager for a cluster, or a distributed files system, such as NFS.  As such, from the perspective of the Grid, a resource can be defined by either the APIs or protocols used to manipulate that resource (create a process, transmit data, access data in a file, etc). 

## Network Protocol

- A formal description of message formats and a set of rules for message exchange
- Rules may define sequence of message exchanges
- Protocol may define state-change in endpoint, e.g., file system state change
- Good protocols designed to do one thing
- Protocols can be layered
- Examples of protocols
- IP, TCP, TLS (was SSL), HTTP, Kerberos

- Globus Toolkit® 2.2

## Application Programmer Interface

- A specification for a set of routines to facilitate application development
- Refers to definition, not implementation
- E.g., there are many MPI implementations
- Spec often language-specific (or IDL)
- Routine name, number, order and type of arguments; mapping to language constructs
- Behavior or function of routine
- Examples
- GSS API (security), MPI (message passing)

- Globus Toolkit® 2.2

## A Protocol can have Multiple APIsE.g., TCP/IP

- TCP/IP APIs include BSD sockets, Winsock, System V streams, ...
- The protocol provides _interoperability_: programs using different APIs can exchange information
- I don’t need to know remote user’s API

**TCP/IP Protocol: Reliable byte streams**

**WinSock API**

**Berkeley Sockets API**

**Application**

**Application**

- Globus Toolkit® 2.2

## An API can have Multiple ProtocolsE.g., Message Passing Interface

- MPI provides portability: any correct program compiles & runs on a platform
- Does not provide interoperability: all processes must link against same SDK
- E.g., MPICH and LAM versions of MPI

- Globus Toolkit® 2.2

## Overview

- Introduction to Grids
- The opportunity
- Some definitions
**The Globus Toolkit Core Services**
- Grid security infrastructure
- Resource management
- Information infrastructure
- Data management services
- Recap and conclusions

- Globus Toolkit® 2.2

## Security Terminology

- Authentication
- Establishing identity (who are you?)
- Authorization
- Establishing permissions (what can you do?)
- Accounting
- What resources have you used?
- Certificate Authority (CA)
- Who says you are who you are?

- Globus Toolkit® 2.2

## Why Grid Security is Hard

- Resources being used may be extremely _valuable_ & the problems being solved extremely _sensitive_
- Resources are often located in distinct administrative domains
- Each resource may have own policies & procedures
- The set of resources used by a single computation may be large, _dynamic_, and/or _unpredictable_
- Not just client/server
- It must be broadly available & applicable
- Standard, well-tested, well-understood protocols
- Integration with wide variety of tools

- Globus Toolkit® 2.2

## Globus Security in a Nutshell

- Authentication based on Grid-wide credential
- X.509 certificates
- Single sign-on, delegation
- proxies
- Authorization
- Gridmap file to map certificate to local account with defined permissions

- Globus Toolkit® 2.2

## General Approach

- Define Grid security protocols & APIs
- Protocol-mediated access to remote resources
- Integrate and extend existing standards
- “On the Grid” = speak Grid protocols = speak GSI
- Develop a reference implementation
- Open source Globus Toolkit
- Client and server SDKs, services, tools
- Grid-enable wide variety of tools
- FTP, SSH, Condor, Globus Toolkit, SRB, MPI, CVS, ...
- Learn through deployment and applications

- Globus Toolkit® 2.2

## GSI Today

- GSI successfully addresses wide variety of Grid security issues
- Broad acceptance, deployment, integration with tools
- GSI adopted by 100s of sites, 1000s of users
- Globus CA has issued >6000 certs (user & host), with >1500 currently active
- Other CA’s now in existence
- NCSA, NPACI, NASA IPG, CERN/HEP
- Standardization on-going in Grid Forum, IETF
- For more information:
- www.gridforum.org/security
- “Grid Security Infrastructure (GSI) Roadmap”

- Globus Toolkit® 2.2

## Current and Future Work

- Ease of use
- CA operation, credential mgt, account mgt, proxy refresh (with Condor)
- Authorization
- Policy languages, community authorization
- Protection (despite compromised resources)
- Restricted delegation, smartcards
- Flexible communication support
- GSS-API extensions
- Independent Data Units (UDP, IP multicast)

- Globus Toolkit® 2.2

## Overview

- Introduction to Grids
- The opportunity
- Some definitions
- The Globus Toolkit Core Services
- Grid security infrastructure
**Resource management**
- Information infrastructure
- Data management services
- Recap and conclusions

- Globus Toolkit® 2.2

## Resource Management Problem

- Enabling secure, controlled remote access to computational resources and management of remote computation
- Authentication and authorization
- Resource discovery & characterization
- Reservation and allocation
- Computation monitoring and control
- Addressed by new protocols & services
- GRAM protocol as a basic building block
- Resource brokering & co-allocation services
- GSI for security, MDS for discovery

- Globus Toolkit® 2.2

## GRAM Components

- Grid Security
- Infrastructure

- Job Manager

- GRAM client API calls to
- request resource allocation
- and process creation.

- MDS client API calls
- to locate resources

- Query current status
- of resource

- Create

- RSL Library

- Parse

- Request

- Allocate &
- create processes

- Process

- Process

- Process

- Monitor &
- control

- Site boundary

- Client

- MDS: Grid Index Info Server

- Gatekeeper

- MDS: Grid Resource Info Server

- Local Resource Manager

- MDS client API calls
- to get resource info

- GRAM client API state
- change callbacks

**1**

**2**

**3**

**4**

**5**

**7**

**6**

- Globus Toolkit® 2.2

## GRAM Protocol

- Simple HTTP-based RPC
- Job request
- Returns a “job contact”: Opaque string that can be passed between clients, for access to job
- Job cancel
- Job status
- Job signal
- Event notification (callbacks) for state changes
- Pending, active, done, failed, suspended

- Globus Toolkit® 2.2

## Resource Specification Language

- Common notation for exchange of information between components
- Syntax similar to MDS/LDAP filters
- RSL provides two types of information:
- Resource requirements: Machine type, number of nodes, memory, etc.
- Job configuration: Directory, executable, args, environment
- Globus Toolkit provides an API/SDK for manipulating RSL

- Globus Toolkit® 2.2

## Resource Specification Language

- Much of the power of GRAM is in the RSL
- Common language for specifying job requests
- A conjunction of (attribute=value) pairs
- GRAM understands a well defined set of attributes

- Globus Toolkit® 2.2

## Some RSL Attributes For GRAM

- (executable=string)
- Program to run
- A file path (absolute or relative) or URL
- (directory=string)
- Directory in which to run (default is $HOME)
- (arguments=arg1 arg2 arg3...)
- List of string arguments to program
- (environment=(E1 v1)(E2 v2))
- List of environment variable name/value pairs

- Globus Toolkit® 2.2

## Job Submission Interfaces

- Globus Toolkit includes several command line programs for job submission
- globus-job-run: Interactive jobs
- globus-job-submit: Batch/offline jobs
- globusrun: Flexible scripting infrastructure
- Others are building better interfaces
- General purpose
- Condor-G, PBS, GRD, Hotpage, etc
- Application specific
- ECCE’, Cactus, Web portals

- Globus Toolkit® 2.2

## Globus Toolkit Implementation

- Gatekeeper
- Single point of entry
- Authenticates user, maps to local security environment, runs service
- In essence, a “secure inetd”
- Job manager
- A gatekeeper service
- Layers on top of local resource management system (e.g., PBS, LSF, etc.)
- Handles remote interaction with the job

- Globus Toolkit® 2.2

## Overview

- Introduction to Grids
- The opportunity
- Some definitions
- The Globus Toolkit Core Services
- Grid security infrastructure
- Resource management
**Information infrastructure**
- Data management services
- Recap and conclusions

- Globus Toolkit® 2.2

## Grid Information Services

- Globus Toolkit® 2.2

## Information Services Facts of Life

- Information is always old
- Time in flight, changing system state
- Need to provide quality metrics
- Distributed system state is hard to obtain
- Complexity of global snapshot
- Components will  fail
- Scalability and overhead
- Many different usage scenarios
- Heterogeneous policy, different information, organizations, ...

- Globus Toolkit® 2.2

## Basic Grid Questions

- Resource  *Discovery*
- “What resources are relevant?”
- Bootstraps selection process
- Resource *Status Query*
- “How do resources compare (now)?”
-  Refines selection knowledge
- Resource *Control*
- “Did I acquire the resources?”
- Not an information service task

- Globus Toolkit® 2.2

## Globus Information Service:Monitoring and Discovery Service (MDS)

- MDS includes:
- Registration & enquiry protocols
- Information models
- Provides or supports
- Standard interfaces to sensors
- Different “directory” structures
- Various discovery/access strategies

- Globus Toolkit® 2.2

## MDS-2 Base Features

- Virtual organizations (VOs)
- Collab. between individuals and institutions
- Enable sharing, community wide goals
- Support community-specific discovery
- Dynamic in nature
- Scalability
- Many resources, people, VOs
- Independence-
- Resources, VOs shouldn't affect one another
- Graceful degradation of service
- Tolerate partitions, prune failures

- Globus Toolkit® 2.2

## Information Service Approach

- Define basic classes of information service
- Resource description services
- Aggregate directory services
- Provide basic protocols for interoperability
- Resource inquiry protocol
- Resource registration protocol

- Globus Toolkit® 2.2

## MDS-2 Architecture

- D

- D

- Customized Aggregate Directories

- R

- R

- R

- R

- Standard Resource Description Services

- Registration
- Protocol

- Users

- Inquiry
- Protocol

- Globus Toolkit® 2.2

## Two Types of Information Service

- Resource description services
- Supplies information about a specific resource (GRIS – Grid Resource Information Service)
- Aggregate directory service
- Supplies collection of information gathered from multiple description servers (GIIS – Grid Index Information Service)
- Customized naming and indexing
- Support VO concept

- Globus Toolkit® 2.2

## Two Classes of Protocols

- Grid resource inquiry protocol (GRIP)
- Used to query and respond to information requests
- Grid Resource Registration Protocol (GRRP)
- Softstate protocol used to notify the existence of a service

- Globus Toolkit® 2.2

## GRIP Resource Inquiry Protocol

- Obtain information about resource
- Define data model for information, request and response formats
- Request may be general query (search)
- Can use different protocols for resource description and aggregate directory
- Advantageous to have uniform protocol
- Take a subtree and use it as any other resource description service

- Globus Toolkit® 2.2

## GRRP Resource Registration Protocol

- Soft-state protocol
- Periodic notification
- Service/resource is available
- Granularity metadata
- Automatic extension
- Add new resources to directories
- Invite resource to join new directory
- Self-cleaning
- Reduce occurrence of “dead” references

- Globus Toolkit® 2.2

## MDS-2 Implementation

- Grid Resource Information Service (GRIS)
- Provides resource description
- Modular content gateway
- Grid Index Information Service (GIIS)
- Provides aggregate directory
- Hierarchical groups of resources
- Lightweight Dir. Access Protocol (LDAP)
- Standard with many client implementations
- Used for GRIP (and GRRP currently)

- Globus Toolkit® 2.2

## Stock MDS-2.1 GRIS Providers

- globus-version reports Globus software
- grid-info-host reports host OS info
- grid-info-host-interfaces reports host NICs
- grid-info-host-load reports host CPU status
- grid-info-host-filesystem reports host disk status
- globus-gram-reporter reports Globus job status
- Also information from Ganglia (cluster monitoring software),  GridFTP Server data, software install data, and more

- Globus Toolkit® 2.2

## Extensible GIIS Framework

- Modular registration actions
- 1) Re-use registration protocol decoding
- 2) Specialize directory update (e.g. prefetch indexed data)
- Modular query actions
- 1) Re-use query protocol decoding
- 2) Specialize query handling (e.g. utilize precomputed indices)
- Provide “caching proxy” as part of release
- Send a request to index, collect info and cache it locally so next time a faster response

- Globus Toolkit® 2.2

## Globus MDS-2

- Service scales with Grid growth
- Loose consistency model tolerates failures
- Interoperability by protocols

- Globus Toolkit® 2.2

## More Information

- MDS-2
- Distributed information service
- HPDC 2001 Paper “Grid Information Services for Distributed Resource Sharing”
- MDS 2.2
- Refined protocols, security
- Fully extensible implementation
- http://www.globus.org/mds

- Globus Toolkit® 2.2

## Overview

- Introduction to Grids
- The opportunity
- Some definitions
- The Globus Toolkit Core Services
- Grid security infrastructure
- Resource management
- Information infrastructure
**Data management services**
- Recap and conclusions

- Globus Toolkit® 2.2

## Data Management Services

- Data transfer and access
**GASS:** Provides services mainly intended for use with GRAM (file staging, I/O redirection)
**GridFTP:** Provides high-performance, reliable data transfer for modern WANs
- Higher Level Data Services (not today)
**Replica Location Service:** Provides a distributed catalog service for keeping track of replicated datasets (Joint work with EDG)
**Replica Management:** Provides services for creating and managing replicated datasets
**Chimera Virtual Data Service: **keep track of provenance of data sets

- Globus Toolkit® 2.2

## GASSRemote I/O and Staging

- Tell GRAM to pull executable from remote location
- Access files from a remote location
- stdin/stdout/stderr from a remote location

- Globus Toolkit® 2.2

## What is GASS?Global Access to Secondary Storage

- (a) GASS file access API
- Replace open/close with globus_gass_open/close; read/write calls can then proceed directly
- (b) RSL extensions
- URLs used to name executables, stdout, stderr
- (c) Remote cache management utility
- (d) Low-level APIs for specialized behaviors

- Globus Toolkit® 2.2

## Example GASS Applications

- On-demand, transparent loading of data sets
- Caching of (small) data sets
- Automatic staging of code and data to remote supercomputers
- GridFTP better suited to staging of large data sets
- (Near) real-time logging of application output to remote server

- Globus Toolkit® 2.2

## GASS summary

- Simple service for small file transfers
- User by Globus_run for automatic staging of code and data to remote supercomputers
- (Near) real-time logging of application output to remote server
- GridFTP better suited to staging of large data sets

- Globus Toolkit® 2.2

## GridFTP: Basic Approach

- FTP is defined by several IETF RFCs
- Start with most commonly used subset
- Standard FTP: get/put etc., 3rd-party transfer
- Implement standard but often unused features
- GSS binding, extended directory listing, simple restart
- Extend in various ways, while preserving interoperability with existing servers
- Parameter set/negotiate, parallel transfers (multiple TCP streams), striped transfers (multiple hosts), partial file transfers, automatic & manual TCP buffer setting, progress monitoring, extended restart (via plug-ins)

- Globus Toolkit® 2.2

## GridFTP Implementation Status

- Modified wu-ftpd server
- Most features
- Modified ncftp client
- Security, TCP buffer setting
- Modified HPSS & Unitree ftpd server
- Security
- Globus Toolkit client and server SDKs, and command line tools
- Most features
- Prototype striped FTP server (aka DPSS2)

- Globus Toolkit® 2.2

## GridFTP at SC’2000: Long-Running Dallas-Chicago Transfer

- SciNet Power Failure

- Other demos starting up (Congestion)

- Parallelism Increases (Demos)

- Backbone problems on the SC Floor

- DNS Problems

- Transition between files (not zero due to averaging)

- Globus Toolkit® 2.2

## A Model Architecture for Data Grids

**A Model Architecture for Data Grids**

- Metadata Catalog

- Replica Catalog

**Tape** **Library**

**Disk Cache**

**Attribute Specification**

**Logical Collection and Logical File Name**

**Disk Array**

**Disk Cache**

- Application

- Replica
- Selection

**Multiple Locations**

- NWS

**Selected**
**Replica**

**GridFTP commands**

**Performance**
**Information &**
**Predictions**

**Replica Location 1**

**Replica Location 2**

**Replica Location 3**

- MDS

- Globus Toolkit® 2.2

## Data Management Summary

- Data transfer and access
**GASS:** Provides services mainly intended for use with GRAM (file staging, I/O redirection)
**GridFTP:** Provides high-performance, reliable data transfer for modern WANs
- Higher level replica and data tracking services
- Current and upcoming work
- Reliable file transfer

- Globus Toolkit® 2.2

## The Globus Toolkit v2 in One Slide

**The Globus Toolkit v2 in One Slide**

- Grid protocols (GSI, GRAM, ...) enable resource sharing within virtual orgs; toolkit provides reference implementation (    = Globus Toolkit services)

- Protocols (and APIs) enable other tools and services for membership, discovery, data mgmt, workflow, ...

- Globus Toolkit® 2.2

## Recap and Conclusions

## Globus ToolkitComponents Include ...

- Core protocols and services
- Grid Security Infrastructure
- Grid Resource Access & Management
- MDS information & monitoring
- GridFTP data access & transfer

- Globus Toolkit® 2.2

## The Grid World: Current Status

- Dozens of major Grid projects in scientific & technical computing/research & education
- Considerable consensus on key concepts and technologies
- Open source Globus Toolkit a de facto standard for major protocols & services
- Far from complete or perfect, but out there, evolving rapidly, and large tool/user base
- Industrial interest emerging rapidly
- Opportunity: _convergence of eScience and eBusiness requirements & technologies_

- Globus Toolkit® 2.2

## Globus Toolkit

- Globus Toolkit is the source of many of the protocols described in “Grid architecture”
- Adopted by almost all major Grid projects worldwide as a source of infrastructure
- Open source, open architecture framework encourages community development
- Active R&D program continues to move technology forward
- Developers at ANL, USC/ISI, NCSA, LBNL, and other institutions

- www.globus.org

- Globus Toolkit® 2.2

## Globus Toolkit 2: Evaluation (+)

- Good technical solutions for key problems, e.g.
- Authentication and authorization
- Resource discovery and monitoring
- Reliable remote service invocation
- High-performance remote data access
- This & good engineering is enabling progress
- Good quality reference implementation, multi-language support, interfaces to many systems, large user base, industrial support
- Growing community code base built on tools

- Globus Toolkit® 2.2

## Globus Toolkit 2: Evaluation (-)

- Protocol deficiencies, e.g.
- Heterogeneous basis: HTTP, LDAP, FTP
- No standard means of invocation, notification, error propagation, authorization, termination, ...
- Significant missing functionality, e.g.
- Databases, sensors, instruments, workflow, ...
- Virtualization of end systems (hosting envs.)
- Little work on total system properties, e.g.
- Dependability, end-to-end QoS, ...
- Reasoning about system properties

- Globus Toolkit® 2.2

## Globus Toolkit v3

- GT3 provides online negotiation of access to services in a standard way, based on OGSA specification for grid services
- GT3 enables the creation of dynamic, extensible systems
- GT3 embraces state-of-the art protocols and leverages community standards
- GT3 is evolutionary, not revolutionary
- Not changing higher-level functionality, changing protocols

- Globus Toolkit® 2.2

## Acknowledgments

- Globus Project
- Ian Foster, Steve Tuecke @ ANL
- Carl Kesselman @ USC/ISI
- The talented team of scientists and engineers at ANL, USC/ISI, elsewhere (see _[http://www.globus.org](http://www.globus.org/)__)__	_
- Support from DOE, NASA, NSF, IBM, Microsoft

- Globus Toolkit® 2.2

## Further Information

**My email: **Jennifer Schopf (jms@mcs.anl.gov)
**GT2**
- General information at _[www.globus.org](http://www.globus.org/)_
- Technical discussion list: _[discuss@globus.org](mailto:discuss@globus.org)_
**Related Publications**
- The Grid: A New Infrastructure for 21st Century Science
- Anatomy of the Grid: Foster, Kesselman, Tuecke
- Proposes an abstract architecture in which intergrid protocols enable interoperability among different grids
- Physiology of the Grid:Foster, Kesselman, Nick, Tuecke
- Introduces the concept of an Open Grid Services Architecture
- Technical Papers:
- _[www.globus.org/research/papers.html](http://www.globus.org/research/papers.html)_

- Globus Toolkit® 2.2