# Presentation: 755313

##  Grid Computing 

- Ian Foster
- Mathematics and Computer Science Division
- Argonne National Laboratory
- and
- Department of Computer Science
- The University of Chicago
- http://www.mcs.anl.gov/~foster

- Seminar at Fermilab, October 31st, 2001

## Grid Computing

## Issues I Will Address

- Grids in a nutshell
- Problem statement
- Major Grid projects
- Grid architecture
- Globus ProjectTM and ToolkitTM
- HENP data grid projects
- GriPhyN, PPDG, iVDGL

## The Grid Problem

*   *Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations

## Grid Communities & Applications:Data Grids for High Energy Physics

- Image courtesy Harvey Newman, Caltech

## Grid Communities and Applications:Network for Earthquake Eng. Simulation 

- Grid Communities and Applications:Network for Earthquake Eng. Simulation

- NEESgrid: national infrastructure to couple earthquake engineers with experimental facilities, databases, computers, & each other
- On-demand access to experiments, data streams, computing, archives, collaboration

- NEESgrid: Argonne, Michigan, NCSA, UIUC, USC   www.neesgrid.org

## Access Grid

- Collaborative work among large groups
- ~50 sites worldwide
- Use Grid services for discovery, security
- www.scglobal.org

- Access Grid: Argonne, others                             www.accessgrid.org

## Why Grids?

- A biochemist exploits 10,000 computers to screen 100,000 compounds in an hour
- 1,000 physicists worldwide pool resources for petaop analyses of petabytes of data
- Civil engineers collaborate to design, execute, & analyze shake table experiments
- Climate scientists visualize, annotate, & analyze terabyte simulation datasets
- An emergency response team couples real time data, weather model, population data

## Why Grids? (contd)

- A multidisciplinary analysis in aerospace couples code and data in four companies
- A home user invokes architectural design functions at an application service provider
- An application service provider purchases cycles from compute cycle providers
- Scientists working for a multinational soap company design a new product
- A community group pools members’ PCs to analyze alternative designs for a local road

## Elements of the Problem

- Resource sharing
- Computers, storage, sensors, networks, ...
- Sharing always conditional: issues of trust, policy, negotiation, payment, ...
- Coordinated problem solving
- Beyond client-server: distributed data analysis, computation, collaboration, ...
- Dynamic, multi-institutional virtual orgs
- Community overlays on classic org structures
- Large or small, static or dynamic

## A Little History

- Early 90s
- _Gigabit testbeds_, metacomputing
- Mid to late 90s
- Early experiments (e.g., _I-WAY_), software projects (e.g., Globus), application experiments
- 2001
- Major application communities emerging
- Major infrastructure deployments are underway
- Rich technology base has been constructed
- Global Grid Forum: >1000 people on mailing lists, 192 orgs at last meeting, 28 countries

## The Grid World: Current Status

- Dozens of major Grid projects in scientific & technical computing/research & education
- Deployment, application, technology
- Considerable consensus on key concepts and technologies
- Membership, security, resource discovery, resource management, ...
- Global Grid Forum has emerged as a significant force
- And first “Grid” proposals at IETF

## Selected Major Grid Projects

| Name | URL & Sponsors | Focus |
| --- | --- | --- |
| Access Grid | www.mcs.anl.gov/FL/accessgrid; DOE, NSF | Create & deploy group collaboration systems using commodity technologies  |
| BlueGrid | IBM | Grid testbed linking IBM laboratories |
| DISCOM | www.cs.sandia.gov/discomDOE Defense Programs | Create operational Grid providing access to resources at three U.S. DOE weapons laboratories |
| DOE Science Grid | sciencegrid.org
DOE Office of Science | Create operational Grid providing access to resources & applications at U.S. DOE science laboratories & partner universities |
| Earth System Grid (ESG) | earthsystemgrid.orgDOE Office of Science | Delivery and analysis of large climate model datasets for the climate research community |
| European Union (EU) DataGrid  | eu-datagrid.org
European Union | Create & apply an operational grid for applications in high energy physics, environmental science, bioinformatics |

*New*

*New*

## Selected Major Grid Projects

| Name | URL/Sponsor | Focus |
| --- | --- | --- |
| EuroGrid, Grid Interoperability (GRIP) | eurogrid.org
European Union | Create technologies for remote access to supercomputer resources & simulation codes; in GRIP, integrate with Globus |
| Fusion Collaboratory | fusiongrid.org
DOE Off. Science | Create a national computational collaboratory for fusion research |
| Globus Project | globus.org
DARPA, DOE, NSF, NASA, Msoft | Research on Grid technologies; development and support of Globus Toolkit; application and deployment |
| GridLab | gridlab.org
European Union | Grid technologies and applications |
| GridPP | gridpp.ac.uk
U.K. eScience | Create & apply an operational grid within the U.K. for particle physics research |
| Grid Research Integration Dev. & Support Center | grids-center.org
NSF | Integration, deployment, support of the NSF Middleware Infrastructure for research & education |

*New*

*New*

*New*

*New*

*New*

## Selected Major Grid Projects

| Name | URL/Sponsor | Focus |
| --- | --- | --- |
| Grid Application Dev. Software | hipersoft.rice.edu/grads; NSF | Research into program development technologies for Grid applications |
| Grid Physics Network | griphyn.org
NSF | Technology R&D for data analysis in physics expts: ATLAS, CMS, LIGO, SDSS |
| Information Power Grid | ipg.nasa.gov
NASA | Create and apply a production Grid for aerosciences and other NASA missions |
| International Virtual Data Grid Laboratory | ivdgl.org
NSF | Create international Data Grid to enable large-scale experimentation on Grid technologies & applications |
| Network for Earthquake Eng. Simulation Grid | neesgrid.org
NSF | Create and apply a production Grid for earthquake engineering |
| Particle Physics Data Grid | ppdg.net
DOE Science | Create and apply production Grids for data analysis in high energy and nuclear physics experiments |

*New*

*New*

## Selected Major Grid Projects

| Name | URL/Sponsor | Focus |
| --- | --- | --- |
| TeraGrid		 | teragrid.org
NSF | U.S. science infrastructure linking four major resource sites at 40 Gb/s  |
| UK Grid Support Center | grid-support.ac.uk
U.K. eScience | Support center for Grid projects within the U.K. |
| Unicore | BMBFT | Technologies for remote access to supercomputers |

*New*

*New*

- Also many technology R&D projects: e.g., Condor, NetSolve, Ninf, NWS
- See also www.gridforum.org

## Grid Architecture & Globus ToolkitTM

- The question:
- What is needed for resource sharing & coordinated problem solving in dynamic virtual organizations (VOs)?
- The answer:
- Major issues identified: membership, resource discovery & access, ..., ...
- _Grid architecture_ captures core elements, emphasizing pre-eminent role of protocols
- _Globus Toolkit_TM has emerged as de facto standard for major protocols & services

## Layered Grid Architecture(By Analogy to Internet Architecture)

- For more info: www.globus.org/research/papers/anatomy.pdf

**Notes:**
We define Grid architecture in terms of a layered collection of protocols. 
Fabric layer includes the protocols and interfaces that provide access to the resources that are being shared, including computers, storage systems, datasets, programs, and networks.  This layer is a logical view rather then a physical view.  For example, the view of a cluster with a local resource manager is defined by the local resource manger, and not the cluster hardware.  Likewise, the fabric provided by a storage system is defined by the file system that is available on that system, not the raw disk or tapes.
The connectivity layer defines core protocols required for Grid-specific network transactions.  This layer includes the IP protocol stack (system level application protocols [e.g. DNS, RSVP, Routing], transport and internet layers), as well as core Grid security protocols for authentication and authorization.
Resource layer defines protocols to initiate and control sharing of (local) resources.  Services defined at this level are gatekeeper, GRIS, along with some user oriented application protocols from the Internet protocol suite, such as file-transfer.
Collective layer defines protocols  that provide system oriented capabilities that are expected to be wide scale in deployment and generic in function.  This includes GIIS,  bandwidth brokers, resource brokers,....
Application layer defines protocols and services that are parochial  in nature, targeted towards a specific application domain or class of applications.  These are   are  are  ... arrgh

## Grid Services Architecture (1):Fabric Layer

- Just what you would expect: the diverse mix of resources that may be shared
- Individual computers, Condor pools, file systems, archives, metadata catalogs, networks, sensors, etc., etc.
- Few constraints on low-level technology: connectivity and resource level protocols form the “neck in the hourglass”
- Globus toolkit provides a few selected components (e.g., bandwidth broker)

## Grid Services Architecture (2):Connectivity Layer Protocols & Services

- Communication
- Internet protocols: IP, DNS, routing, etc.
- Security: Grid Security Infrastructure (GSI)
- Uniform authentication & authorization mechanisms in multi-institutional setting
- Single sign-on, delegation, identity mapping
- Public key technology, SSL, X.509, GSS-API (several Internet drafts document extensions)
- Supporting infrastructure: Certificate Authorities, key management, etc.

## GSI in Action: “Create Processes at A and B that Communicate & Access Files at C”

- (Kerberos)

-                Site B
-                (Unix)

- Site C
- (Kerberos)

- Computer

- User

- Computer

- Storage
- system

## Grid Services Architecture (3):Resource Layer Protocols & Services

- Resource management: GRAM
- Remote allocation, reservation, monitoring, control of [compute] resources
- Data access: GridFTP
- High-performance data access & transport
- Information: MDS (GRRP, GRIP)
- Access to structure & state information
- & others emerging: catalog access, code repository access, accounting, ...
- All integrated with GSI

## GRAM Resource Management Protocol

- Grid Resource Allocation & Management
- Allocation, monitoring, control of computations
- Secure remote access to diverse schedulers
- Current evolution
- Immediate and advance reservation
- Multiple resource types: manage anything
- Recoverable requests, timeout, etc.
- Evolve to Web Services
- Policy evaluation points for restricted proxies

- Karl Czajkowski, Steve Tuecke, others

## Data Access & Transfer

- GridFTP: extended version of popular FTP protocol for Grid data access and transfer
- Secure, efficient, reliable, flexible, extensible, parallel, concurrent, e.g.:
- Third-party data transfers, partial file transfers
- Parallelism, striping (e.g., on PVFS)
- Reliable, recoverable data transfers
- Reference implementations
- Existing clients and servers: wuftpd, nicftp
- Flexible, extensible libraries

- Bill Allcock, Joe Bester, John Bresnahan, Steve Tuecke, others

## Grid Services Architecture (4):Collective Layer Protocols & Services

- Community membership & policy
- E.g., Community Authorization Service
- Index/metadirectory/ brokering services
- Custom views on community resource collections (e.g., GIIS, Condor Matchmaker)
- Replica management and replica selection
- Optimize aggregate data access performance
- Co-reservation and co-allocation services
- End-to-end performance
- Etc., etc.

## The Grid Information Problem

- Large numbers of distributed “sensors” with different properties
- Need for different “views” of this information, depending on community membership, security constraints, intended purpose, sensor type

## Globus Toolkit Solution: MDS-2

-    Registration & enquiry protocols, information models, query languages
- Provides standard interfaces to sensors
- Supports different “directory” structures supporting various discovery/access strategies

- Karl Czajkowski, Steve Fitzgerald, others

## Community Authorization(Prototype shown August 2001)

- Community Authorization(Prototype shown August 2001)

** **

**User**

- Laura Pearlman, Steve Tuecke, Von Welch, others

## HENP Related Data Grid Projects

- Funded (or about to be funded) projects
- PPDG I		USA	DOE	$2M			1999-2001
- GriPhyN		USA	NSF	$11.9M + $1.6M	2000-2005
- EU DataGrid	EU	EC	€10M			2001-2004
- PPDG II		USA	DOE	$9.5M			2001-2004
- iVDGL		USA	NSF	$13.7M + $2M	2001-2006
- DataTAG	EU	EC	€4M			2002-2004
- Proposed projects
- GridPP		UK	PPARC	>$15M?		2001-2004
- Many national projects of interest to HENP
- Initiatives in US, UK, Italy, France, NL, Germany, Japan, ...
- EU networking initiatives (Géant, SURFNet)
- US Distributed Terascale Facility ($53M, 12 TFL, 40 Gb/s net)

## Background on Data Grid Projects

- They support several disciplines
- GriPhyN: CS, HEP (LHC), gravity waves, digital astronomy
- PPDG: CS, HEP (LHC + current expts), Nuc. Phys., networking
- DataGrid: CS, HEP (LHC), earth sensing, biology, networking
- They are already joint projects
- Multiple scientific communities
- High-performance scientific experiments
- International components and connections
- They have common elements & foundations
- Interconnected management structures
- Sharing of people, projects (e.g., GDMP)
- Globus infrastructure
- Data Grid Reference Architecture
- HENP Intergrid Coordination Board

## Grid Physics Network (GriPhyN)

-    Enabling R&D for advanced data grid systems, focusing in particular on Virtual Data concept

- ATLAS
- CMS
- LIGO
- SDSS

## Virtual Datain Action

- Data request may
- Access local data
- Compute locally
- Compute remotely
- Access remote data
- Scheduling & execution subject to local & global policies

## GriPhyN Status, October 2001

- Data Grid Reference Architecture defined
- v1: core services (Feb 2001)
- v2: request planning/mgmt, catalogs (RSN)
- Progress on ATLAS, CMS, LIGO, SDSS
- Requirements statements developed
- Testbeds and experiments proceeding
- Progress on technology
- DAGMAN request management
- Catalogs, security, policy
- Virtual Data Toolkit v1.0 out soon

## GriPhyN/PPDGData Grid Architecture

- Application

- Planner

- Executor

- Catalog Services

- Info Services

- Policy/Security

- Monitoring

- Repl. Mgmt.

- Reliable Transfer
- Service

- Compute Resource

- Storage Resource

- DAG

- DAG

- = initial solution is operational

- Ewa Deelman, Mike Wilde

## Catalog Architecture

## Python codes, Java codes 

**Fabric**

- Tape Storage Elements

- Request Formulator and Planner

**Client Applications**

- Compute Elements

- Indicates component that will be replaced

- Disk Storage Elements

- LANs and
- WANs

- Resource and Services Catalog

- Replica Catalog

- Meta-data Catalog

**Authentication and Security**

- GSI

- SAM-specific user, group, node, station registration

- Bbftp ‘cookie’

**Connectivity and Resource**

- CORBA

- UDP

- File transfer protocols -
- ftp, bbftp, rcp

- GridFTP

- Mass Storage systems protocols
- e.g. encp, hpss

**Collective Services**

- Catalog
- protocols

- Significant Event Logger

- Naming Service

- Database Manager

- Catalog Manager

- SAM Resource Management

- Batch Systems - LSF,  FBS, PBS, Condor

- Data Mover

- Job Services

- Storage Manager

- Job Manager

- Cache Manager

- Request Manager

**“****Dataset Editor”**

**“****File Storage Server”**

**“****Project Master”**

**“****Station Master”**

**“****Station Master”**

**Web**

**Python codes, Java codes**

**Command line**

**D0 Framework C++ codes**

**“****Stager”**

**“****Optimiser”**

- Code
- Repostory

- Name in **“quotes”** is SAM-given software component name

- or added

- enhanced

- using PPDG and Grid tools

## Early GriPhyN Challenge Problem:CMS Data Reconstruction

**NCSA Linux cluster **

**5) Secondary reports complete to master**

**Master Condor job running at Caltech**

**7) GridFTP fetches data from UniTree**

**NCSA UniTree - GridFTP-enabled FTP server**

**4) 100 data files transferred via GridFTP, ~ 1 GB each**

**Secondary Condor job on WI pool**

**3) 100 Monte Carlo jobs on Wisconsin Condor pool**

**2) Launch secondary job on WI pool; input files via Globus GASS**

**Caltech workstation**

**6) Master starts reconstruction jobs via Globus jobmanager on cluster**

**8) Processed objectivity database stored to UniTree**

**9) Reconstruction job reports complete to master**

- Early GriPhyN Challenge Problem:CMS Data Reconstruction

- Scott Koranda, Miron Livny, others

## Trace of a Condor-G Physics Run

## The 13.6 TF TeraGrid:Computing at 40 Gb/s

*26*

*24*

*8*

*4*

- HPSS

*5*

- HPSS

- HPSS

- UniTree

- External Networks

- External Networks

- External Networks

- External Networks

- Site Resources

- Site Resources

- Site Resources

- Site Resources

**NCSA/PACI**
**8 TF**
**240 TB**

**SDSC**
**4.1 TF**
**225 TB**

**Caltech**

**Argonne**

- TeraGrid/DTF: NCSA, SDSC, Caltech, Argonne       www.teragrid.org

## International Virtual Data Grid Lab

## Grids and Industry

- Scientific/technical apps in industry
- “IntraGrid”: Aerospace, pharmaceuticals, ...
- “InterGrid”: multi-company teams
- Globus Toolkit provides a starting point
- Compaq, Cray, Entropia, Fujitsu, Hitachi, IBM, Microsoft, NEC, Platform, SGI, Sun all porting etc.
- Enable resource sharing outside S&TC
- “Grid Services”—extend Web Services with capabilities provided by Grid protocols
- Focus of IBM-Globus collaboration
- Future computing infrastructure based on Grid-enabled xSPs & applications?

## Acknowledgments

- Globus R&D is joint with numerous people
- Carl Kesselman, Co-PI; Steve Tuecke, principal architect at ANL; others acked below
- GriPhyN R&D is joint with numerous people
- Paul Avery, Co-PI; Mike Wilde, project coordinator; Carl Kesselman, Miron Livny CS leads; numerous others
- PPDG R&D is joint with numerous people
- Richard Mount, Harvey Newman, Miron Livny, Co-PIs; Ruth Pordes, project coordinator
- Numerous other project partners
- Support: DOE, DARPA, NSF, NASA, Microsoft

## Summary

- “Grids”: Resource sharing & problem solving in dynamic virtual organizations
- Many projects now working to develop, deploy, apply relevant technologies
- Common protocols and services are critical
- Globus Toolkit a source of protocol and API definitions, reference implementations
- Rapid progress on definition, implementation, and application of Data Grid architecture
- First indications of industrial adoption