# Presentation: 751021

##  Grid Technologies & Applications:Architecture & Achievements 

- Ian Foster
- Mathematics and Computer Science Division
- Argonne National Laboratory
- and
- Department of Computer Science
- The University of Chicago
- http://www.mcs.anl.gov/~foster

- Computing in High Energy and Nuclear Physics, Beijing, P.R. China, 2001

## Acknowledgments

- Globus R&D is joint with numerous people
- Carl Kesselman, Co-PI
- Steve Tuecke, principal architect at ANL
- Others to be acknowledged
- GriPhyN R&D is joint with numerous people
- Paul Avery, Co-PI; Newman, Lazzarini, Szalay
- Mike Wilde, project coordinator
- Carl Kesselman, Miron Livny CS leads
- ATLAS, CMS, LIGO, SDSS participants; others
- Support: DOE, DARPA, NSF, NASA, Microsoft

## Issues I Will Address

- Grids in a nutshell
- Problem statement
- Major Grid projects
- Progress and plans in two specific projects
- Globus Project and Toolkit
- GriPhyN project
- Enabling international cooperation
- Grids and commodity technologies

## The Grid Problem

*   *Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations

## Elements of the Problem

- Resource sharing
- Computers, storage, sensors, networks, ...
- Sharing always conditional: issues of trust, policy, negotiation, payment, ...
- Coordinated problem solving
- Beyond client-server: distributed data analysis, computation, collaboration, ...
- Dynamic, multi-institutional virtual orgs
- Community overlays on classic org structures
- Large or small, static or dynamic

## Grid Communities & Applications:Data Grids for High Energy Physics

- Image courtesy Harvey Newman, Caltech

## A Little History

- Early 90s
- _Gigabit testbeds_, metacomputing
- Mid to late 90s
- Early experiments (e.g., _I-WAY_), software projects (e.g., Globus), application experiments
- 2001
- Major application communities emerging
- Major infrastructure deployments are underway
- Rich technology base has been constructed
- Global Grid Forum: >1000 people on mailing lists, 192 orgs at last meeting, 28 countries

## Selected Major Grid Projects

| Name | URL & Sponsors | Focus |
| --- | --- | --- |
| Access Grid | www.mcs.anl.gov/FL/accessgrid; DOE, NSF | Create & deploy group collaboration systems using commodity technologies  |
| BlueGrid | IBM | Grid testbed linking IBM laboratories |
| DISCOM | www.cs.sandia.gov/discomDOE Defense Programs | Create operational Grid providing access to resources at three U.S. DOE weapons laboratories |
| DOE Science Grid | sciencegrid.org
DOE Office of Science | Create operational Grid providing access to resources & applications at U.S. DOE science laboratories & partner universities |
| Earth System Grid (ESG) | earthsystemgrid.orgDOE Office of Science | Delivery and analysis of large climate model datasets for the climate research community |
| European Union (EU) DataGrid  | eu-datagrid.org
European Union | Create & apply an operational grid for applications in high energy physics, environmental science, bioinformatics |

*New*

*New*

## Selected Major Grid Projects

| Name | URL/Sponsor | Focus |
| --- | --- | --- |
| EuroGrid, Grid Interoperability (GRIP) | eurogrid.org
European Union | Create technologies for remote access to supercomputer resources & simulation codes; in GRIP, integrate with Globus |
| Fusion Collaboratory | fusiongrid.org
DOE Off. Science | Create a national computational collaboratory for fusion research |
| Globus Project | globus.org
DARPA, DOE, NSF, NASA, Msoft | Research on Grid technologies; development and support of Globus Toolkit; application and deployment |
| GridLab | gridlab.org
European Union | Grid technologies and applications |
| GridPP | gridpp.ac.uk
U.K. eScience | Create & apply an operational grid within the U.K. for particle physics research |
| Grid Research Integration Dev. & Support Center | grids-center.org
NSF | Integration, deployment, support of the NSF Middleware Infrastructure for research & education |

*New*

*New*

*New*

*New*

*New*

## Selected Major Grid Projects

| Name | URL/Sponsor | Focus |
| --- | --- | --- |
| Grid Application Dev. Software | hipersoft.rice.edu/grads; NSF | Research into program development technologies for Grid applications |
| Grid Physics Network | griphyn.org
NSF | Technology R&D for data analysis in physics expts: ATLAS, CMS, LIGO, SDSS |
| Information Power Grid | ipg.nasa.gov
NASA | Create and apply a production Grid for aerosciences and other NASA missions |
| International Virtual Data Grid Laboratory | ivdgl.org
NSF | Create international Data Grid to enable large-scale experimentation on Grid technologies & applications |
| Network for Earthquake Eng. Simulation Grid | neesgrid.org
NSF | Create and apply a production Grid for earthquake engineering |
| Particle Physics Data Grid | ppdg.net
DOE Science | Create and apply production Grids for data analysis in high energy and nuclear physics experiments |

*New*

*New*

## Selected Major Grid Projects

| Name | URL/Sponsor | Focus |
| --- | --- | --- |
| TeraGrid		 | teragrid.org
NSF | U.S. science infrastructure linking four major resource sites at 40 Gb/s  |
| UK Grid Support Center | grid-support.ac.uk
U.K. eScience | Support center for Grid projects within the U.K. |
| Unicore | BMBFT | Technologies for remote access to supercomputers |

*New*

*New*

- Also many technology R&D projects: e.g., Condor, NetSolve, Ninf, NWS
- See also www.gridforum.org

## Grid Communities and Applications:Network for Earthquake Eng. Simulation 

- Grid Communities and Applications:Network for Earthquake Eng. Simulation

- NEESgrid: national infrastructure to couple earthquake engineers with experimental facilities, databases, computers, & each other
- On-demand access to experiments, data streams, computing, archives, collaboration

- NEESgrid: Argonne, Michigan, NCSA, UIUC, USC   www.neesgrid.org

## The 13.6 TF TeraGrid:Computing at 40 Gb/s

*26*

*24*

*8*

*4*

- HPSS

*5*

- HPSS

- HPSS

- UniTree

- External Networks

- External Networks

- External Networks

- External Networks

- Site Resources

- Site Resources

- Site Resources

- Site Resources

**NCSA/PACI**
**8 TF**
**240 TB**

**SDSC**
**4.1 TF**
**225 TB**

**Caltech**

**Argonne**

- TeraGrid/DTF: NCSA, SDSC, Caltech, Argonne       www.teragrid.org

## Access Grid

- Collaborative work among large groups
- ~50 sites worldwide
- Use Grid services for discovery, security
- See also www.scglobal.org

- Access Grid: Argonne, others           www.mcs.anl.gov/FL/accessgrid

## Layered Grid Architecture(By Analogy to Internet Architecture)

- For more info: www.globus.org/research/papers/anatomy.pdf

**Notes:**
We define Grid architecture in terms of a layered collection of protocols. 
Fabric layer includes the protocols and interfaces that provide access to the resources that are being shared, including computers, storage systems, datasets, programs, and networks.  This layer is a logical view rather then a physical view.  For example, the view of a cluster with a local resource manager is defined by the local resource manger, and not the cluster hardware.  Likewise, the fabric provided by a storage system is defined by the file system that is available on that system, not the raw disk or tapes.
The connectivity layer defines core protocols required for Grid-specific network transactions.  This layer includes the IP protocol stack (system level application protocols [e.g. DNS, RSVP, Routing], transport and internet layers), as well as core Grid security protocols for authentication and authorization.
Resource layer defines protocols to initiate and control sharing of (local) resources.  Services defined at this level are gatekeeper, GRIS, along with some user oriented application protocols from the Internet protocol suite, such as file-transfer.
Collective layer defines protocols  that provide system oriented capabilities that are expected to be wide scale in deployment and generic in function.  This includes GIIS,  bandwidth brokers, resource brokers,....
Application layer defines protocols and services that are parochial  in nature, targeted towards a specific application domain or class of applications.  These are   are  are  ... arrgh

## Where Are We With Architecture?

- No “official” standards exist
- But:
- Globus Toolkit has emerged as the de facto standard for several important Connectivity, Resource, and Collective protocols
- GGF has an architecture working group
- Technical specifications are being developed for architecture elements: e.g., security, data, resource management, information

## Globus and GriPhyN:The Focus of this Talk

- Globus Project and Toolkit
- R&D project at ANL, UChicago, USC/ISI
- Open source software and community
- Emphasis on core protocols and services
- Adopted by essentially all major Grid efforts
- Grid Physics Network (GriPhyN)
- Data Grid R&D (ATLAS, CMS, LIGO, SDSS)
- Defines Data Grid Reference Architecture in partnership with Particle Physics Data Grid
- Emphasis on higher-level protocols/services

## Grid Architecture andthe Globus Toolkit

## Grid Services Architecture (1):Fabric Layer

- Just what you would expect: the diverse mix of resources that may be shared
- Individual computers, Condor pools, file systems, archives, metadata catalogs, networks, sensors, etc., etc.
- Few constraints on low-level technology: connectivity and resource level protocols form the “neck in the hourglass”
- Globus toolkit provides a few selected components (e.g., bandwidth broker)

## Grid Services Architecture (2):Connectivity Layer Protocols & Services

- Communication
- Internet protocols: IP, DNS, routing, etc.
- Security: Grid Security Infrastructure (GSI)
- Uniform authentication & authorization mechanisms in multi-institutional setting
- Single sign-on, delegation, identity mapping
- Public key technology, SSL, X.509, GSS-API (several Internet drafts document extensions)
- Supporting infrastructure: Certificate Authorities, key management, etc.

## GSI in Action: “Create Processes at A and B that Communicate & Access Files at C”

- (Kerberos)

-                Site B
-                (Unix)

- Site C
- (Kerberos)

- Computer

- User

- Computer

- Storage
- system

## GSI Futures

- Scalability in numbers of users & resources
- Credential management
- Online credential repositories (“MyProxy”)
- Account management
- Authorization
- Policy languages
- Community authorization
- Protection against compromised resources
- Restricted delegation, smartcards

## Community Authorization(Prototype shown August 2001)

- Community Authorization(Prototype shown August 2001)

** **

**User**

- Laura Pearlman, Steve Tuecke, Von Welch, others

## Grid Services Architecture (3):Resource Layer Protocols & Services

- Resource management: GRAM
- Remote allocation, reservation, monitoring, control of [compute] resources
- Data access: GridFTP
- High-performance data access & transport
- Information: MDS (GRRP, GRIP)
- Access to structure & state information
- & others emerging: catalog access, code repository access, accounting, ...
- All integrated with GSI

## GRAM Resource Management Protocol

- Grid Resource Allocation & Management
- Allocation, monitoring, control of computations
- Simple HTTP-based RPC
- Job request:
- Returns a “job contact”: Opaque string that can be passed between clients, for access to job
- Job cancel, Job status, Job signal
- Event notification (callbacks) for state changes
- Pending, active, done, failed, suspended
- Servers for most schedulers; C and Java APIs

## Resource Management Futures:GRAM-2 (planned for late 2001)

- Advance reservations
- As prototyped in GARA in previous 2 years
- Multiple resource types
- Manage anything: storage, networks, etc., etc.
- Recoverable requests, timeout, etc.
- Build on early work with Condor group
- Use of SOAP (RPC using HTTP + XML)
- First step towards Web Services
- Policy evaluation points for restricted proxies

- Karl Czajkowski, Steve Tuecke, others

## Data Access & Transfer

- GridFTP: extended version of popular FTP protocol for Grid data access and transfer
- Secure, efficient, reliable, flexible, extensible, parallel, concurrent, e.g.:
- Third-party data transfers, partial file transfers
- Parallelism, striping (e.g., on PVFS)
- Reliable, recoverable data transfers
- Reference implementations
- Existing clients and servers: wuftpd, nicftp
- Flexible, extensible libraries

- Bill Allcock, Joe Bester, John Bresnahan, Steve Tuecke, others

## Grid Services Architecture (4):Collective Layer Protocols & Services

- Index servers aka metadirectory services
- Custom views on dynamic resource collections assembled by a community
- Resource brokers (e.g., Condor Matchmaker)
- Resource discovery and allocation
- Replica management and replica selection
- Optimize aggregate data access performance
- Co-reservation and co-allocation services
- End-to-end performance
- Etc., etc.

## The Grid Information Problem

- Large numbers of distributed “sensors” with different properties
- Need for different “views” of this information, depending on community membership, security constraints, intended purpose, sensor type

## Globus Toolkit Solution: MDS-2

-    Registration & enquiry protocols, information models, query languages
- Provides standard interfaces to sensors
- Supports different “directory” structures supporting various discovery/access strategies

- Karl Czajkowski, Steve Fitzgerald, others

## Resource Management Architecture

- GRAM

- GRAM

- LSF

- Condor

- NQE

- Application

- RSL

- Simple ground RSL

- Information Service

- Local
- resource
- managers

- RSL
- specialization

- Ground RSL

- Queries

- & Info

- DUROC
- MPICH-G2

## GriPhyN Project

## The Virtual Data Concept

-  “[a virtual data grid enables] the definition and delivery of a potentially unlimited virtual space of data products derived from other data.  In this virtual space, requests can be satisfied via direct retrieval of materialized products and/or computation, with local and global resource management, policy, and security constraints determining the strategy used.”

## Virtual Datain Action

- Data request may
- Access local data
- Compute locally
- Compute remotely
- Access remote data
- Scheduling & execution subject to local & global policies

## Grid Physics Network (GriPhyN)

-    Enabling R&D for advanced data grid systems, focusing in particular on Virtual Data concept

- ATLAS
- CMS
- LIGO
- SDSS

## GriPhyN Status, August 2001

- Data Grid Reference Architecture defined
- v1: core services (Feb 2001)
- v2: request planning/mgmt, catalogs (RSN)
- Progress on ATLAS, CMS, LIGO
- Requirements statements developed
- Testbeds and experiments proceeding
- Progress on technology
- DAGMAN request management
- Catalogs, security, policy
- Virtual Data Toolkit v1.0 out soon

## GriPhyN/PPDGData Grid Architecture

- Application

- Planner

- Executor

- Catalog Services

- Info Services

- Policy/Security

- Monitoring

- Repl. Mgmt.

- Reliable Transfer
- Service

- Compute Resource

- Storage Resource

- DAG

- DAG

- = initial solution is operational

- Ewa Deelman, Mike Wilde

## Example Data Grid Technologies

- Replica catalog
- Centralized in place, decentralized in design
- Transformation, virtual data catalogs
- Various prototypes
- DAGMAN (M. Livny, U.Wisconsin)
- Reliably execute a directed acyclic graph of compute and data movement tasks on Grid

- Ann Chervenak, Ewa Deelman, Miron Livny, Mike Wilde, others

## Catalog Architecture

## Early GriPhyN Challenge Problem:CMS Data Reconstruction

**NCSA Linux cluster **

**5) Secondary reports complete to master**

**Master Condor job running at Caltech**

**7) GridFTP fetches data from UniTree**

**NCSA UniTree - GridFTP-enabled FTP server**

**4) 100 data files transferred via GridFTP, ~ 1 GB each**

**Secondary Condor job on WI pool**

**3) 100 Monte Carlo jobs on Wisconsin Condor pool**

**2) Launch secondary job on WI pool; input files via Globus GASS**

**Caltech workstation**

**6) Master starts reconstruction jobs via Globus jobmanager on cluster**

**8) Processed objectivity database stored to UniTree**

**9) Reconstruction job reports complete to master**

- Early GriPhyN Challenge Problem:CMS Data Reconstruction

- Scott Koranda, Miron Livny, others

## Trace of a Condor-G Physics Run

## International Cooperation

## International Cooperation

- Critical for long-term success
- We hang together or hang separately
- Scope of effort demands it
- International science demands it
- Too early to tell if we will succeed
- Some positives: e.g., use of Globus, GDMP
- Still too few mutual interdependencies
- International Data Grid Coordination Cmte
- International “laboratories” seem key

## iVDGL

- International Virtual-Data Grid Laboratory
- A place to conduct Data Grid tests at scale
- Concrete manifestation of world-wide grid activity
- Continuing activity that will drive Grid awareness
- Scale of effort
- For national, intl scale Data Grid tests, operations
- Computation & data intensive computing
- Who
- Initially US-UK-Italy-EU; Japan, Australia
- & Russia, China, Pakistan, India, South America?
- StarLight and other international networks vital

- U.S. Co-PIs: Avery, Foster, Gardner, Newman, Szalay

## iVDGL Map Circa 2002-2003

## Commodity Technologies

## And What’s This Got To Do With ...

- CORBA?
- Grid-enabled CORBA underway
- Java, Jini, Jxta?
- Java CoG Kit.  Jini, Jxta: future uncertain
- Web Services, .NET, J2EE?
- A significant focus within Globus project
- GRAM-2 will use SOAP and WSDL
- Q: What can Grid offer to Web services?
- Next revolutionary technology of the month?
- They’ll need Grid technologies too

## Summary

- “Grids”: Resource sharing & problem solving in dynamic virtual organizations
- Many projects now working to develop, deploy, apply relevant technologies
- Common protocols and services are critical
- Globus Toolkit a source of protocol and API definitions, reference implementations
- Rapid progress on definition, implementation, and application of Data Grid architecture
- Harmonizing U.S. and E.U. efforts important