# Presentation: 637177

## DOE Science Grid:

**Computing and Data Infrastructure for Large-Scale Science**
**DOE Science Grid:**
**Support for Computational Science**

***William Johnston,******	******Lawrence Berkeley National Lab******               and NASA Ames Research Center***
***Ray Bair, Pacific Northwest National Lab***
***Ian Foster, Argonne National Lab***
***Al Giest, Oak Ridge National Lab***
***Bill Kramer, National Energy Research******	******Scientific Computing Center***
***http://doesciencegrid.org***

## Motivation for Science Grids

- Large-scale science and research engineering are done through the interaction of
- people
- heterogeneous computing resources
- information systems
- instruments
- all of which are _*geographically and organizationally dispersed*_.
*The overall motivation for “Grids” is to facilitate the *_*routine interactions*_* of all of these resources in order to support widely distributed, multi-institutional science and engineering.*

## Applications MotivatingGrid Computing Environments

- Multi-disciplinary simulations are an example of a class of applications that require aggregation and coordinated use of many widely distributed computing, data, and intellectual resources
- These system simulations require coupling codes and data across many compute and data systems located at multiple organizations

## Multi-disciplinary Simulations:Aviation Safety

**Virtual**
**National Air Space**
**(VNAS)**

- FAA Ops Data
- Weather Data
- Airline Schedule Data
- Digital Flight Data
- Radar Tracks
- Terrain Data
- Surface Data

*Multi-component aircraft simulations are inserted into a realistic environment, which requires adding many types of operations data as drivers for the system simulations.*

**Wing Models**

**Stabilizer Models**

**Engine Models**

**Landing Gear Models**

**Human Models**

**Airframe Models**

## Large-Scale Science Problems

- This example is typical - there are many large-scale science and engineering problems that involve
- A complex mix of instrument based data collection, storage, and analysis, coordinated with supercomputer simulation that leads to instrumentre-targeting (e.g. supernova cosmology)
- Coupled, world-wide collection of computing and data systems, and collaborators (e.g. multi-component climate simulation)
- Management of Petabytes/year of data across organizations (e.g. high energy physics)

## Application Motivation forGrid Computing Environments

- These large-scale science and engineering problems *require integrating applications and data that are developed and/or maintained by different teams of researchers or obtained from different instruments*.
- These activities almost always involve
- large data volumes and high data transfer rates to and from the instruments and data archives and between computing systems
- supercomputing in complex workflow situations

## Application Motivation for Grid Computing Environments

- These activities are inherently dispersed:*The research teams are the only ones that have the expertise to maintain and improve the simulation code and/or the body of data that drives the simulations or data analysis, or involve fixed location or remote instruments*
-    This results in an inherently heterogeneous, distributed computing and data management environment

## Analysis of Various Science and Engineering Scenarios Provide Requirements for Grids

- _Requirements for Grids_
*Uniform access* and management are needed for many *heterogeneous compute and data resources*
- Existing heterogeneous *simulation and data analysis components need to be coupled and operated simultaneously* across distributed resources
- Interfaces to computational and data analysis tools must provide *appropriate levels of abstraction* for discipline problem solvers

## Requirements for Grids

*Complex workflow scenarios* involving many compute and data intensive steps occurring at different geographic locations must be managed reliably and securely across distributed resources
- Tools and services for autonomous *fault management* and recovery are required for both applications and infrastructure

## Large, complex data archives – e.g. geometry (structure) and performance of airframes and turbomachines – are maintained by discipline experts at different sites, and must be accessed and updated by collaborating analysts.
Massive datasets must be distributed and managed so that they are accessible by world-wide collaborations

*Massive datasets* *must be distributed and managed* so that they are accessible by world-wide collaborations

***Requirements for Grids***

## Requirements for Grids

*Data streams from instrument systems* must be managed and analyzed in real-time by computational data analysis systems
- Scientists and engineers must be able to *securely and selectively share* all aspects of their work process
- There should be a *single security mechanism, invoked once* by the user, to gain access to all resources needed to solve a problem

## Requirements for Grids

*There are additional requirements from the tool developers* – the application domain computational scientists – primarily in the area of supporting code development and execution environment access and management. E.G.:
- uniform batch queue management across all systems
- tools to minimize queue wait time
- tools to manage the available pool of compute resources, including allocation management across the pool
- tools to ensure that large collections of related and/or interdependent jobs run reliably as a work unit

## Requirements for Grids

- uniform view of all tertiary storage systems
- file location management to minimize wait time
- cataloguing mechanisms to characterize and access datasets independent of physical storage location

## Requirements for Grids

- How do Grids address these requirements?

## What are Grids?

- Middleware foruniform, secure, and highly capable access to large and small scale computing, data, and instrument systems that are distributed across organizations
- Ancillary services supporting application frameworks/portals
- Persistent infrastructure (e.g. DOE Science Grid and NASA’s IPG) supporting
- Grid services on the compute and data systems of interest(Grid sysadmin)
- authentication supporting single sign-on(X.509 Certification Authorities)
- resource discovery(Grid Information Service – distributed directory service)
- 200 people working on standards at the IETF-likeGlobal Grid Forum (www.gridforum.org)

## Basic Grid Common Services / Middleware

- Uniform and versatile authentication, authorization, and security services for users and applications
- Standardized access to data cataloguing, data archiving, and data location management
- Standardized access to compute resources
- Tools and services supporting construction and management of collaborations (virtual organizations)

## Basic Grid Common Services

- Dynamic construction of execution environments supporting complex distributed applications
- co-scheduling many resources to support, e.g., transient and complex, science and engineering experiments that require combinations of instruments, compute systems, data archives, and network bandwidth at multiple locations
- individual resource managers must provide the underlying capabilities needed to support the semantics of the Grid services - e.g. advance reservation in batch queuing systems

## Ancillary Services

- Ancillary services are built using the basic Grid services in order to *support structured and collaborative use of multiple applications and data archives*
- “knowledge Grid” services that abstract the scientist’s view of applications and Grid services
- tools for constructing discipline specific portals and Web clients that easily incorporate/use Grid services
- workflow management systems that can express and manipulate the interactions of the many simulation and data analysis components of a problem solving environment
- Grid interfaces to existing frameworks: CORBA, MPI, Condor-G, DCOM (not yet)

## Architecture of a Grid

**Grid  Information  Service**

**Uniform****Resource****Access**

**Brokering**

**Global Queuing**

**Global Event****Services**

**Co-Scheduling**

**Data Cataloguing**

**Uniform Data Access**

**Collaboration and Remote Instrument Services**

**Network Cache**

**Communication Services**

**Authentication Authorization**

**Security Services**

**Auditing**

**Fault Management**

**Monitoring**

_***Grid Common Services***_: Standardized Services and Resources Interfaces

_***Applications***_: Simulations, Data Analysis, etc.
_***Toolkits***_: Visualization, Data Publication/Subscription, etc.
_***Execution support***_: Globus MPI, Condor-G, CORBA

_***Distributed Resources***_

- Discipline Specific _***Portals***_ andScientific _***Workflow Management***_ Systems

- Condor poolsof workstations

- network caches

- tertiary storage

- national user facilities

- clusters

- national supercomputer facilities

- High-speed Networks and Communications Services

**= Globus services, SRB, GridFTP**

***Architecture of a Grid***

## DOE Science Grid

- SciDAC project to explore the issues for providing persistent operational Grid support in the DOE environment: LBNL, NERSC, PNNL, ANL, and ORNL
- Initial computing resources
-  10 small, medium, and large clusters
- High bandwidth connectivity end-to-end(high-speed links from site systems to ESNet gateways)
- Storage resources: four tertiary storage systems (NERSC, PNNL, ANL, and ORNL)
*Globus* providing the Grid Common Services

## DOEScience Grid

- Initial Science Grid Configuration

**NERSC****Supercomputing****& Large-Scale Storage**

**PNNL**

**LBNL**

**ANL**

**ESnet**

**Europe**

**DOE****Science Grid**

**ORNL**

**ESNet**

**MDS**
**CA**

**Grid Managed Resources**

**User Interfaces**
**Application Frameworks**
**Applications**

**Grid Services:**** Uniform access to distributed resources**

**Asia-Pacific**

**Funded by the U.S. Dept. of Energy, Office of Science,****Office of Advanced Scientific Computing Research,****Mathematical, Information, and Computational Sciences Division**

## How are Grids Being Used:Multi-disciplinary Simulation Example

*    **Multiple sub-systems, e.g. a wing lift model operating at NASA Ames and a turbo-machine model operating at NASA Glenn, are combined using an application framework like NPSS that manage the interactions of multiple models and uses IPG services to coordinate computing and data storage systems across NASA.*

## Multi-disciplinary Simulations:Aviation Safety

- operational conditions
- database
- (Objectivity)

- Web client User Interface

- environment database

- NPSS CORBA framework (coupling, sequencing and data input/output mgmt for multiple simulations)

**Grid Services: Uniform access to distributed resources**

- Globus initiatedCORBA server running computational simulation

- IPG resources

*NASA Ames*

*NASA Glenn*

- Grid Secure Communication Services

- Web server
- JSP servlets

## High Speed Distributed  Data Access:(NASA Grid Milestone Completed 3/2000)

** ****Data access and analysis capabilities of IPG are demonstrated by *****parallel data mining***
** ****512 node SGI Origin at Ames uses IPG uniform interface data access tools (SRB) to simultaneously mine hydrology  data from four sites**
** ****SDSC**
** ****CalTech**
** ****GRC**
** ****Washington U.**

- Mining Daemon &
- Control Database

**IPG Processor**

**Satellite Data **
**From GRC**
**via GASS**

**IPG Mining **
**Agent**

**IPG Processor**

**IPG Mining **
**Agent**

**IPG Processor**

**Satellite Data **
**From SDSC **
**via SRB**

**Satellite Data **
**From CalTech**
**via SRB**

**Result from one agent**

**IPG Mining **
**Agent**

**IPG Processor**

- . . .

**Satellite Date**
**From Wash. U.**
**via SRB**

## These Examples Illustrate a Grid Success

- Standardized access to multi-institutional resources
- A common security approach and single sign-on
- Persistent Grid services (Globus job management and scheduling) that are *used to run application frameworks on an as-needed basis*
- CORBA
- Agent systems / servers (data mining example)
- CONDOR job manager (“Glide-in”)
-    This allows users great flexibility in building their applications in the framework of their choice. *They do not have to rely on that framework being provided as a persistent service on all of the computing systems where they need to run* – they can instantiate their own environment using persistent Grid services.

## Grid Services for Applications

- The Science Grid will initially provide
- a set of basic Grid resource uniform access services
- a set of “collective” services that manage collections of individual services through higher-level Grid services
- sophisticated multi-resource job manager (Condor-G)
- metadata catalogue based, uniform tertiary storage system service (SDSC’s MCAT/SRB)
- all of these services operate in a uniform, single sign-on security environment (which also supports other utilities such as Globus ssh and GridFTP)
- auxiliary usability services such as Web “hotpages” for execution environment management and MyProxy server (to simplify user credential management)

## Grid Services for Applications

- Globus services through native C language bindings, and through wrappers in several other languages: Python, Java, Perl, CORBA

## Supernova Cosmology Requires Complex,Widely Distributed Workflow Management

- “Supernova cosmology” is observational cosmology that is based on finding and observing special types of supernova
- The process requires
- A complex mix of instrument based data collection, storage, and analysis, coordinated with supercomputer simulation that leads to instrument re-targeting
- A loosely coupled, world-wide collection of instruments, computing and data systems, and collaborators

## Supernova Cosmology Requires Complex,Widely Distributed Workflow Management

## The Unified Science Environment

***The Unified Science Environment***

- A model that is emerging in today's practice of science is characterized by _time constrained cycles of interactions_ in which computation and/or human interactions are gating functions.

_**time critical**_
_**experiment /**__**observation**_

**theory via**
**computational simulation**

**modify**

**modify**

**data**

**computational**
**analysis**

**“****servoing,” steering /**
**human interaction**

- In one cycle, simulations are used to explain or elaborate on the current state of the experiment, and experiment results are used to refine the simulations
- In another cycle, computational analysis is used to “steer” the instrument

## Grid/Globus Basic Services: Libraries

- Globus application I/O
- multi-method communication library
- uses Globus Security Infrastructure
- secure, and optionally confidential, messaging

## Grid/Globus Basic Services: File Access

- GASS provides remote file access via http, and some local cache management
- GASS file
- open(), close(): works on files and URLs
- GASS transfer
- client & server handling of get/put requests
- GASS server_ez
- simple, embeddable server
- GASS cache
- cache for remote files (prefetch etc.)
- DRM / HRM cache management (Shoshani, LBNL)

## Grid/Globus Basic Services: File Access

- GridFTP - the Grid universal access/transport protocol

| GSI, Kerberos security | data channel reuse |
| --- | --- |
| third-party transfers | integrated instrumentation |
| communication parameter set/negotiate | logging/audit trail |
| extended directory listing | parallel transfers |
| partial file access | server data striping (cf DPSS and HPSS striped tapes) |
| reliability/restart | policy-based access control |
| large file support | server-side computation |
|  | proxies (firewall, load bal) |

## Grid/Globus Basic Services: Health and Status

- Globus system / process monitoring
- HBM client
- add a process to watch by a local monitor
- HBM local monitor
- one local monitor per host
- process monitor, sends UDP heart beats to HBM collector
- HBM data collector
- acts on info sent by many local monitors
- Will probably be replaced by the GGFGrid Monitoring Architecture services

## Grid/Globus Basic Services: Information Services

- Grid Information Service / MDS tools
- provides Grid resource information (e.g. computing system architecture, configuration, compiler versions, etc.)
- client programs to query for available resources that match a given criteria

## Grid/Globus Basic Services: Resource Management

- GRAM-2
- basic job initiator
- includes advance reservation _*if supported by the underlying resource*_
- client API (submit)
- job manager API (query)
- gatekeeper (compute/resource side authorization)
- DUROC
- resource broker and co-allocator
- control API
- runtime API

## Grid/Globus Basic Services: Security

- gssapi_ssl
- Grid Security Infrastructure, including proxies (delegation)
- GSS-API application library
- SSL authentication (public key)
- gss_assist
- simplified interface to GSS-API

## Grid/Globus Ancillary Services: MPI

- MPICH-G2 is a grid-enabled implementation of the MPI v1.1 standard using Globus services (e.g., job startup, security)
- allows you to couple multiple machines, potentially of different architectures, to run MPI applications
- automatically converts data in messages sent between machines of different architectures
- supports multiprotocol communication by automatically selecting TCP for inter-machine messaging and (where available) vendor-supplied MPI for intra-machine messaging

## Grid Ancillary Services: Global Queuing

- Global queuing is intended to provide an automatic queue wait time minimization and load balancing across those resources whose resource managers (batch queuing systems) have the necessary support
- available early CY02

## Grid/Globus Collective Services: Data Grids

- Replica Catalog

- Attribute Specification

- Logical Collection and Logical File Name

- Disk Array

- Disk Cache

- Application

- Replica
- Selection

- Multiple Locations

- NWS

- Selected
- Replica

- GridFTP Control Channel

- Performance
- Information &
- Predictions

**Replica Location**

**Replica Locations**

**Replica Location**

- MDS

- GridFTPDataChannel

**CY01-02**

**SRB**

**GridFTP**

## Condor-G: Grid Job Management

- The user defines the tasks to be executed and their inter-relationships
- Condor-G then
- handles all aspects of discovering and acquiring appropriate resources, regardless of their location;
- initiates, monitors, and manages execution on those resources
- detects and responds to failure such as proxy expiry, failure of the target systems or of the Condor job management services on those systems
- notifies the user of termination
- provides “execute once, and only once” semantics
- The result is a powerful tool for managing a variety of parallel computations in Grid environments.

## Condor-G: Grid Job Management

- When combined with the Condor ***Glide-In***, which builds a Condor execution environment on the target resources, Condor-G can
- minimize queuing delays by
- managing a dynamically changing set of computational resources
- distributing the jobs across the GlideIns that have started executing out of the batch queues of the target resources
- dynamically checkpoint executing jobs and move them to other computing systems (e.g. when an allocation runs out)

## DOE Science Grid

- The mission of DOE’s Office of Science is large-scale science
*We are seeing examples of substantial changes in how large-scale science is conducted as the result of the being able to closely integrate distributed computing and data management with the process of science*
- In many of these instances, this integration is the only way in which significant science can be done
- supernova cosmology
- high energy physics data management
- climate comparative simulation studies

## Quasi-real-time, CollaborativeSupercomputer Analysis ofMicro-tomographic Data usingGrid Services and Infrastructure

- supercomputer*
- tomographic
- reconstruction

- real-time
- collection*

- wide-area
- disseminationbandwidth*

- desktop
- & VR clients

- Advanced Photon Source

- shared controls

- Ian Foster

- ARGONNE  CHICAGO

- *must be co-scheduled

***Quasi-real-time, ***_***Collaborative***__***Supercomputer Analysis***_*** of******Micro-tomographic Data using******Grid Services and Infrastructure***

## High Energy Physics

- HEP experiments cost $US 1B, last 20 years, involve thousands of collaborators at hundreds of institutions world-wide, and collect and analyze petabytes of data per year
- The Data Grid services (e.g. GriPhyN, PPDG, and EU DataGrid projects)
- virtual data
- transparency wrt materialization
- uniform and high-speed access
- data replica catalogues
- transparency wrt location
-     are being driven by HEP

## LHC Data Grid Hierarchy

**Online System**

**event****reconstruction**

**French Regional Center**

**German Regional Center**

- Institute

- Institute

- Institute

**Institute ~0.25TIPS**

- Workstations

**~100 MBytes/sec**

**~0.6-2.5 Gbps**

- 100 - 1000 Mbits/sec

- Physics data cache

**~PByte/sec**

**~2.5 Gbits/sec**

**~0.6-2.5 Gbps**

***Tier 0 +1***

***Tier 3***

***Tier 4***

**LHC Data Grid Hierarchy**
**CMS as example, Atlas is similar**

***Tier 2***

***CERN/CMS data goes to 6-8 Tier 1 regional centers, and from each of these to 6-10 Tier 2 centers.***
***Physicists work on analysis “channels” at 135 institutes. Each institute has ~10 physicists working on one or more channels.***
***2000 physicists in 31 countries are involved in this  20-year experiment in which DOE is a major player.***

***CMS detector: 15m X 15m X 22m***
***12,500 tons, $700M.***

- human=2m

- analysis

- event simulation

-  **Italian Center**

**FermiLab, USA Regional Center**

- HPSS

- Courtesy Harvey Newman, CalTech and CERN

## DOE Science Gridand the

**Grid Services:**** Uniform access to distributed resources**

**Application framework**

**compute and data management requests**

- Grid Managed Resources

**NERSC****Supercomputing****Large-Scale Storage**

**PNNL**

**LBNL**

**ANL**

**ORNL**

**SNAP**

**ESnet**

**PPDG**

**Asia-Pacific**

**Europe**

**DOE Science Grid****and the**
**Unified Science Environment**

## Grid Based Web Services Architecture

***Grid Based Web Services Architecture***

**Clients**

**Application****Portals**

**Grid Web**
**Services**

**Grid Services****Collective and Resource Access**

**Resources**

**GRAM**

**GridFTP**

**Replica Catalog /**
**Management**

**Grid****Monitoring****Architecture**

**Grid****Information****Service**

**Web Browser**

**Grid Web Service****Description (WSDL)**
**& Discovery (UDDI)**

**Grid X.509****Certification****Authority**

**SRB/MCAT**

**Condor-G**

**CORBA**

**MPI**

**Secure, ****Reliable****Group Comm.**

**Discipline /**
**Application**
**Specific****Portals**
**(e.g. SDSC****TeleScience)**

**Problem****Solving****Environments****(AVS, SciRun,****Cactus)**

**Environment****Management****(LaunchPad,****HotPage)**

**Job Submission /**
**Control **

**File Transfer **

**Data Management**

**Credential**
**Management**

**Monitoring**

**Events**

**Workflow**
**Management**

**other services:**
**visualization**
**interface builders**
**collaboration tools**
**numerical grid****generators**
**etc.**

***Apache Tomcat&WebSphere******&Cold Fusion=JVM + servlet instantiation + routing***

**CoG Kits implementing**
** ****Web Services in servlets, servers, etc.**

**Python, Java, etc.,**
**JSPs format to html**

**composition****frameworks**

**XML / SOAP over Grid Security Infrastructure**

**Grid Protocols and Grid Security Infrastructure **

**Grid Protocols and Grid Security Infrastructure **

***Apache SOAP,******.NET, etc.***

**......**

**http, https**

## Acknowledgements

- The DOE Science Grid work is funded by the U.S. Dept. of Energy, Office of Science, Office of Advanced Scientific Computing Research, Mathematical, Information, and Computational Sciences Division (http://www.sc.doe.gov/production/octr/mics) under contract DE-AC03-76SF00098 with the University of California.
- The NASA Information Power Grid related work is funded primarily by NASA’s Aero-Space Enterprise, Information Technology (IT) program (http://www.nas.nasa.gov/IT/overview.html).

## References and Acronyms

- www.globus.org provides full information about the Globus system.
- The Grid: Blueprint for a New Computing Infrastructure, edited by Ian Foster and Carl Kesselman. Morgan Kaufmann, Pub. August 1998. ISBN 1-55860-475-8. http://www.mkp.com/books_catalog/1-55860-475-8.asp
- “Grids as Production Computing Environments: The Engineering Aspects of NASA's Information Power Grid,” William E. Johnston, Dennis Gannon, and Bill Nitzberg. Eighth IEEE International Symposium on High Performance Distributed Computing, Aug. 3-6, 1999, Redondo Beach, California. (Available at http://www.ipg.nasa.gov “About IPG” -> presentations)
- The Particle Physics Data Grid has two long-term objectives. Firstly: the delivery of an infrastructure for very widely distributed analysis of particle physics data at multi-petabyte scales by hundreds to thousands of physicists. Secondly: the acceleration of the development of network and middleware infrastructure aimed broadly at data-intensive collaborative science. http://www.cacr.caltech.edu/ppdg/
- “The Data Grid: Towards an Architecture for the Distributed Management and Analysis of Large Scientific Datasets.” A. Chervenak, I. Foster, C. Kesselman, C. Salisbury, S. Tuecke, (to be published in the Journal of Network and Computer Applications).

- The Global Grid Forum (www.gridforum.org) is an informal consortium of institutions and individuals working on wide area computing and computational Grids. Current working groups include Security (authentication, authorization), Scheduling and Resource Management, Grid Information Services, Application and Tool Requirements, Advanced Programming Models, Grid User Services and Operations, Account Management, Remote Data Access, Grid Performance
- Storage Resource Broker (SRB) provides uniform access mechanism to diverse and distributed data sources. http://www.sdsc.edu/MDAS/
- Condor is a High Throughput Computing environment that can manage very large collections of distributively owned workstations. http://www.cs.wisc.edu/condor/
- Ecce - www.emsl.pnl.gov
- “A CORBA-based Development Environment for Wrapping and Coupling Legacy Codes,”  Gregory Follen, Chan Kim, Isaac Lopez, Janche Sang  and Scott Townsend. To be presented at Tenth IEEE International Symposium on High Performance Distributed Computing, August 7-9, 2001, San Francisco, California.