# Presentation: 215450

## Computational Infrastructures for ScienceMarty HumphreyAssistant ProfessorComputer Science DepartmentUniversity of VirginiaNeSSI WorkshopOctober 13, 2003

**Computational Infrastructures for Science**Marty HumphreyAssistant ProfessorComputer Science DepartmentUniversity of VirginiaNeSSI WorkshopOctober 13, 2003

**Notes:**

today i’ll be talking about the work done on wide-area data access in the context of the legion probject with my advisor Andrew Grimshaw and Anh Nguyen-Tuong.

## “Traditional” Computational Science

- SP3, O2K, Linux clusters, etc.
- PBS, LSF, LoadLeveler, etc.
- Archival storage
- MPI
- Viz
_*SSH, SCP*_

## Grid Definition (Foster and Kesselman)

- “Coordinates resources that are not subject to centralized control....”
- “Using standard, open, general-purpose protocols and interfaces...”
- “To deliver non-trivial qualities of service.”

## Grid Computing

- Host/OS 1,1

- Host/OS 2,1

- Host/OS 3,1

## Grid User Wish-List

- Who cares *where* it is?
- It must always be available when I need it
- Make it secure
- no one can steal my data
- no one can pretend to be me
- don’t tell me who I will/can trust
- Choose secure, fast, cheap resources
- Give me reasonable quality of service

- Don’t make me manually move/copy stuff around
- Don’t make me learn a new OS
- Allow me to run my existing apps
- I don’t want errors
- If errors occur, tell me in plain English how I can avoid them next time
- Allow me to more easily collaborate

*Darnit, make my life easier !*

## Example: Transparent Remote Execution

- User initiates “run”
- User/Grid SW selects site/resource
- Grid SW copies binaries (if necessary)
- Grid SW copies/moves input files
- Grid SW starts job(s)
- Grid SW monitors progress
- Grid SW copies output files
*Forms the basis of parameter-space or monte carlos*

## Web Interface for CHARMM and  Amber

## Status of CHARMM or Amber Run

## Grid Focus: Virtual Organizations

- Logical grouping of resources and users
- Support community-specific discovery
- Specialized “views”
- Dynamic collaborations of individuals and institutions
_*Policy negotiation and enforcement*_ *will be key issues looking forward*

## Grid Landscape Today: Globus

- Grid Resource Allocation and Management (GRAM)
- Gatekeeper, Jobmanager (RSL  “schedulerspeak”)
- Grid Security Infrastructure (GSI)
- Metacomputing Directory Service (MDS) (via OpenLDAP)
- Grid Index Information Service (GIIS)
- Grid Resource Information Service (GRIS)
- GridFTP

## Grid Landscape Today: Globus (cont.)

- “Add-ons”:
- MPICH-G2
- Replica Catalog and Management
- Community Authorization Service (CAS)
- Condor-G
- etc.
- Basis of _many_ large-scale Grids...

## Selected Major Grid Projects (Oct 2001)

| Name | URL & Sponsors | Focus |
| --- | --- | --- |
| Access Grid | www.mcs.anl.gov/FL/accessgrid; DOE, NSF | Create & deploy group collaboration systems using commodity technologies  |
| BlueGrid | IBM | Grid testbed linking IBM laboratories |
| DISCOM | www.cs.sandia.gov/discomDOE Defense Programs | Create operational Grid providing access to resources at three U.S. DOE weapons laboratories |
| DOE Science Grid | sciencegrid.org
DOE Office of Science | Create operational Grid providing access to resources & applications at U.S. DOE science laboratories & partner universities |
| Earth System Grid (ESG) | earthsystemgrid.orgDOE Office of Science | Delivery and analysis of large climate model datasets for the climate research community |
| European Union (EU) DataGrid  | eu-datagrid.org
European Union | Create & apply an operational grid for applications in high energy physics, environmental science, bioinformatics |

*New*

*New*

## Selected Major Grid Projects

| Name | URL/Sponsor | Focus |
| --- | --- | --- |
| EuroGrid, Grid Interoperability (GRIP) | eurogrid.org
European Union | Create tech for remote access to supercomp resources & simulation codes; in GRIP, integrate with Globus ToolkitTM |
| Fusion Collaboratory | fusiongrid.org
DOE Off. Science | Create a national computational collaboratory for fusion research |
| Globus ProjectTM | globus.org
DARPA, DOE, NSF, NASA, Msoft | Research on Grid technologies; development and support of Globus ToolkitTM; application and deployment |
| GridLab | gridlab.org
European Union | Grid technologies and applications |
| GridPP | gridpp.ac.uk
U.K. eScience | Create & apply an operational grid within the U.K. for particle physics research |
| Grid Research Integration Dev. & Support Center | grids-center.org
NSF | Integration, deployment, support of the NSF Middleware Infrastructure for research & education |

*New*

*New*

*New*

*New*

*New*

## Selected Major Grid Projects

| Name | URL/Sponsor | Focus |
| --- | --- | --- |
| Grid Application Dev. Software | hipersoft.rice.edu/grads; NSF | Research into program development technologies for Grid applications |
| Grid Physics Network | griphyn.org
NSF | Technology R&D for data analysis in physics expts: ATLAS, CMS, LIGO, SDSS |
| Information Power Grid | ipg.nasa.gov
NASA | Create and apply a production Grid for aerosciences and other NASA missions |
| International Virtual Data Grid Laboratory | ivdgl.org
NSF | Create international Data Grid to enable large-scale experimentation on Grid technologies & applications |
| Network for Earthquake Eng. Simulation Grid | neesgrid.org
NSF | Create and apply a production Grid for earthquake engineering |
| Particle Physics Data Grid | ppdg.net
DOE Science | Create and apply production Grids for data analysis in high energy and nuclear physics experiments |

*New*

*New*

## Selected Major Grid Projects

| Name | URL/Sponsor | Focus |
| --- | --- | --- |
| TeraGrid		 | teragrid.org
NSF | U.S. science infrastructure linking four major resource sites at 40 Gb/s  |
| UK Grid Support Center | grid-support.ac.uk
U.K. eScience | Support center for Grid projects within the U.K. |
| Unicore | BMBFT | Technologies for remote access to supercomputers |

*New*

*New*

## PetaScale Virtual-Data Grids

***Virtual Data Tools***

**Request Planning &**

**Scheduling Tools**

**Request Execution &**

**Management Tools**

***Transforms***

**Distributed resources****(code, storage, CPUs,****networks)**

- Resource

- Management

- Services

**Resource **

**Management **

**Services**

- Security and

- Policy

- Services

**Security and**

**Policy **

**Services**

- Other Grid

- Services

**Other Grid**

**Services**

***Interactive User Tools***

**Production Team**

**Individual Investigator**

**Workgroups**

**Raw data**

**source**

**~1 Petaflop**
**~100 Petabytes**

***Slide courtesy ***
*** ******of Paul Avery***

## Data Grid Architecture 

- Application

- Planner

- Executor

- Catalog Services

- Info Services

- Policy/Security

- Monitoring

- Repl. Mgmt.

- Reliable Transfer
- Service

- Compute Resource

- Storage Resource

- DAG

- DAG

***Slide courtesy ***
***of Ian Foster***

## US-iVDGL Data Grid

**Partners?**
- EU
- CERN
- Brazil
- Australia
- Korea
- Japan

***Slide courtesy ***
***of Paul Avery***

## Data Grids for High Energy Physics

## Global Grid Forum (GGF)

- Grid standards
- Best practices
- Broad academic, national lab, and industry involvement
- _Areas_: Applications and programming environments, architecture, data, information systems and performance, Peer-to-Peer, Scheduling and Resource Management, Security
- GGF9 was last week in Chicago

## Many Excellent DOE Grid and Middleware Projects

- Reliable and Secure Group Communication
- Commodity Grid Kits (CoGKits)
- Middleware for Science Portals
- Scientific Annotation Middleware (SAM)
- Storage Resource Management for Data Grid Applications
- Common Component Architecture (CCA)
- Scalable Software Initiative

## Next-Generation Grids

- Web Services
- “Semantically encapsulate discrete functionality”
- Loosely coupled, reusable components
- XML, SOAP, WSDL, UDDI, etc.
- Broad industrial support: Microsoft, IBM, Sun, BEA, etc.
- _Open Grid Services Architecture (OGSA)_
- Combine Grids (Globus, Legion) with Web Services
- GT3: Java, AXIS, J2EE, etc.

## OGSI.NET

- University of Virginia hosting environment for Grid Services based on Microsoft Web Services approach
- Focus: Grid security (e.g., explicit trust management)
- Focus: Grid programming models
- Focus: Connection between UNIX and Win*

## Biomolecular VO based on OGSI.NET

## Grid Challenges: “UK E-Science Gap Analysis”(Fox and Walker, Jun 30 2003)

- Security: VPNs/Firewalls, fine-grain access control
- Workflow (“orchestration”) specs and engines
- Fault tolerance
- Grid adaptability (e.g., real-time support)
- Ease of use
- Grid federations

## Future Directions

- Grid has come a long way
- Merging of Grid and Web Services shows promise
- Many difficult issues remain
- Manageable security
- Integration with legacy applications/tools
- Challenge for SNS: Identify and meet requirements not being met by current Grid technologies