# Presentation: 395978

## Grid Introduction and Overview

- Ian Foster
- Argonne National Lab
- University of Chicago
- Globus Project
- www.mcs.anl.gov/~foster

- Grid Session, Lepton-Photon ’03, FermiLab, August 14, 2003

*Image Credit: Electronic Visualization Lab, UIC*

## Breaking News, August 14, 2003

## Breaking News, August 14, 2013

- computational overload, not

- data

- _ __data _

## It’s Easy to ForgetHow Different 2003 is From 1993

- Enormous quantities of data: Petabytes
- For an increasing number of communities, gating step is not collection but analysis
- Ubiquitous Internet: 100+ million hosts
- Collaboration & resource sharing the norm
- Ultra-high-speed networks: 10+ Gb/s
- Global optical networks
- Huge quantities of computing: 100+ Top/s
- Moore’s law gives us all supercomputers

## Consequence: The Emergence ofGlobal Knowledge Communities

- Teams organized around common goals
- Communities: “Virtual organizations”
- With diverse membership & capabilities
- Heterogeneity is a strength not a weakness
- And geographic and political distribution
- No location/organization possesses all required skills and resources
- Must adapt as a function of the situation
- Adjust membership, reallocate responsibilities, renegotiate resources

**Notes:**
There is an increasing need in the military to support virtual organizations: human / agent communities assembled as needed out of existing resources in order to pursue common goals.  Virtual organizations frequently must be created to carry out multinational peace keeping operations, joint force operations, or to respond to natural disasters.

Virtual organizations pose a range of problems, some of which are potentially amenable to technological solutions.  Commanders need to locate the resources that the virtual organization requires.  Elements of the VO may be part of multiple real organizations and may be distributed geographically, so proper coordination is paramount.  The composition and function of the virtual organization must change as the situation changes.  People may need help in adapting their work practices to meet the needs of the virtual organization.

## For Example: High Energy Physics

- For Example: High Energy Physics

## Resource Integrationas a Fundamental Challenge

## Grid TechnologiesAddress Key Requirements

- Infrastructure (“middleware”) for establishing, managing, and evolving multi-organizational federations
- Dynamic, autonomous, domain independent
- On-demand, ubiquitous access to computing, data, and services
- Mechanisms for creating and managing workflow within such federations
- New capabilities constructed dynamically and transparently from distributed services
- Service-oriented, virtualization

## The Grid World: Current Status

- Substantial number of Grid success stories
- Major projects in science
- Emerging infrastructure deployments
- Growing number of commercial deployments
- Open source Globus Toolkit® a de facto standard for major protocols & services
- Simple protocols & APIs for authentication, discovery, access, etc.: infrastructure
- Large user and developer base
- Multiple commercial support providers
- Global Grid Forum: community & standards
- Emerging Open Grid Services Architecture

## The Emergence ofOpen Grid Standards

- Increased functionality,
- standardization

- Custom
- solutions

- 1990

- 1995

- 2000

- 2005

- The Emergence ofOpen Grid Standards

- 2010

## Grid Evolution: OGSA (Open Grid Services Architecture)

- Goals
- Refactor Globus protocol suite to enable common base and expose key capabilities
- Service orientation to virtualize resources and unify resources/services/information
- Embrace key Web services standards, leverage commercial efforts
- Result = standard interfaces & behaviors for distributed system mgmt: the _Grid Service_
- Standardization within Global Grid Forum
- GT3 open source implementation
- _OGSA = Web services on steroids!_

## Open Grid Services Infrastructure (OGSI)

**Service registry**

**Service requestor (e.g. user application)**

**Service factory**

**Create Service**

**Grid Service Handle**

**Resource allocation **

**Service instances**

**Register Service**

**Service discovery**

- Interactions standardized using WSDL and SOAP

**Service data Keep-alives Notifications Service invocation**

**Authentication & Authorization are applied to all requests**

- Open Grid Services _Infrastructure_ (OGSI)

## OGSA Standardization & Implementation

- OGSI defines core interfaces and behaviors for manageable services
- Efforts are underway within GGF, OASIS, and other bodies to define standards for
- Agreement negotiation
- Common management model
- Data access and integration
- Security and policy
- Etc., etc., etc.
- Supported by strong open source technology & major commercial vendors

## Grid Infrastructure

- Broadly deployed services in support of fundamental collaborative activities
- Formation & operation of virtual organizations
- Authentication, authorization, discovery, ...
- Services, software, and policies enabling on-demand access to critical resources
- Computers, databases, networks, storage, software services,...
- Operational support for 24x7 availability
- Integration with campus and commercial infrastructures

## Open Infrastructure 

- Open Infrastructure

## Where We Are:What We Can Do Today

- A core set of Grid capabilities are available and distributed in good quality form, e.g.
- Globus Toolkit: security, discovery, access, data movement, etc.
- Condor: scheduling, workflow management
- Virtual Data Toolkit, NMI, EDG, etc.
- Deployed at moderate scales
- WorldGrid, TeraGrid, NEESgrid, DOE SG, EDG, ...
- Usable with some hand holding, e.g.
- US-CMS event prod.: O(6) sites, 2 months
- NEESgrid: earthquake engineering experiment

## Slide 17

## NEESgrid Earthquake Engineering Collaboratory

- U.Nevada Reno

- www.neesgrid.org

## CMS Event Simulation Production

- Production Run on the Integration Testbed
- Simulate 1.5 million full CMS events for physics studies: ~500 sec per event on 850 MHz processor
- 2 months continuous running across 5 testbed sites
- Managed by a single person at the US-CMS Tier 1

## CMS Event Simulation Production

- Production Run on the Integration Testbed
- Simulate 1.5 million full CMS events for physics studies: ~500 sec per event on 850 MHz processor
- 2 months continuous running across 5 testbed sites
- Managed by a single person at the US-CMS Tier 1

**1.5 Million Events **
**Delivered to CMS Physicists!**
**(nearly 30 CPU years)**

## Where We Are:Key Areas of Concern

- Integration with site operational procedures
- Many challenging issues
- Scalability in multiple dimensions
- Number of sites, resources, users, tasks
- Higher-level services in multiple areas
- Virtual data, policy, collaboration
- Integration with end-user science tools
- Science desktops
- Coordination of international contributions
- Integration with commercial technologies

## Summary:Grid Past, Present, Future

- Past
- Origins and broad adoption in eScience, fueled by open source Globus Toolkit
- Present
- Rapidly growing commercial adoption focused on intra-enterprise resource sharing
- Increasingly large scale infrastructures
- Open Grid Services Architecture (OGSA)
- Future
- Key enabler of new applications & industries based on resource virtualization and distributed service integration

## For More Information

- GriPhyN, iVDGL, PPDG
- www.griphyn.org, www.ivdgl.org, www.ppdg.net
- The Globus Project®
- www.globus.org
- Global Grid Forum
- www.ggf.org
- Background information
- www.mcs.anl.gov/~foster
**GlobusWORLD 2004**
**www.globusworld.org**
**Jan 20–23, San Fran**

- 2nd Edition: November 2003