# Presentation: 829446

## Data and the Grid:From Databases to Global Knowledge Communities

- Ian Foster
- Argonne National Laboratory
- University of Chicago
- www.mcs.anl.gov/~foster

- Keynote Talk, 15th Intl Conf on Scientific and Statistical Database Management, Boston, July 11, 2003

*Image Credit: Electronic Visualization Lab, UIC*

## My Presentation

- 1) Data integration as a new opportunity
- Driven by advances in technology & science
- The need to discover, access, explore, analyze diverse distributed data sources
- Grid technologies as a substrate for essential management functions
- 2) Science as collaborative workflow
- The need to organize, archive, reuse, explain, and schedule scientific workflows
- Virtual data as a unifying concept

## It’s Easy to ForgetHow Different 2003 is From 1993

- Enormous quantities of data: Petabytes
- For an increasing number of communities, gating step is not collection but analysis
- Ubiquitous Internet: 100+ million hosts
- Collaboration & resource sharing the norm
- Ultra-high-speed networks: 10+ Gb/s
- Global optical networks
- Huge quantities of computing: 100+ Top/s
- Moore’s law gives us all supercomputers

## Consequence: The Emergence ofGlobal Knowledge Communities

- Teams organized around common goals
- Communities: “Virtual organizations”
- With diverse membership & capabilities
- Heterogeneity is a strength not a weakness
- And geographic and political distribution
- No location/organization possesses all required skills and resources
- Must adapt as a function of the situation
- Adjust membership, reallocate responsibilities, renegotiate resources

**Notes:**
There is an increasing need in the military to support virtual organizations: human / agent communities assembled as needed out of existing resources in order to pursue common goals.  Virtual organizations frequently must be created to carry out multinational peace keeping operations, joint force operations, or to respond to natural disasters.

Virtual organizations pose a range of problems, some of which are potentially amenable to technological solutions.  Commanders need to locate the resources that the virtual organization requires.  Elements of the VO may be part of multiple real organizations and may be distributed geographically, so proper coordination is paramount.  The composition and function of the virtual organization must change as the situation changes.  People may need help in adapting their work practices to meet the needs of the virtual organization.

## The Emergence of Global Knowledge Communities

- The Emergence of Global Knowledge Communities

## Global Knowledge CommunitiesOften Driven by Data: E.g., Astronomy

- No. & sizes of data sets as of mid-2002,               grouped by wavelength
-   12 waveband coverage of large     areas of the sky
-   Total about 200 TB data
-   Largest catalogs near 1B objects

-                         Data and images courtesy Alex Szalay, John Hopkins

## Data Integrationas a Fundamental Challenge

## Performance Requirements Demand Whole-System Management

- Assume
- Remote data at 1 GB/s
- 10 local bytes per remote
- 100 operations per byte

- Local
- Network

- Wide area link (end-to-end
- switched lambda?) 1 GB/s

- Parallel I/O: 10 GB/s

- Parallel
- computation:
- 1000 Gop/s

- Remote
- data

- >1 GByte/s achievable today
- (FAST, 7 streams, LAGeneva)

## Data Integration: Key Challenges

- Of course, familiar issues: data organization, schema definition/mediation, etc., etc.
- But also new challenges relating to dynamic, distributed communities
- Establishment, negotiation, management, & evolution of multi-organizational federations
- And to the sheer number of resources, speed of networks, and volume of data
- Coordination, management, provisioning, & monitoring of workflows & required resources

## Enter Grid Technologies

- Infrastructure (“middleware”) for establishing, managing, and evolving multi-organizational federations
- Dynamic, autonomous, domain independent
- On-demand, ubiquitous access to computing, data, and services
- Mechanisms for creating and managing workflow within such federations
- New capabilities constructed dynamically and transparently from distributed services
- Service-oriented, virtualization

## The Emergence ofOpen Grid Standards

- Increased functionality,
- standardization

- Custom
- solutions

- 1990

- 1995

- 2000

- 2005

- The Emergence ofOpen Grid Standards

- 2010

## Open Grid Services Architecture

- Service-oriented architecture
- Key to virtualization, discovery, composition, local-remote transparency
- Leverage industry standards
- In particular, Web services
- Distributed service management
- A “component model for Web services”
- A framework for the definition of composable, interoperable services

- “The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration”, Foster, Kesselman, Nick, Tuecke, 2002

## Web Services 

- XML-based distributed computing technology
- Web service = a server process that exposes typed ports to the network
- Described by the Web Services Description Language, an XML document that contains
- Type of message(s) the service understands & types of responses & exceptions it returns
- “Methods” bound together as “port types”
- Port types bound to protocols as “ports”
- A WSDL document completely defines a service and how to access it

## OGSA Structure

- A standard substrate: the Grid service
- Standard interfaces and behaviors that address key distributed system issues: naming, service state, lifetime, notification
- A Grid service is a Web service
- ... supports standard service specifications
- Agreement, data access & integration, workflow, security, policy, diagnostics, etc.
- Target of current & planned GGF efforts
- ... and arbitrary application-specific services based on these & other definitions

## Open Grid Services Infrastructure

- Implementation

- Hosting environment/runtime
- (“C”, J2EE, .NET, ...)

- Data
- access

- Lifetime management
-  Explicit destruction
-  Soft-state lifetime

## Open Grid Services Infrastructure

- GWD-R (draft-ggf-ogsi- gridservice-23) 		**Editors:**
- Open Grid Services Infrastructure (OGSI) 		S. Tuecke, ANL
- http://www.ggf.org/ogsi-wg 			K. Czajkowski, USC/ISI
- 						I. Foster, ANL
- 						J. Frey, IBM
- 						S. Graham, IBM
- 						C. Kesselman, USC/ISI
- 						D. Snelling, Fujitsu Labs
- 						P. Vanderbilt, NASA
- 						February 17, 2003
**Open Grid Services Infrastructure (OGSI)**

- “The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration”, Foster, Kesselman, Nick, Tuecke, 2002

## Example:Reliable File Transfer Service

- File
- Transfer

*Internal*
*State*

*Data transfer operations*

## OGSA and Data Integration

- OGSI provides key enabling mechanisms for distributed data integration
- Introspect on distributed system elements
- Create and manage distributed state
- We need more than OGSI, of course, e.g.,
- WS-Agreement: negotiate agreements between service provider and consumer
- OGSA-DAI: Data Access and Integration
- WS-Management: service management
- Security and policy

## Infrastructure Architecture

- OGSI: Interface to Grid Infrastructure

- Data Intensive Applications for X-ology Research

- Compute, Data & Storage Resources

- Distributed

- Simulation, Analysis & Integration Technology for X-ology

- Data Intensive X-ology Researchers

**Virtual Integration Architecture**

*Generic* Virtual Data Access and Integration Layer

-                         Slide Courtesy Malcolm Atkinson, UK eScience Center

## Data as Service:OGSA Data Access & Integration

- Service-oriented treatment of data appears to have significant advantages
- Leverage OGSI introspection, lifetime, etc.
- Compatibility with Web services
- Standard service interfaces being defined
- Service data: e.g., schema
- Derive new data services from old (views)
- Externalize to e.g. file/database format
- Perform queries or other operations

## Data Access & Integration Services

-                         Slide Courtesy Malcolm Atkinson, UK eScience Center

## Globus Toolkit v3 (GT3)Open Source OGSA Technology

- Implements and builds on OGSI interfaces
- Supports primary GT2 interfaces
- Public key authentication
- Scalable service discovery
- Secure, reliable resource access
- High-performance data movement (GridFTP)
- Numerous new services included or planned
- SLA negotiation, service registry, community authorization, data access & integration, ...
- Rapidly growing adoption and contributions
- E.g., OGSA-DAI from U.K. eScience program

## My Presentation

- 1) Data integration as a new opportunity
- Driven by advances in technology & science
- The need to discover, access, explore, analyze diverse distributed data sources
- Grid technologies as a substrate for essential management functions
- 2) **Science as collaborative workflow**
**The need to organize, archive, reuse, explain, & schedule scientific workflows**
**Virtual data as a unifying concept**

## Science as Workflow

- Data integration = the derivation of new data from old, via coordinated computation(s)
- May be computationally demanding
- The workflows used to achieve integration are often valuable artifacts in their own right
- Thus we must be concerned with how we
- Build workflows
- Share and reuse workflows
- Explain workflows
- Schedule workflows

- Sloan Digital Sky Survey Production System

## Virtual Data Concept

- Capture and manage information about relationships among
- Data (of widely varying representations)
- Programs (& their execution needs)
- Computations (& execution environments)
- Apply this information to, e.g.
- Discovery: Data and program discovery
- Workflow: Structured paradigm for organizing, locating, specifying, & requesting data
- Explanation: provenance
- Planning and scheduling
- Other uses we haven’t thought of

## Motivations

- Transformation

- Derivation

- Data

*created-by*

*execution-of*

*consumed-by/*
*generated-by*

- “I’ve detected a calibration error in an instrument and want to know which derived data to recompute.”

- “I’ve come across some interesting data, but I need to understand the nature of the corrections applied when it was constructed before I can trust it for my purposes.”

- “I want to search an astronomical database for galaxies with certain characteristics. If a program that performs this analysis exists, I won’t have to write one from scratch.”

- “I want to apply an astronomical analysis program to millions of objects. If the results already exist, I’ll save weeks of computation.”

## Virtual data catalog
Transformations, derivations, data
Virtual data language
Catalog definitions
Query tool
Applications include browsers and data analysis applications

- Transformations, derivations, data
- Virtual data language
- Catalog definitions
- Query tool
- Applications include browsers and data analysis applications

- Chimera Virtual Data System(www.griphyn.org/chimera)

## Chimera Virtual Data Schema

**Metadata**

- describes

- describes

## Virtual Data in CMS HEP Analysis

**mass = 200**
**decay = WW**
**stability = 1**
**event = 8**

**mass = 200**
**decay = WW**
**stability = 1**
**plot = 1**

**mass = 200**
**decay = WW**
**plot = 1**

**mass = 200**
**decay = WW**
**event = 8**

**mass = 200**
**decay = WW**
**stability = 1**

**mass = 200**
**decay = WW**
**stability = 3**

**mass = 200**

**mass = 200**
**decay = WW**

**mass = 200**
**decay = ZZ**

**mass = 200**
**decay = bb**

**mass = 200**
**event = 8**

-       Work and slide by Rick Cavanaugh and Dimitri Bourilkov, University of Florida

- Define a virtual data space for exploration by other scientists

**mass = 200**
**plot = 1**

-  **Knowledge capture**

## Virtual Data in CMS HEP Analysis

**mass = 200**
**decay = WW**
**stability = 1**
**event = 8**

**mass = 200**
**decay = WW**
**stability = 1**
**plot = 1**

**mass = 200**
**decay = WW**
**plot = 1**

**mass = 200**
**decay = WW**
**event = 8**

**mass = 200**
**decay = WW**
**stability = 1**

**mass = 200**
**decay = WW**
**stability = 3**

**mass = 200**

**mass = 200**
**decay = WW**

**mass = 200**
**decay = ZZ**

**mass = 200**
**decay = bb**

**mass = 200**
**event = 8**

-       Work and slide by Rick Cavanaugh and Dimitri Bourilkov, University of Florida

- Search for
- WW decays of
- the Higgs Boson
- for which only
- stable, final state
- particles are
- recorded?**    **
**stability = 1**

**mass = 200**
**plot = 1**

-  Knowledge capture
-  **On-demand data gen**
** ****Workload mgmt**

## Virtual Data in CMS HEP Analysis

**mass = 200**
**decay = WW**
**stability = 1**
**event = 8**

**mass = 200**
**decay = WW**
**stability = 1**
**plot = 1**

**mass = 200**
**decay = WW**
**plot = 1**

**mass = 200**
**decay = WW**
**event = 8**

**mass = 200**
**decay = WW**
**stability = 1**

**mass = 200**
**decay = WW**
**stability = 3**

**mass = 200**

**mass = 200**
**decay = WW**

**mass = 200**
**decay = ZZ**

**mass = 200**
**decay = bb**

**mass = 200**
**event = 8**

-       Work and slide by Rick Cavanaugh and Dimitri Bourilkov, University of Florida

- Search for
- WW decays of
- the Higgs Boson
- and where only
- stable, final state
- particles are
- recorded:**    **
**stability = 1**

- Scientist discovers
- an interesting result – wants to know how
- it was derived.

**mass = 200**
**plot = 1**

-  Knowledge capture
-  On-demand data gen.
-  Workload mgmt
-  **Explain provenance**

## Virtual Data in CMS HEP Analysis

**mass = 200**
**decay = WW**
**stability = 1**
**event = 8**

**mass = 200**
**decay = WW**
**stability = 1**
**plot = 1**

**mass = 200**
**decay = WW**
**plot = 1**

**mass = 200**
**decay = WW**
**event = 8**

**mass = 200**
**decay = WW**
**stability = 1**

**mass = 200**
**decay = WW**
**stability = 3**

**mass = 200**

**mass = 200**
**decay = WW**

**mass = 200**
**decay = ZZ**

**mass = 200**
**decay = bb**

**mass = 200**
**plot = 1**

**mass = 200**
**event = 8**

-       Work and slide by Rick Cavanaugh and Dimitri Bourilkov, University of Florida

- Search for
- WW decays of
- the Higgs Boson
- and where only
- stable, final state
- particles are
- recorded:**    **
**stability = 1**

- Scientist discovers
- an interesting result – wants to know how
- it was derived.

-  Knowledge capture
-  On-demand data gen.
-  Workload mgmt
-  Explain provenance
-  **Collaboration**

## Virtual Data “Explorations” Can be Long-Lived Computations

- Virtual Data “Explorations” Can be Long-Lived Computations

- Production Run on the Integration Testbed
- Simulate 1.5 million full CMS events for physics studies: ~500 sec per event on 850 MHz processor
- 2 months continuous running across 5 testbed sites
- Managed by a single person at the US-CMS Tier 1

## Virtual Datain Sloan Galaxy Cluster Analysis

- Galaxy cluster
- size distribution

- DAG

- Virtual Datain Sloan Galaxy Cluster Analysis

- Sloan Data

- Jim Annis, Steve Kent, Vijay Sehkri, Fermilab, Michael Milligan, Yong Zhao, Chicago

## Virtual Data in Genome Analysis

**DOESG Resource**

## Bringing it All Together:A Virtual Data Grid

## Also Very Relevant:Workflow & Web Services

**EWF**

***Abstract Task***
***(AT) Repository***

***Data & Parameter   Ontologies***

** ****ET**

**ET**

***Genbank***

***BLAST***

*query *
*rewriting*

*web service*
*invocation*

***Executable Task***
***(ET) Repository***

*web service*
*invocation*

*semantic type*
*checking*

*data type*
*conversion*

***Datatype & Conversion Repository***

*web service*
*matching*

- B. Ludäscher, I. Altintas, A. Gupta – http://kbi.sdsc.edu/SciDAC-SDM/scidac-tn-02-01.pdf

- Also Very Relevant:Workflow & Web Services

## Summary

- 1) Data integration as a new opportunity
- Driven by advances in technology & science
- The need to discover, access, explore, analyze diverse distributed data sources
- Grid technologies as a substrate for essential management functions
- 2) Science as collaborative workflow
- The need to organize, archive, reuse, explain, and schedule scientific workflows
- Virtual data as a unifying concept

## The Globus ProjectTM
www.globus.org
Technical articles
www.mcs.anl.gov/~foster
Open Grid Services Arch.
www.globus.org/ogsa
Chimera
www.griphyn.org/chimera
Global Grid Forum
www.ggf.org

- www.globus.org
- Technical articles
- www.mcs.anl.gov/~foster
- Open Grid Services Arch.
- www.globus.org/ogsa
- Chimera
- www.griphyn.org/chimera
- Global Grid Forum
- www.ggf.org

- For More Information

- 2nd Edition: November 2003