# Presentation: 084116

## Third International Conference on e-Social ScienceService-Oriented Science:Globus Software In Action

- Third International Conference on e-Social ScienceService-Oriented Science:Globus Software In Action

- Ian Foster and Lee LimingUniversity of Chicago • Argonne National Laboratory

## Outline

- http://www.mcs.anl.gov/~liming/sos-solutions/

- Scaling e-science impact
- WSRF and Globus Toolkit 4.0
- Three case studies
- Data replication for LIGO
- Scientific workflow in a computational economics problem
- Distributed information service for TeraGrid
- General lessons and patterns

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## The Big Questions

- Nature of the universe

- Consciousness

- Life & death

- Future of the planet

- Service-Oriented Science: Globus Software in Action

- eSS 2007

**Notes:**
Life and nature: structures, processes and interactions
Matter and universe: origins, structure, manipulation & futures; interactions, systems and context
Humanity: creativity, socialization and community

## How Do We Answer Them?

- <0

- 1700

- 1950

- 1990

- Service-Oriented Science: Globus Software in Action

- eSS 2007

**Notes:**
Aristotle, Gottfried Leibniz, John von Neuman, Jim Gray

Theory=math
Simulation = math + CS
Data = bits and algorithms, i.e., CS.

Different disciplines advanced to different stages: e.g., parking still empirical (-:

## The Same is True of Smaller Questions

- Designing new chemical catalysts
- Selling advertising
- Creating entertainment
- Finding parking

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Information Technology and Science

- “Paul Erdös claimed that a mathematician is a machine for turning coffee into theorems. The scientist is arguably a machine for turning data into insight.”
**Service-Oriented Science, **I. Foster, *Science*, 308, p. 814.

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## What are the Products of Science?

- Papers
- “We learned this, and here’s how.”
- Data and Datasets
- “We collected this data. Download it, write an analysis program, and see what you can learn from it.”
- Web Portals
- “We constructed this scientific model. Use our data or bring your own, supply some parameters, and see how it behaves.”
- Requires manual operation.
- Web Services
- “Here’s our climate model. Integrate it with your models for [ocean currents/weather/crop forecasts] and see what happens.”
- “Here’s our indexed data from the latest experiment run. Run your filters against it and see if you can find anything interesting.”
- “Here’s our genome analysis engine. Upload your proteins and see what they will do in a cell.”

- Increasing degrees of collaboration

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Grid:An Enabler of eScience

- Grid:An Enabler of eScience

-    Must we buy (or travel to) a power source?

-    Or can we ship power to where we want to work?

- Enable on-demand access to, and integration of,
- diverse resources & services, regardless of location

**The dubious electrical power grid analogy**

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## 1st Generation Grids

-   Focus on aggregation of many resources for massively (data-)parallel applications

- EGEE

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Second-Generation Grids

- Empower many more users by enabling on-demand access to **services**
- Science gateways (TeraGrid)
- Service oriented science
- Or, “Science 2.0”

- “Service-Oriented Science”, *Science*, 2005

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## “Web 2.0”

- Software as services
- Data- & computation-richnetwork services
- Services as platforms
- Easy composition of services to create new capabilities (“mashups”)—that themselves may be made accessible as new services
- Enabled by massive infrastructure buildout
- Google projected to spend $1.5B on computers, networks, and real estate in 2006
- Many others are spending substantially
- Paid for by advertising

- Declan
- Butler,
- Nature

- Service-Oriented Science: Globus Software in Action

- eSS 2007

**Notes:**
Services
Composition
Infrastructure

## Automating Science

- Human access to data is nice
- Automated access by software tools is revolutionary
- “In the time that a human user takes to locate one useful piece of information within a Web site, a program may access and integrate data from many sources and identify relationships that a human might never discover unaided.” - Foster

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Science 2.0:E.g., Virtual Observatories

**Data Archives**

- Gateway

- Figure: S. G. Djorgovski

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Service-Oriented Science

- People **create** services (data or functions) ...
- which I **discover** (& decide whether to use) ...
- & **compose** to create a new function ...
- & then **publish** as a new service.
-  *I find “someone else” to ****host**** services, **so I don’t have to become an expert in operating* *services & computers!*
-  *I hope that this “someone else” can ****manage**** security, reliability, scalability, ...*

- !

- !

- “Service-Oriented Science”, *Science*, 2005

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Are Scientists Really Developing Web Services?

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Cancer Bioinformatics Grid

- Common system architecture (caGrid) provides the service interface “plumbing” and the service hosting capability
- Web services
- Community participants supply useful services
- Data access, analysis, modeling, filtering, authoring, etc.
- https://cabig.nci.nih.gov/

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## The Introduce Authoring Tool

- Define service
- Create skeleton
- Discover types
- Add operations
- Configure security
- Modify service

**Introduce**: Hastings, Saltz, et al., Ohio State University

- Generates GT4-compatible WebServices

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## The Importance of “Hosting”and “Management”

*Tell me about*
*this star*

- Tell me about
- these 20K stars

- Support 1000sof users

- E.g., Sloan DigitalSky Survey, ~10 TB;
- others much bigger

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Skyserver Sessions(Thanks to Alex Szalay)

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Hosting & Management:Application Hosting Services

**Hosting** & **Management**:Application Hosting Services

- Resource
- Provider

- Appln
- Code

- Appln
- Code

**Application client**

**AHS****management**

- Hosting Service

**Author ization **

- Resource
- Provider

**Provisioning**

**Persistence**

*Users*

*Admins*

**PDP**

**Policy****management**

**Application deployment**

- Application
- Prep Tool(s)

- Appln
- Code

*Application*
*providers*

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Who Will Host Your Services?

- Your institution (campus resources)
- (Inter)national systems
- TeraGrid, Open Science Grid, UK Nat’l Grid Service, ChinaGrid, NaukaGrid, etc.
- Science domain systems
- caBIG, NEES, Earth System Grid, Orion*, LEAD, NEON*, LHC Computing Grid, etc.
- Commercial systems
- Amazon, Google, etc.

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Examples ofProduction Scientific Grids

- Examples ofProduction Scientific Grids

- APAC (Australia)
- China Grid
- China National Grid
- DGrid (Germany)
- EGEE
- NAREGI (Japan)
- Open Science Grid
- Taiwan Grid
- TeraGrid
- ThaiGrid
- UK Nat’l Grid Service

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Application-Infrastructure Gap

** ****Dynamic****and/or****Distributed****Applications**

- Service-Oriented Science: Globus Software in Action

- eSS 2007

**Notes:**
Talk about “silos”
Two fundamental problems in “the gap”: (a) for applications that have distribution, how do you coordinate the resources that this application needs to call upon (thus, access protocols, reservation, QoS, security); (b) how do you do this without dedicating resources. Hard enough to build reliable distributed systems with dedicated resources. (Thus, we see many people dedicating resources because they don’t trust the robustness of the sharing mechanisms—will the resources I need really be available when I need them, in a useful state?.) AND, is the resource in a form that will be useful to me? (Right version of an OS, right bandwidth, enough space?)

## Bridging the Application-Resource Gap

- Bridging the Application-Resource Gap

- Computers

- Storage

- Specialized
-  resource

- User
- Application

- Database

- Service-Oriented Science: Globus Software in Action

- eSS 2007

**Notes:**
Not just WSDLs on things, but common abstractions that apply across many resources and services. (A work in progress.)

## Grid Infrastructure

- Distributed management
- Of physical resources
- Of software services
- Of communities and their policies
- Unified treatment
- Build on Web services framework
- Use WS-RF, WS-Notification (or WS-Transfer/Man) to represent/access state
- Common management abstractions & interfaces

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Web Services, WSRF, and Globus

- What can they do for me?

## “Stateless” vs. “Stateful” Services

- “Stateless” vs. “Stateful” Services

- Without state, how does client:
- Determine what happened (success/failure)?
- Find out how many files completed?
- Receive updates when interesting events arise?
- Terminate a request?
- Few useful services are truly “stateless”, but WS interfaces alone do not provide built-in support for state

**Client**

**FileTransfer****Service**

**move (A to B)**

- move

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## FileTransferService (without WSRF)

- Developer reinvents wheel for each new service
- Custom management and identification of state: **transferID**
- Custom operations to inspect state synchronously (**whatHappen**) and asynchronously (**tellMeWhen**)
- Custom lifetime operation (**cancel**)

**Client**

**FileTransfer****Service**

**move (A to B) : ****transferID**

- move

- state

- whatHappen

- tellMeWhen

- cancel

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## WSRF in a Nutshell

- Service
- State representation
- Resource
- Resource Property
- State identification
- Endpoint Reference
- State Interfaces
- GetRP, QueryRPs, GetMultipleRPs, SetRP
- Lifetime Interfaces
- SetTerminationTime
- ImmediateDestruction
- Notification Interfaces
- Subscribe
- Notify
- ServiceGroups

**Service**

**GetRP**

**GetMultRPs**

**SetRP**

**QueryRPs**

**Subscribe**

**SetTermTime**

**Destroy**

- EPR

- EPR

- EPR

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## FileTransferService (w/ WSRF)

- Developer specifies custom method to createResource and leaves the rest to WSRF standards:
- State exposed as Resource + Resource Properties and identified by Endpoint Reference (EPR)
- State inspected by standard interfaces (GetRP, QueryRPs)
- Lifetime management by standard interfaces (Destroy)

**Client**

**FileTransferService**

**createResource (A to B) : ****EPR**

- createResource

**getRP**

**queryRPs**

**destroy**

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## The Introduce Authoring Tool

- Define service
- Create skeleton
- Discover types
- Add operations
- Configure security
- Modify service

**Introduce**: Hastings, Saltz, et al., Ohio State University

- Generates GT4-compatible WebServices

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Globus Philosophy

- Globus was first established as an open source project in 1996
- The Globus Toolkit is open source to:
- Allow for inspection
- for consideration in standardization processes
- Encourage adoption
- in pursuit of ubiquity and interoperability
- Encourage contributions
- harness the expertise of the community
- The Globus Toolkit is distributed under the (BSD-style) Apache License version 2

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## dev.globus 

- Governance model based on Apache Jakarta
- Consensus based decision making
- Globus software is organized as several dozen “Globus Projects”
- Each project has its own “Committers” responsible for their products
- Cross-project coordination through shared interactions and committers meetings
- A “Globus Management Committee”
- Overall guidance and conflict resolution

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## http://dev.globus.org

- http://**dev.globus**.org

- Guidelines(Apache
- Jakarta)
- Infrastructure(CVS, email,bugzilla, Wiki)
- Projects
- Include
**...**

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Open Source != “Free time”

- Globus development is well-funded
- The open source model facilitates contributions
- NSF and DOE sponsor Globus development at several institutions via multiple grants, totaling >$5M/yr
- Non-U.S. science agencies also contribute to Globus development
- Corporations also sponsor developers
- NSF explicitly funds Globus improvements
- CDIGS: Community-Driven Improvements to Globus Software

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Globus Technology Areas

- Core runtime
- Infrastructure for building new services
- Security
- Apply uniform policy across distinct systems
- Execution management
- Provision, deploy, & manage services
- Data management
- Discover, transfer, & access large data
- Monitoring
- Discover & monitor dynamic services

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Incubator

**Incubator**
**Projects**

**Security**

**Execution**
**Mgmt**

**Info**
**Services**

**Common**
**Runtime**

**Globus Projects**

**Other**

- MPICH G2

- GridWay

**Data Mgmt**

- Incubation
- Mgmt

- Cog WF

- LRMA

- GAARDS

- OGRO

- GDTE

- UGP

- HOC-SA

- PURSE

- GridShib

- Introduce

- Dyn Acct

- WEEP

- Gavia JSC

- Gavia MS

- DDM

- Virt WkSp

- SGGC

- Metrics

- ServMark

- GridFTP

- Reliable
- File
- Transfer

- OGSA-DAI

- GRAM

- MDS4

- CAS

- Data
- Rep

- Delegation

- Replica
- Location

- Java Runtime

- C Runtime

- Python Runtime

- C Sec

- GT4 Docs

- MEDICUS

- GSI-
- OpenSSH

- MyProxy

- Swift

- MonMan

- NetLogger

- GEMLCA

**Globus****Toolkit**

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## What Is the Globus Toolkit?

- The Globus Toolkit is a collection of solutions to problems that frequently come up when trying to build collaborative distributed applications
- Heterogeneity
- To date (v1.0 - v4.0), the Toolkit has focused on *simplifying heterogeneity* for application developers
- We are increasingly including more “vertical solutions” that implement typical application patterns
- Security
- The Grid Security Infrastructure (GSI) allows collaborators to share resources without blind trust
- Standards
- Our goal has been to capitalize on and encourage use of existing standards (IETF, W3C, OASIS, GGF)
- The Toolkit also includes reference implementations of new/proposed standards in these organizations

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## What’s In the Globus Toolkit?

- A Grid development environment
- Develop new OGSA-compliant Web Services
- Develop applications using Java or C/C++ Grid APIs
- Secure applications using basic security mechanisms
- A set of basic Grid services
- Job submission/management
- File transfer (individual, queued)
- Database access
- Data management (replication, metadata)
- Monitoring/Indexing system information
- Tools and Examples
- The prerequisites for many Grid community tools

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Defining Community: Membership and Laws

- Identify VO participants and roles
- For people and services
- Specify and control actions of members
- Empower members  delegation
- Enforce restrictions  federate policy

- Access granted by community to user

- Site admission-control policies

- Effective
- Access

- Policy of site to community

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## XACML

- Courtsey : DOE report : LBNL-41349 : Authorization & Attribute Certificates for Widely Distributed Access Control

- SAML

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Globus Authorization Framework

- VOMS

- Shibboleth

- LDAP

- PERMIS

- ...

- GT4 Client

- GT4 Server

- PDP

- Attributes

- Authorization
- Decision

- PIP

- PIP

- PIP

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Case Study 1:Data Replication for LIGO

- Scott KorandaAnn Chervenak

## Laser Interferometer Gravitational Wave Observatory

- Goal: Observe gravitational waves predicted by theory
- Three physical detectors in two locations (plus GEO detector in Germany)
- 10+ data centers for data analysis
- Collaborators in ~40 institutions on at least three continents

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## LIGO by the Numbers

- LIGO records thousands of channels, generating approx. 1 TB per day of data during a detection run
- Data is published and data centers subscribe to portions that local users want for analysis or for local storage
- Data analysis results in derived data
- Currently ~30% of all LIGO data
- This also is published and replicated
- Over 30 million files in LDR network (April 2005)*
- 6+ million unique logical files
- 30+ million physical copies of those files

- * Scott Koranda, UW-Milwaukee, April 2005

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## The Challenge

- Replicate 1 TB/day of data to 10+ international sites
- Publish/subscribe (pull) model
- Provide scientists with the means to specify and discover data based on application criteria (metadata)
- Provide scientists with the means to locate copies of data

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Issues - Technical

- Efficiency
- Avoid unused bandwidth while data transfers are taking place, esp. on high-bandwidth links (10+ Gbps)
- Avoid idle time on the network between transfers

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Issues - Social (1)

- Workflow
- The publish/subscribe model matches how scientists think about their use of the data
- Use of metadata is critical
- Security
- Authenticate endpoints of data transfers to prevent unauthorized data manipulation
- Heterogeneity
- Can’t tell data centers what storage systems to use
- Can’t get everyone to do accounts the same way

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Issues - Social (2)

- Maintenance
- LIGO is focused on science, not IT
- Ideally, they would not have to build or maintain data management software
- GOAL: If software must be produced, keep it simple and do it in a way that non-CS people can understand
- GOAL: Produce a solution that others will adopt and maintain for them

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## GridFTP

- A high-performance, secure data transfer service optimized for high-bandwidth wide-area networks
- FTP with extensions
- Uses basic Grid security (control and data channels)
- Multiple data channels for parallel transfers
- Partial file transfers
- Third-party (direct server-to-server) transfers
- GGF recommendation GFD.20

- Basic Transfer
- One control channel, several parallel data channels

- Third-party
- Transfer
- Control channels to each server, several parallel data channels between servers

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Striped GridFTP

- GridFTP supports a striped (multi-node) configuration
- Establish control channel with one node
- Coordinate data channels on multiple nodes
- Allows use of many NICs in a single transfer
- Requires shared/parallel filesystem on all nodes
- On high-performance WANs, aggregate performance is limited by filesystem data rates

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## globus-url-copy

- Command-line client for GridFTP servers
- Text interface
- No “interactive shell” (single command per invocation)
- Many features
- Grid security, including data channel(s)
- HTTP, FTP, GridFTP
- Server-to-server transfers
- Subdirectory transfers and lists of transfers
- Multiple parallel data channels
- TCP tuning parameters
- Retry parameters
- Transfer status output

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## RFT - File Transfer Queuing

- A WSRF service for queuing file transfer requests
- Server-to-server transfers
- Checkpointing for restarts
- Database back-end for failovers
- Allows clients to request transfers and then “disappear”
- No need to manage the transfer
- Status monitoring available if desired

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## RLS - Replica Location Service

- A distributed system for tracking replicated data
- Consistent local state maintained in Local Replica  Catalogs (LRCs)
- Collective state with relaxed consistency maintained  in Replica Location Indices (RLIs)
- Performance features
- Soft state maintenance of RLI state
- Compression of state updates
- Membership and partitioning information maintenance

- Simple Hierarchy
- The most basic
- deployment of RLS

- Fully Connected
- High availability of
- the data at all sites

- Tiered Hierarchy
- For very large
- systems and/or very
- Large collections

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## pyGlobus*

- High-level, object-oriented interface in Python to GT4 Pre-WS APIs.
- GSI security
- GridFTP
- GRAM
- XIO
- GASS
- MyProxy
- RLS

- * pyGlobus contributed to GT4 by the Distributed Computing Department of LBNL

- Also includes tools and services
- GridFTP server
- GridFTP GUI client
- Other GT4 clients

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Lightweight Data Replicator*

- Ties together 3 basic Grid services:
- Metadata Service
- info about files such as size, md5, GPS time, ...
- sets of interesting metadata propagate
- answers question “What files or data are available?”
- Globus Replica Location Service (RLS)
- catalog service maps filenames to URLs
- also maps filenames to sites
- answers question “Where are the files?”
- GridFTP Service
- server and customized, tightly integrated client
- use to actually replicate files from site to site
- answers question “How do we move the files?”

- * Slide courtesy of Scott Koranda, UW-Milwaukee

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## LDR Architecture

- Each site has its own machinery for pulling data needed to satisfy local users
- Scheduler daemon queries metadata and replica catalogs to identify missing files, which it records in a prioritized list
- Transfer daemon manages transfers for files on the list, and updates LRC when files arrive
- If a transfer fails, the transfer daemon simply re-adds the file to the list for future transfers
- Daemons and metadata service written in Python

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Results - LIGO*

- LIGO/GEO S4 science run completed March
- Replicated over 30 TB in 30 days
- Mean time between failure now one month (outside of CIT)
- Over 30 million files in LDR network
- 6+ million unique LFNs in RLS network
- 30+ million PFNs for those LFNs
- Performance currently appears to be limited by Python performance
- Partnership with Globus Alliance members is leading to LDR-like tools being added to Globus Toolkit

- * Slide courtesy of Scott Koranda, UW-Milwaukee

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Results - GT and e-Science

- Data Replication Service (DRS)
- Reimplementation of LDR’s subscription capabilities
- Uses WSRF-compliant services written in Java
- Works with RLS and RFT services in GT4
- Tech preview in GT4
- LIGO evaluating as a replacement for LDR’s subscription system
- Remaining piece is metadata service

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## A Few LDR/DRS Lessons

- Understanding performance requirements is important
- Performance is complicated. Requirements must be expressed very specifically and must be thoroughly informed by use
- LIGO’s operational use was specific enough to give the RLS team good requirements
- Reliability doesn’t have to be sophisticated
- LDR’s strategy for recovering from a transfer failure is simply to put the transfer request back on the queue
- Standard interfaces are important
- LIGO was not successful at getting other projects to rally around LDR’s non-standard interfaces
- DRS *might* overcome that hurdle by using WSRF interfaces

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Case Study 2:Scientific Workflow for Computational Economics

- Tiberiu Stef-PraunGabriel MadeiraIan FosterRobert Townsend

## The Challenge

- Expand capability of economists to develop and validate models of social interactions at large scales
- Harness large computation systems
- Simplify programming model (eye toward easy integration of science code)
- Improve automation
- Requires an end-to-end approach, but through integration, not the “silo” model

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Moral Hazard Problem

- An entity in control of some resources (the entrepreneur) contracts with other entities that use these resources to produce outputs (the workers)
- Two organizational forms are available
- The workers cooperate on their efforts and divide up their income (thus sharing risks)
- The workers are independent of each other, and are rewarded based on relative performance
- Both are stylized versions of what is observed in tenancy data in villages such as in Maharastra, India (Townsend and Mueller 1998)

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Moral Hazard Solver

- Five stages, each solved by linear programming
- Balance between promises for future and consumption to optimally reward agents
- In each stage: Given a set of parameters: consumption, effort, technology, output, wealth
- Do a linear optimization to find out the best behavior
- Parameter sweep (grid of parameter values)
- Linear solver is run independently on each point of the parameter grid
- Results are merged at end of the stage
- Across stages: Different organization (parameters) for similar stage structure
- Most stages depend on results of other stages

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## 50 Min

- 30 Min

- 3 Min

- 40 Min

- 2 Min

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Issues - Technical

- Language
- Science code written in MATLAB/Octave
- End to end system must be language-independent
- Code prerequisites
- Each solver task requires MATLAB/Octave pre-installed on the execution node, and solver code staged in prior to execution
- Each solver task requires files from previous stages
- Automation
- ~200 tasks must be executed
- This is a lot of “babysitting” if performed manually

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Issues - Social

- Licensing
- MATLAB licensing has a per-node cost
- Expensive if you’re using O(10)+ nodes
- Provenance
- Task execution, data integrity
- Not a huge concern at this scale, but for larger scales (10,000 tasks) it is important to record how the work is performed
- Provisioning, resource sharing
- This problem used a shared campus cluster (at U Chicago)
- We know of problems with 2-3 orders of magnitude more tasks, which require (inter)national-scale resources to accomplish in a timely fashion

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Swift System

- Clean separation of logical/physical concerns
**XDTM** specification of logical data structures
- + Concise specification of parallel programs
**SwiftScript**, with iteration, etc.
- + Efficient execution on distributed resources
**Karajan** threading, **Falkon** provisioning, **Globus** interfaces, pipelining, load balancing
- + Rigorous provenance tracking and query
- Virtual data schema & automated recording
-  **Improved usability and productivity**
- Demonstrated in numerous applications

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Workflow Language - SwiftScript

- Goal: Natural feel to expressing distributed applications
- Variables (basic, data structures)
- Conditional operators (if, foreach, )
- Functions (atomic / compound)
- Used to connect outputs to inputs
- It does not specify invocation order, only dependencies
- It can be seen as a metadata for expressing experiments

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Execution Engine

- Karajan engine (event-based execution)
- Has a scheduler to map tasks to resources
- Score-based planning
- Recovers from failures (retries)
- Falkon resource manager creates a “virtual private cluster”
- Uses Globus GRAM4 (PBS/Condor/Fork) to acquire resources from Grid systems

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Dynamic Provisioning:Swift Architecture

- SwiftScript

- Abstract
- computation

- Virtual Data
- Catalog

- SwiftScript
- Compiler

**Specification**

**Execution**

- Dynamic Provisioning:Swift Architecture

- Yong Zhao, Mihael Hatigan, Ioan Raicu, Mike Wilde, Ben Clifford

- Service-Oriented Science: Globus Software in Action

- eSS 2007

**Notes:**
Note that current version has not yet integrated the VDC from the VDS VDL implementation.

One advantage of this is simplicity for the user.  When we integrate it back in, its likely that we’ll make it optional to retain this simplicity, especially for users less interested in provenance storage.

Explain:  what is the format of the abstract workflow.

## The Solution

- Code changes
- Solver code was broken into modules (atomic blocks) to allow parallel execution
- Code ported from MATLAB to Octave to avoid per-node licensing fees
- Workflow was described in SwiftScript
- Software installation
- Swift engine, Karajan, Falkon deployed locally
- Shared resource (already available)
- Existing compute cluster with GRAM4, GridFTP, etc.

- Service-Oriented Science: Globus Software in Action

- eSS 2007

**Notes:**
Breaking the science code into modules did not require major restructuring because the solution was already largely parallelized.
Octave is an open source (free) alternative to MATLAB. 

## Moral Hazard SwiftScript Code Excerpts

- // A second atomic procedure: merge (file mergeSolutions[]) econMerge (file merging[]) {         app{                 econMerge @filenames(mergeSolutions)  @filenames(merging);         }}  // We define the stage one procedure–a compound procedure (file solutions[]) stageOne (file inputData[], file prevResults[]) {         file script<"scripts/interim.m">;         int batch_size = 26;         int batch_range = [0:25];         string inputName = "IRRELEVANT";         string outputName = "stageOneSolverOutput";         // The foreach statement specifies that the calls can be performed concurrently                 foreach i in batch_range {              int position = i*batch_size;              solutions[i] = moralhazard_solver(script,batch_size,position,                                                                     	                              inputName, outputName, inputData,  prevResults);         }} // These get used in the “main program” as followsstageOneSolutions = StageOne(stageOneInputFiles,stageOnePrevFiles);stageOneOutputs = econMerge(stageOneSolutions);

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Execution on 40 Processors

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Results - Moral Hazard Solver

- Performance
- Original run time: ~2 hrs
- Swift run time: ~28 min
- Depending on the stage structure, speedup up to 10x, or slowdown (because of overhead)
- Only used one grid site (UC), on multiple sites could get better performance
- Execution has been automated
- Human labor greatly reduced
- Separation of human concerns (science code, system operation, task management)
- Easy to repeat, modify & rerun, etc.

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Other Applications

| Application | #Jobs/computation | Levels |
| --- | --- | --- |
| ATLAS*
HEP Event Simulation | 500K | 1 |
| fMRI DBIC*
AIRSN Image Processing | 100s | 12 |
| FOAM
Ocean/Atmosphere Model | 2000 (core app runs
250 8-CPU jobs) | 3 |
| GADU*
Genomics: (14 million seq. analyzed) | 40K | 4 |
| HNL
fMRI Aphasia Study | 500 | 4 |
| NVO/NASA*
Photorealistic Montage/Morphology | 1000s | 16 |
| QuarkNet/I2U2*
Physics Science Education | 10s | 3-6 |
| RadCAD*
Radiology Classifier Training | 1000s | 5 |
| SIDGrid
EEG Wavelet Proc, Gaze Analysis, ... | 100s | 20 |
| SDSS*
Coadd, Cluster Search | 40K, 500K | 2, 8 |

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Case Study 3:A Distributed Information Service for TeraGrid

- John-Paul NavarroLee Liming

## NSF’s TeraGrid*

**UCSD**

**UT**

**UC/ANL**

**NCSA**

**PSC**

**ORNL**

**PU**

**IU**

**A National Science Foundation Investment in Cyberinfrastructure**
**$100M 3-year construction (2001-2004)**
**$150M 5-year operation & enhancement ****(2005-2009)**

- NSF’s TeraGrid*

- TeraGrid DEEP: Integrating NSF’s most powerful computers (60+ TF)
- 2+ PB Online Data Storage
- National data visualization facilities
- World’s most powerful network (national footprint)
- TeraGrid WIDE Science Gateways: Engaging Scientific Communities
- 90+ Community Data Collections
- Growing set of community partnerships spanning the science community.
- Leveraging NSF ITR, NIH, DOE and other science community projects.
- Engaging peer Grid projects such as Open Science Grid in the U.S. as peer Grids in Europe and Asia-Pacific.
- Base TeraGrid Cyberinfrastructure:Persistent, Reliable, National
- Coordinated distributed computing and information environment
- Coherent User Outreach, Training, and Support
- Common, open infrastructure services

- * Slide courtesy of Ray Bair, Argonne National Laboratory

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Provide a mechanism that allows resource providers, users, and partners the ability to publish and discover information about available capabilities
What are the TG compute resources?
What capabilities does resource X provide?
Where are the login services?
Where can I get data collection Y?
Who has a queue prediction service?
Who has a weather forecasting service?
Provide a mechanism that is suitable for TeraGrid’s open community
Publishers register information (as opposed to turning it over to a central database)
Central index (like Google) enables aggregation, discovery
Multiple access interfaces (WS/SOAP, WS/ReST, browser)

- What are the TG compute resources?
- What capabilities does resource X provide?
- Where are the login services?
- Where can I get data collection Y?
- Who has a queue prediction service?
- Who has a weather forecasting service?
- Provide a mechanism that is suitable for TeraGrid’s open community
- Publishers *register* information (as opposed to turning it over to a central database)
- Central index (like Google) enables aggregation, discovery
- Multiple access interfaces (WS/SOAP, WS/ReST, browser)

- The Challenge

- Service-Oriented Science: Globus Software in Action

- eSS 2007

**Notes:**
Grid concept: distributed set of standard interfaces (like data movement, job submission, data management, login)

## TG Information Services...

| ...IS NOT... | ...IS... |
| --- | --- |
| A central database (Data Warehouse) | A central index/aggregation (Google) |
| A new user interface | A way user interfaces access information |
| A single implementation/tool | Includes several tools |
| A single software interface | Accessed using several useful interfaces |
| A specific set of data | Phased growing set of data  |
| Changed data ownership | Ownership maintained as appropriate |
| Way to manage scientific information | Way to manage Grid meta-data |
| A data management system (database) | An information publishing system |

- ...is a coordinated way to publish, index, and access public [Tera]Grid information using software interfaces.

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Issues - Technical

- Information is stored in many legacy systems
- Databases (several types, restricted access)
- Static & dynamic web browser interfaces
- Schema are many and diverse
- Impractical to design a relational database that supports all of these data types and relations
- Many kinds of clients (browsers, SOAP, ReST)
- High availability is critical
- The service will be depended on both by TG operations (testing, documentation, planning) and by many TG users and partners, so it must be available all the time and very stable
- Goal is 99.5% availability

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Issues - Social

- TG is a community of independent service providers
- Independence is prized
- Ownership of information (and its quality control) is important
- Participation in other grids is typical
- Publishers have low threshold for tech hassles
- Publishing mechanism must be simple
- The solution must add to (not replace) existing interfaces

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## MDS4 Overview

- Components
- Index Service – aggregates information and provides a query interface
- Trigger Service – aggregates information and takes actions when conditions are met
- WebMDS - subsets and transforms XML based on XPath queries, XSLT transforms and style sheets
- Information provider APIs – integration with legacy systems
- APIs and command-line clients for developers
- Implemented as Web services
- Uses WSRF (lifecycle, resource properties, etc.)
- Included in the Globus Toolkit 4.0

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## The MDS4 Hourglass

**Schemas**

**Information Users :**
**Schedulers, Portals, Warning Systems, etc.**

**Queueing Systems**
** ****(PBS, Torque, etc.)**

**WS standard interfaces for subscription, registration, notification**

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## TeraGrid’s IS Architecture 

- Clients

- Clients

- TeraGrid’s IS Architecture

- Cache

- WS/RESTHTTP GET

- WS/SOAP

- WS MDS4

- TomcatWebMDS

- Apache 2.0

- TeraGrid Central Services

- TeraGrid Repositories

- Partners

- WS/SOAP

- WS MDS4

- Resource Provider Services

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Central vs. Distributed Services

- Publisher Content
- Publisher-owned and maintained information
- Data probably originates somewhere in the local system
- Publisher Code
- An MDS4 index service
*Or:* Any Web service that has WS-ResourceProperties

- Central Content
- Aggregated publisher content
- Central Code
- Redundant servers
- Information caching (persistence)
- MDS4 index services (WS/SOAP)
- WebMDS/Tomcat, Apache 2.0, ... (WS/REST)
- Content published in: HTML, XHTML/XML, XML, Atom, RSS, ...

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Registration

- Publisher *registers* available content
- Local service maintains a *registration* with the central indices
- Registration expires automatically, so refresh is needed periodically
- Publishers retain ownership and operation of their own information service (can be registered with other grids!)
- Index services pull content
- Registrations are subject to access control
- Uses registration data to contact service and get latest content
- Caches content locally, subject to purge policy
- Cache allows for service outages, etc.

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## High-Availability Design 

- High-Availability Design

- ...

- info.dyn.teragrid.org

- info.teragrid.org

- TeraGrid Dynamic DNS

**Dynamically direct clients to one or more servers**
**Set by Information Services administrators**
**Changes propagate globally fast (TTL = 15 minutes)**

**Clients**

**Dynamically Changes**

**Doesn’t Change**

- RP/partner services

- TG wide servers

**(Patrick Dorn & NCSA NetEng)**

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Information Services Users 

- Information Services Users

**User Documentation**

**User Portal**

- Database?

- Gateways

- Peer Grids

- User Applications

**info.teragrid.org**

**Others**

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Queue Contents in User Portal

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Resource/Capability Info

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Where are GridFTP services?

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Results - TeraGrid

- Considerable excitement from information owners...
- A way to raise awareness for their information & capabilities
- Doesn’t require them to replace legacy systems or turn information over to someone else
- ...and information consumers
- Simple, consistent access mechanisms for lots of information types
- A mediating agency for independent service operators
- Integration to date:
- Compute service descriptions and queue status
- Software & service availability registry
- Central documentation
- Verification & validation testing service

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## [Not so] Farfetched Possibilities

- Science gateway service availability
- Data collection availability
- Community software area catalog
- Community accounts availability
- Service outages publication
- Resource policies
- Peer grid resource/capability availability

- Service-Oriented Science: Globus Software in Action

- eSS 2007

**Notes:**
Software Integration is staffed to support expanding information services content

## Lessons from the Grid

## Lesson 0 - We speak and think differently

- CS and IT people have a professional language and set of design patterns that are foreign to most scientists (and vice versa)
- This communication gap is a challenge that must be addressed head-on in collaborations
- We have found over and over that when we successfully communicate CS patterns and see how they apply to scientific work, the results are powerful, even transformational
- Science benefited greatly from logic, algebra, calculus, etc.
- Science + CS = {computational science, Science 2.0, e-Science}

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Lesson 1 - It’s All About People

- Collaborations are complicated
- To support collaboration, IT needs to be flexible and adaptive to human needs
- Security
- Service provisioning and management
- Resource allocation
- Heterogeneity
- IT should not make assumptions to simplify things and then expect users to behave according to the assumptions

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Lesson 1 (cont.)

- Most IT is currently not very flexible or adaptable
- Networks are statically configured
- Compute systems are all different, even after you get your job launched
- Storage management has a long way to go (reservation, reconfiguration)
- Grid technology is aimed at this problem
- Extends existing Web and Web services capabilities
- Add a layer of virtualization to deal with heterogeneity
- Add a layer of management to deal with provisioning and adaptation

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Lesson 2 - The Application/Infrastructure Gap

- As scientists, we mustn’t build silos
- Don’t custom-build apps to specific systems
- Don’t assume a dedicated infrastructure
- Use tools like Globus, Swift, etc. to free your apps from resource providers
- Resource providers are not monopolies
- RPs of the future will be service hosts, not application magnets
- Qualities of service, specialization, and hosting models will be competitive characteristics

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Lesson 3 -This Isn’t New (But It Really Is)

- We’ve known how to solve these types of problems in general for a while now
- Abstraction
- Virtualization
- Plug-in interfaces
- Modularity
- Parallelism
- Striping
- Virtual machines
- Encapsulation
- Service interfaces
- Notification
- Soft-state registration

- The trick is to apply them in the right places...
- Security
- Execution
- Storage
- Network
- ...to get the desired behaviors...
- Provisioning
- Authorization
- Management
- ...at the right levels
- Department
- Institution
- Agency
- Virtual organization

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Lesson 4 - Metadata is a Different Problem

- Communities that share data need metadata standards
- This is often overlooked in project planning, but it is essential and non-trivial
- Metadata is as much a social challenge as a technical one
- Requires different expertise, including a heavy dose of expertise and experience in the problem domain(s)
- Don’t expect your IT team to solve this one for you
- Data curation is a job description, not a software feature
- Software tools can _help_ do the job, *if *they are designed with the human processes in mind

- Service-Oriented Science: Globus Software in Action

- eSS 2007

## Conclusions

- Science 2.0 (Service Oriented Science) is a work in progress
- Early adopters are banging away at it
- Results from early applications are compelling, even transformational
- It’s only as we build, deploy, and use these systems that we identify the key capabilities
- Abstracting these capabilities and implementing them in modular form is hard (very technical) work
- A growing suite of tools is available for use
- http://www.globus.org/grid_software/
- We have examples of successes
- http://www.globus.org/solutions/

- Service-Oriented Science: Globus Software in Action

- eSS 2007