# Presentation: 165405

## A New Era in Computing Moving Enterprise Services onto Grid

- Ian Foster
- Argonne National Laboratory
- University of Chicago
- Univa Corporation

## Abstract

-    The Grid seems to be everywhere, with announcements appearing almost every day of Grid products, sales, and deployments from major vendors. However, in spite of the popularity of the term, there is often confusion as to what the Grid is and what problems it solves. Is there any "there there" or is it all just marketing hype?
-    In this talk, I will address these questions, describing what the Grid is, what problems it solves, and what technology has been developed to build Grid infrastructure and create Grid applications. I will review the current status of Grid infrastructure and deployment and give examples of where Grid technology is being used not only to perform current tasks better, but to provide fundamentally new capabilities that are not possible otherwise.

## The Future of Enterprise IT?

*Adaptable Computing*

*Utility Computing*

*Grid*

*Agile IT*

*On Demand Computing*

*Autonomic Computing*

*Cluster Computing*

*.Net*

*Software as Service*

*Service-oriented Architecture*

*Virtualization*

## The Enterprise IT Environment

- (1) The Good Old Days

-   *>95% utilization with mixed application workload*
*  **High reliability*
*  **Uniform environment*

## The Enterprise IT Environment(2) Attack of the Killer Micros

*And the Internet and high-speed networks ...*

## Overprovisioning: x10 or more

High management costs

Inflexible environment

- The Enterprise IT Environment

- (3) The Emergence of Silos

*Overprovisioning: x10 or more*
*High management costs*
*Inflexible environment*

## Define & compose services

- (4) IT as Service

*Define & compose services*
*** **Service-oriented applications*
*Allocate resources to services*
*** **Service-oriented infrastructure*

**Notes:**
Define services – without regard to location or implementation

Compose services to build applications

## Service-oriented Gridinfrastructure
Provision physicalresources to support application workloads

- Service-oriented **Grid****infrastructure**
- Provision physicalresources to support application workloads

- ApplnService

- ApplnService

- Users

- Workflows

*Composition*

*Invocation*

- Service-oriented **applications**
- Wrap applications as services
- Compose applicationsinto workflows

- The Enterprise IT Environment

- (4) IT as Service

**Notes:**
“Provisioning” – reservation to configuration to ... ... make sure resource will do what I want it to do, with the right qualities of service
Virtualization = separation of concerns between provider & consumer of “content”
Client and service
Service provider and resource provider 
Provisioning = assemble & configure resources to meet user needs
Management = sustain desired qualities of service despite dynamic environment

## SAP Demonstration @ TechEdVariable Demand on Applications

- SAP Demonstration @ TechEdVariable Demand on Applications

**   ****Scenario I**: Too few IPC servers**Result:** Bad user experience

- Three applications
**IPC**: Internet Pricing Configurator
**CRM**: Workforce Management
**SCM**: Advanced Planner & Optimizer

**   **
**   ****Scenario II**: Too many IPC servers**Result:** Unnecessarily high TCO

## SAP Demonstration @ TechEd Grid-enabling Apps using Globus

- SAP Demonstration @ TechEd Grid-enabling Apps using Globus

**   ****Result**: Always the right number of IPC server applications

- Provision New
- Worker Process

**   ****Grid-enabled scenario**:IPC server applications started and decommissioned by a re-factored IPC dispatcher & Globus

## Realizing IT as Service:The Crucial Role of Grid Infrastructure

**IT Facility**

- Realizing IT as Service:The Crucial Role of **Grid Infrastructure**

**IT Facility**

**IT Resources**
**Computing**
**Storage**
**Communications**

**Application**

## The Future of Enterprise IT?

*Adaptable Computing*

*Utility Computing*

*Grid*

*Agile IT*

*On Demand Computing*

*Autonomic Computing*

*Cluster Computing*

*.Net*

*Software as Service*

*Service-oriented Architecture*

*Virtualization*

- IT as Service

- Grid Infrastructure

## Grid Infrastructure:Problems & Opportunities

- Overprovisioning
-  Reduce IT capital expenses 5-10x
- Diverse, independently managed silos
-  Reduce IT operational expenses
- Application platform limitations
-  Optimize performance with QoS
- Inflexible IT infrastructure
-  Improved responsiveness to business needs
- Vendor lock-in
-  Open source, standards, commodity IT

**Notes:**
E.g., financial services firm with average 10% utilization after development, test, production, failover, continuity

No more individually managed application silos!

e.g. multiple legacy applications requiring custom configurations (OS + platform), hand-crafted administration

e.g. replace costly SMP servers with Linux clusters

## Value Proposition Analysis

| Industry | Characteristic | Core Value Drivers, Needs |
| --- | --- | --- |
| Financial Services | High percentage of home-grown applications | Value: Time-to-value for new applications
Need: Rapid rollout of new applications to  support new business capabilities |
| Manufacturing, Automotive, R&D | Core business driven by product design and engineering | Value: Effective collaboration among globally distributed design teams
Need: Efficient access to large data sets and HPC capabilities |
| Energy, Pharmaceutical | Exploration/research is the highest cost component | Value: Time-to-market for new products
Need: Seamless integration of custom devices delivering large volumes of data into analytical applications |

**Notes:**
Value proposition is in the business capabilities enabled by Grid, not in the Grid technology itself
Value proposition varies by industry vertical

## Grid Infrastructure Specifics

- Provide
- Resource modeling
- Monitoring and notification
- Reservation
- Provisioning, lifecycle, decommissioning
- Accounting & auditing
- For
- Computers, storage, networks, software, ...

**Notes:**
describe available resources, their capabilities and the relationships between them in order to facilitate discovery, provisioning, and quality of service management.
provide visibility into the state of resources—and notify applications and infrastructure management services of changes in state—to enable discovery and maintain quality of service. Logging of significant events and state transitions is also needed in order to support accounting and auditing functions.
assure quality of service across an entire set of resources for the lifetime of their use by an application. This is enabled by negotiating the required level(s) of service and ensuring the availability of appropriate resources through some form of reservation: essentially the dynamic creation of a service level agreement.
enable an allocated resource to be configured automatically for application use, manage the resource for the duration of the task at hand, and restore the resource to its original state for future use.
track the usage of shared resources and provide mechanisms for transferring cost amongst user communities and for charging for resource use by applications and users.

## Open Standards & Open Source

**Internet**: TCP/IP                  ... BSD Unix

**Web**: HTTP                         ... Apache

**Grid**:  OGSA                 ... Globus

## Grid Infrastructure:Open Standards

- Web services(WSDL, SOAP, WS-Security, WS-ReliableMessaging, ...)

- WS-Resource Framework & WS-Notification(Resource identity, lifetime, inspection, subscription, ...)

- WS-Agreement(Agreement negotiation)

- WS Distributed Management(Lifecycle, monitoring, ...)

- Applications of the framework(Compute, network, storage provisioning,job reservation & submission, data management,application service QoS, ...)

**Notes:**
Service-Oriented Frameworks

General functionality can be used, extended, specialized for specific purposes
Both specifications and implementations
Captures commonality that can be applied across resources/services
Avoid least common denominator solutions
Allows service consumers to operate at different levels of abstraction
General operations against any service that implements the frameworks
Specific operations against services that extend/specialize the frameworks

## Web Servicesand Stateful Resources

- “State” appears in almost all applications
- Data in a purchase order
- Current usage agreement for resources
- Metrics associated with work load on a server
- Web services can model, access and manage state in many different ways
- Ad-hoc, per-application approaches
- WS-Resource Framework (WSRF) & WS-Notification propose a standard approach

- Modeling Stateful Resources with Web Services, Foster, Frey, Graham, et al. www.globus.org/wsrf, 2004

## WSRF & WS-Notification

**Naming and bindings** (basis for virtualization)
- Every resource can be _uniquely referenced_, and has one or more _associated services_ for interacting with it
**Lifecycle** (basis for fault resilient state mgmt)
- Resources created by services following _factory_ pattern
- Resources destroyed _immediately_ or _scheduled_
**Information model** (basis for monitoring, discovery)
- _Resource properties_ associated with resources
- Operations for _querying_ and _setting_ this info
- Asynchronous _notification_ of changes to properties
- Service groups (basis for registries, collective svcs)
- Group membership rules & membership management
- Base Fault type

## Bringing it All Together

*Scenario: Resource management & scheduling*

## Grid Infrastructure: Open Source

- Data Mgmt

- Security

- CommonRuntime

- Execution Mgmt

- Info Services

- Web ServicesComponents

- Non-WS Components

- Pre-WS
- Authentication
- Authorization

- GridFTP

- Pre-WS
- Grid Resource
- Alloc. & Mgmt

- Pre-WSMonitoring
- & Discovery

- C Common
- Libraries

- Authentication
- Authorization

- Reliable
- File
- Transfer

- Data Access
- & Integration

- Grid Resource
- Allocation &
-  Management

- Index

- Java WS Core

- CommunityAuthorization

- Replica
- Location

- eXtensible
- IO (XIO)

- Credential
- Mgmt

- Community
- Scheduling
- Framework

- Delegation

- Data
- Replication

- Trigger

- C WS Core

- Python WS Core

- WebMDS

- Workspace
- Management

- Grid Telecontrol Protocol

- Contrib/Preview

- Core

- Depre-cated

**www.globus.org**

- Globus Toolkit           version 4 (GT4)

## GT4 & Web Services

- Custom
- Services

- WS-A, WSRF, WS-Notification

- Custom
- WSRF
- Services

- GT4WSRF Web
- Services

- Registry
- & Admin

- GT4 Container(e.g., Apache Axis)

## Open Source With Support:Univa Globus

- IT Facility

- Open Source With Support:Univa Globus

- IT Facility

- IT Resources
- Computing
- Storage
- Communications

- Application

- Univa nPlus

## Managing Computers & Computation

- GRAM (Grid Resource Allocation & Management) service
- Negotiate access
- Stage code
- Monitor service
- Manage service
- Collect accounting data
- Can negotiate access to clusters,creation of virtual machines,establishment of virtual networks, ...

- GRAM

- Client

## Dynamic Provisioning ofComputational Services  

- Dynamic Provisioning ofComputational Services

- Provision New
- Worker Process

**   ****Grid-enabled scenario**:IPC server applications started and decommissioned by a re-factored IPC dispatcher & Globus

## Open Science Grid use over 6 months

- Usage: CPUs

- Dynamic Provisioning ofComputational Services

*Open Science Grid use over 6 months*

## Managing Storage & Data

- Service interfaces for managing storage & data movement
- Storage management (SRM, NeST)
- Data movement (GridFTP, RFT)
- Replica management (RLS, DRS)
- Service interfaces for accessing data in diverse formats
- OGSA Data Access & Integration
- GridFTP data access & movement

## Reliable Wide Area Data Replication

**Birmingham****•**

- Replicating >1 Terabyte/day to 8 sites
- >30 million replicas so far
- MTBF = 1 month

- LIGO Gravitational Wave Observatory

## Managing Networks

- The **network** is a vital part of a virtualized infrastructure
- Must be able to negotiate **E2E paths**, e.g.
- Firewall traversal
- Security properties
- Bandwidth and related QoS parameters
- Optical paths
- Also: establishment of **overlay networks** with negotiated properties
- Flat IP name space (IPv6) & IPsec make all of this easier!

## Managing Networks (Nortel with Globus)

- Managing Networks (Nortel with Globus)

- A Grid Domain:
- Virtual Organization

***MDS Index***

**A**

***Grid Overlay***

**Access**

***Network Service Overlay***

**Access**

**Core Network**

**Application requests broker/metascheduler for job services and resources**
**Broker/metascheduler generates RSL2 for resource allocation requests after consulting MDS/Index**
**xSPs co-ordinate to allocate requested resources**

**1**

**2**

**2**

**RSL2**

**RSL2**

**3**

*Broker/Metascheduler*

## Managing Security:Dynamic Policy Overlays

- Grid security must define mechanisms for bridging identity, policy and trust across autonomous domains with minimal a priori arrangements

## Authentication & Authorization

- Users

## Managing Instrumentation:Network for Earthquake Eng. Simulation

-  Links instruments, data,
- computers, people* *

- IT as Service

## The Integrating Role ofGrid Infrastructure

## Grid Infrastructure and Enterprise IT

- Coordinate management of previously distinct application silos
- Reduce over-provisioning
- Improve application performance
- Improve IT responsiveness to business needs
- Enable new applications
- Avoid vendor lock-in
- Provide a basis for utility computing, software as service, etc.

## For More Information

- Globus Alliance
- www.globus.org
- Global Grid Forum
- www.ggf.org
- Globus Consortium
- www.globusconsortium.org
- Univa Corporation
- www.univa.com
- Background information
- www.mcs.anl.gov/~foster

- 2nd Edition
- www.mkp.com/grid2