# Presentation: 541985

## Grid Computing and the Globus Toolkit

- Bill Allcock
- Jennifer M. Schopf
- Argonne National Lab
- (National eScience Centre)
- http://www.mcs.anl.gov/~jms/Talks/

## What is a Grid

- Resource sharing
- Computers, storage, sensors, networks, ...
- Sharing always conditional: issues of trust, policy, negotiation, payment, ...
- Coordinated problem solving
- Beyond client-server: distributed data analysis, computation, collaboration, ...
- Dynamic, multi-institutional virtual orgs
- Community overlays on classic org structures
- Large or small, static or dynamic
- Made more difficult by the lack of central control, shared resources, and increased need for communication

## Globus is an Hour Glass

- Local sites have an their own policies, installs – heterogeneity!
- Queuing systems, monitors, network protocols, etc
- Globus unifies
- Build on Web services
- Use WS-RF, WS-Notification to represent/access state
- Common management abstractions & interfaces

- Local heterogeneity

- Higher-Level Services
- and Users

- Standard GT4
- Interfaces

## Globus is a Building Block

- Basic components for grid functionality
- Highest-level services are often application specific, we let applications concentrate there
- Easier to reuse than to reinvent
- Compatibility with other Grid systems comes for free
- We provide basic infrastructure to get you one step closer

## GT Domain Areas

- Core runtime
- Infrastructure for building new services
- Security
- Apply uniform policy across distinct systems
- Execution management
- Provision, deploy, & manage services
- Data management
- Discover, transfer, & access large data
- Monitoring
- Discover & monitor dynamic services

## Globus Toolkit: Open Source Grid Infrastructure

**Data Mgmt**

**Security**

**Common****Runtime**

**Execution Mgmt**

**Info Services**

- GridFTP

- Authentication
- Authorization

- Reliable
- File
- Transfer

- Data Access
- & Integration

- Grid Resource
- Allocation &
-  Management

- Index

- CommunityAuthorization

- Data
- Replication

- Community
- Scheduling
- Framework

- Delegation

- Replica
- Location

- Trigger

- Java Runtime

- C Runtime

- Python Runtime

- WebMDS

- Workspace
- Management

- Grid Telecontrol Protocol

**Globus Toolkit v4**
**www.globus.org**

- CredentialMgmt

**Globus Toolkit:**** Open Source Grid Infrastructure**

## dev.globus.org

- Created to facilitate contributions and open development
- Individual development efforts organized as projects
- Consensus-based decision making
- Control over each project in the hands of its most active and respected contributors (committers)
- Similar to Apache Jakarta process

## General Globus Help and Support

- Each component also has mailing lists
- *-user@globus.org for user issues
- *-dev@globus.org for developer issues
- Bugzilla
- bugzilla.globus.org

## Globus Toolkit: Open Source Grid Infrastructure

**Data Mgmt**

**Security**

**Common****Runtime**

**Execution Mgmt**

**Info Services**

- GridFTP

- Authentication
- Authorization

- Reliable
- File
- Transfer

- Data Access
- & Integration

- Grid Resource
- Allocation &
-  Management

- Index

- CommunityAuthorization

- Data
- Replication

- Community
- Scheduling
- Framework

- Delegation

- Replica
- Location

- Trigger

- Java Runtime

- C Runtime

- Python Runtime

- WebMDS

- Workspace
- Management

- Grid Telecontrol Protocol

**Globus Toolkit v4**
**www.globus.org**

- CredentialMgmt

**Globus Toolkit:**** Open Source Grid Infrastructure**

## GT4 Web Services Runtime

- Supports both GT (GRAM, RFT, Delegation, etc.) & user-developed services
- Redesign to enhance scalability, modularity, performance, usability
- Leverages existing WS standards
- WS-I Basic Profile: WSDL, SOAP, etc.
- WS-Security, WS-Addressing
- Adds support for emerging WS standards
- WS-Resource Framework, WS-Notification
- Java, Python, & C hosting environments
- Java is standard Apache

## What does Core give you?

- Reference implementation of WSRF and WS-N functions
- Naming and bindings (basis for virtualization)
- Every resource can be _uniquely referenced_ and has one or more _associated services_ for interacting
- Lifecycle (basis for resilient state management)
- Resources created by svcs following a _factory_ pattern
- Resource destroyed _immediately_ or _scheduled_
- Information model (basis for monitoring & discovery)
- _Resource properties_ associated with resources
- Operations for _querying_ and _setting_ this info
- Asynchronous _notification_ of changes to properties
- Service groups (basis for registries & collective svcs)
- Group membership rules and membership management
- Base fault type

## Globus Toolkit: Open Source Grid Infrastructure

**Data Mgmt**

**Security**

**Common****Runtime**

**Execution Mgmt**

**Info Services**

- GridFTP

- Authentication
- Authorization

- Reliable
- File
- Transfer

- Data Access
- & Integration

- Grid Resource
- Allocation &
-  Management

- Index

- CommunityAuthorization

- Data
- Replication

- Community
- Scheduling
- Framework

- Delegation

- Replica
- Location

- Trigger

- Java Runtime

- C Runtime

- Python Runtime

- WebMDS

- Workspace
- Management

- Grid Telecontrol Protocol

**Globus Toolkit v4**
**www.globus.org**

- CredentialMgmt

**Globus Toolkit:**** Open Source Grid Infrastructure**

## Globus Security

- Control access to shared services
- Address autonomous management, e.g., different policy in different work-groups
- Support multi-user collaborations
- Federate through mutually trusted services
- Local policy authorities rule
- Allow users and application communities to set up dynamic trust domains
- Personal/VO collection of resources working together based on trust of user/VO

## Virtual Organization (VO) Concept

**Virtual Organization (VO) Concept**

- VO for each application or workload
- Carve out and configure resources for a particular use and set of users

## GT4 Security

- Public-key-based authentication
- Transport- and message-level authentication
- Extensible authorization framework based on Web services standards
- SAML-based authorization callout
- Integrated policy decision engine
- XACML policy language, per-operation policies, pluggable

## Security Tools

- Basic Grid Security Mechanisms
- Certificate Generation Tools
- Certificate Management Tools
- Getting users “registered” to use a Grid
- Getting Grid credentials to wherever they’re needed in the system
- Authorization/Access Control Tools
- Storing and providing access to system-wide authorization information
- Credential management service
- MyProxy (One time password support)

## Other Security Services Include ...

- MyProxy
- Simplified credential management
- Web portal integration
- Single-sign-on support
- KCA & kx.509
- Bridging into/out-of Kerberos domains
- SimpleCA
- Online credential generation
- CAS
- Community Authorization Service

## A Cautionary Note

- Grid security mechanisms are tedious to set up
- If exposed to users, hand-holding is usually required
- These mechanisms can be *hidden entirely* from end users, but still used behind the scenes
- These mechanisms exist for good reasons.
- Many useful things can be done without Grid  security
- It is unlikely that an ambitious project could go into production operation without security like this
- Most successful projects end up using Grid security, but using it in ways that end users don’t see much

## GT Authorization Framework

## GT-XACML Integration

- eXtensible Access Control Markup Language
- OASIS standard, open source implementations
- XACML: sophisticated policy language
- Globus Toolkit ships with XACML runtime
- Included in every client and server built on GT
- Turned-on through configuration
- ... that can be called transparently from runtime and/or explicitly from application ...
- ... and we use the XACML-”model” for our Authz Processing Framework

## Globus Toolkit: Open Source Grid Infrastructure

**Data Mgmt**

**Security**

**Common****Runtime**

**Execution Mgmt**

**Info Services**

- GridFTP

- Authentication
- Authorization

- Reliable
- File
- Transfer

- Data Access
- & Integration

- Grid Resource
- Allocation &
-  Management

- Index

- CommunityAuthorization

- Data
- Replication

- Community
- Scheduling
- Framework

- Delegation

- Replica
- Location

- Trigger

- Java Runtime

- C Runtime

- Python Runtime

- WebMDS

- Workspace
- Management

- Grid Telecontrol Protocol

**Globus Toolkit v4**
**www.globus.org**

- CredentialMgmt

**Globus Toolkit:**** Open Source Grid Infrastructure**

## Execution Management (GRAM)

- Common WS interface to schedulers
- Unix, Condor, LSF, PBS, SGE, ...
- More generally: interface for process execution management
- Lay down execution environment
- Stage data
- Monitor & manage lifecycle
- Kill it, clean up
- A basis for application-driven provisioning

## GRAM - Basic Job Submission and Control Service

- A uniform service interface for remote job submission and control
- Includes file staging and I/O management
- Includes reliability features
- Supports basic Grid security mechanisms
- Available in Pre-WS and WS
- GRAM is *not* a scheduler.
- No scheduling
- No metascheduling/brokering
- Often used as a front-end to schedulers, and often used to simplify metaschedulers/brokers

## Globus Toolkit: Open Source Grid Infrastructure

**Data Mgmt**

**Security**

**Common****Runtime**

**Execution Mgmt**

**Info Services**

- GridFTP

- Authentication
- Authorization

- Reliable
- File
- Transfer

- Data Access
- & Integration

- Grid Resource
- Allocation &
-  Management

- Index

- CommunityAuthorization

- Data
- Replication

- Community
- Scheduling
- Framework

- Delegation

- Replica
- Location

- Trigger

- Java Runtime

- C Runtime

- Python Runtime

- WebMDS

- Workspace
- Management

- Grid Telecontrol Protocol

**Globus Toolkit v4**
**www.globus.org**

- CredentialMgmt

**Globus Toolkit:**** Open Source Grid Infrastructure**

## Monitoring and Discovery System(MDS4)

**Monitoring and Discovery System****(MDS4)**

- Grid-level monitoring system
- Aid user/agent to identify host(s) on which to run an application
- Warn on errors
- Uses standard interfaces to provide publishing of data, discovery, and data access, including subscription/notification
- WS-ResourceProperties, WS-BaseNotification, WS-ServiceGroup
- Functions as an hourglass to provide a common interface to lower-level monitoring tools

## Standard Schemas

- (GLUE schema, eg)

- Information Users :
- Schedulers, Portals, Warning Systems, etc.

- WS standard interfaces for subscription, registration, notification

## MDS4 Components

- Information providers
- Monitoring is a part of every WSRF service
- Non-WS services are also be used
- Higher level services
- Index Service – a way to aggregate data
- Trigger Service – a way to be notified of changes
- Both built on common aggregator framework
- Clients
- WebMDS
- All of the tool are schema-agnostic, but interoperability needs a well-understood common language

## Working with TeraGrid

- Large US project across 9 different sites
- Different hardware, queuing systems and lower level monitoring packages
- Starting to explore MetaScheduling approaches
- GRMS (Poznan)
- W. Smith (TACC)
- K. Yashimoto (SDSC)
- User Portal
- Need a common source of data with a standard interface for basic scheduling info

## Data Collected

- Provide data at the subcluster level
- Sys admin defines a subcluster, we query one node of it to dynamically retrieve relevant data
- Can also list per-host details
- Interfaces to Ganglia, Hawkeye, CluMon, and Nagios available now
- Other cluster monitoring systems can write into a .html file that we then scrape
- Also collect basic queuing data, some TeraGrid specific attributes

## Globus Toolkit: Open Source Grid Infrastructure

**Data Mgmt**

**Security**

**Common****Runtime**

**Execution Mgmt**

**Info Services**

- GridFTP

- Authentication
- Authorization

- Reliable
- File
- Transfer

- Data Access
- & Integration

- Grid Resource
- Allocation &
-  Management

- Index

- CommunityAuthorization

- Data
- Replication

- Community
- Scheduling
- Framework

- Delegation

- Replica
- Location

- Trigger

- Java Runtime

- C Runtime

- Python Runtime

- WebMDS

- Workspace
- Management

- Grid Telecontrol Protocol

**Globus Toolkit v4**
**www.globus.org**

- CredentialMgmt

**Globus Toolkit:**** Open Source Grid Infrastructure**

## GT4 Data Management

**Stage/move** large data to/from nodes
- GridFTP, Reliable File Transfer (RFT)
- Alone, and integrated with GRAM
**Locate** data of interest
- Replica Location Service (RLS)
**Replicate** data for performance/reliability
- Distributed Replication Service (DRS)
- Provide **access** to diverse data sources
- File systems, parallel file systems, hierarchical storage: GridFTP
- Databases: OGSA DAI

## GridFTP

- A high-performance, secure, reliable data transfer protocol optimized  for high-bandwidth wide-area networks
- FTP with well-defined extensions
- Uses basic Grid security (control and data channels)
- Multiple data channels for parallel transfers
- Partial file transfers
- Third-party (direct server-to-server) transfers
- Reusable data channels
- Command pipelining
- GGF recommendation GFD.20

## Striped GridFTP Service

- A distributed GridFTP service that runs on a storage cluster
- Every node of the cluster is used to transfer data into/out of the cluster
- Head node coordinates transfers
- Multiple NICs/internal busses lead to very high performance
- Maximizes use of Gbit+ WANs

## GridFTP in GT4

- 100% Globus code
- No licensing issues
- Stable, extensible
- IPv6 Support
- XIO for different transports
- Striping  multi-Gb/sec wide area transport
- Pluggable
- Front-end: e.g., future WS control channel
- Back-end: e.g., HPSS, cluster file systems
- Transfer: e.g., UDP, NetBLT transport

- Disk-to-disk onTeraGrid

**Notes:**
Future: integration of provisioning
E.g., storage reservation

## RFT - File Transfer Queuing

- A WSRF service for queuing file transfer requests
- Server-to-server transfers
- Checkpointing for restarts
- Database back-end for failovers
- Allows clients to requests transfers and then “disappear”
- No need to manage the transfer
- Status monitoring available if desired

## Reliable File Transfer:Third Party Transfer

**Reliable File Transfer:****Third Party Transfer**

- RFT Service

- RFT Client

- SOAP Messages

- Notifications(Optional)

- GridFTP Server

- GridFTP Server

- Fire-and-forget transfer
- Web services interface
- Many files & directories
- Integrated failure recovery
- Has transferred 900K files

## Replica Location Service

- Identify location of files via logical to physical name map
- Distributed indexing of names, fault tolerant update protocols
- GT4 version scalable & stable
- Managing ~40 million files across ~10 sites

- Index

- Index

| Local DB | Update send (secs) | Bloom filter  (secs)  | Bloom filter (bits) |
| --- | --- | --- | --- |
| 10K | <1 | 2 | 1 M |
| 1 M | 2 | 24 | 10 M |
| 5 M | 7 | 175 | 50 M |

## What is GridFTP?

- A secure, robust, fast, efficient, standards based, widely accepted data transfer protocol
- A Protocol
- Multiple independent implementations can interoperate
- This works.  Both the Condor Project at Uwis and Fermi Lab have home grown servers that work with ours.
- Lots of people have developed clients independent of the Globus Project.
- We also supply a reference implementation:
- Server
- Client tools (globus-url-copy)
- Development Libraries

## Striped GridFTP Service

- A distributed GridFTP service that runs on a storage cluster
- Every node of the cluster is used to transfer data into/out of the cluster
- Head node coordinates transfers
- Multiple NICs/internal busses lead to very high performance
- Maximizes use of Gbit+ WANs

## GridFTP: The Protocol

- FTP protocol is defined by several IETF RFCs
- Start with most commonly used subset
- Standard FTP: get/put etc., 3rd-party transfer
- Implement standard but often unused features
- GSS binding, extended directory listing, simple restart
- Extend in various ways, while preserving interoperability with existing servers
- Striped/parallel data channels, partial file, automatic & manual TCP buffer setting, progress monitoring, extended restart

## GridFTP: The Protocol (cont)

- Existing standards
- RFC 959: File Transfer Protocol
- RFC 2228: FTP Security Extensions
- RFC 2389: Feature Negotiation for the File Transfer Protocol
- Draft: FTP Extensions
- GridFTP: Protocol Extensions to FTP for the Grid
- Grid Forum Recommendation
- GFD.20
- http://www.ggf.org/documents/GWD-R/GFD-R.020.pdf

## wuftpd based GridFTP

- _Functionality prior to GT3.2_
- Security
- Reliability / Restart
- Parallel Streams
- Third Party Transfers
- Manual TCP Buffer Size
- Partial File Transfer
- Large File Support
- Data Channel Caching
- Integrated Instrumentation
- De facto standard on the Grid

- _New Functionality in 3.2_
- Server Improvements
- Structured File Info
- MLST, MLSD
- checksum support
- chmod support (client)
- globus-url-copy changes
- File globbing support
- Recursive dir moves
- RFC 1738 support
- Control of restart
- Control of DC security

## New GT4 GridFTP Implementation

- NOT based on wuftpd
- 100% Globus code.  No licensing issues.
- Striping support has been added
- Has IPV6 support included (EPRT, EPSV), but we have limited environment for testing.
- Extremely modular to allow integration with a variety of data sources (files, mass stores, etc.)
- Based on XIO
- wuftpd specific functionality, such as virtual domains, will NOT be present

## Striped Server

- Multiple nodes work together and act as a single GridFTP server
- An underlying parallel file system allows all nodes to see the same file system and must deliver good performance (usually the limiting factor in transfer speed)
- I.e., NFS does not cut it
- Each node then moves (reads or writes) only the pieces of the file that it is responsible for.
- This allows multiple levels of parallelism, CPU, bus, NIC, disk, etc.
- Critical if you want to achieve better than 1 Gbs without breaking the bank

## Slide 47

## Globus XIO

- Overview
- Motivation
- Basic Architecture
- Globus XIO User API
- Data Types
- Example usage
- Driver Development
- Framework support
- Interface Functions

- eXtensible Input Output library

**Notes:**
This lays out what we will talk about
Portable between different flavours of unfix, windows, threads, system configurations, etch...

## Grid IO

- Geographically Disperse Resources
- Super Computers / Clusters
- Large Data Stores
- Specialized Scientific Devices
- APS, Telescopes, Environmental Sensors
- Collaborative Sessions
- AccessGrid
- Varying Network Protocols
- GridFTP, HTTP, etc.

## Typical Approach

- Network Protocol

- Network Protocol

**Typical Approach**

- Application

- Network Protocol

- Protocol API

- POSIX IO

- Proprietary API

## Example Application

- Application

- Collaborative Display

- Video Input Stream

- Proprietary API

- Proprietary API

- Remote Data Store

- HPSS

- Grid FTP Client Library

- HPSS Client Library

## Development Time
Application must learn to use many different APIs.
Proprietary APIs can be complex
Each API has its own semantics (and bugs).
Asynchronous/Synchronous
Threaded/non-thread
Different programming models in the same application.
Scalability and Compatibility
New devices or protocols
Application must be modified to work with new devices
Application must keep up with orthogonal research issues.

- Application must learn to use many different APIs.
- Proprietary APIs can be complex
- Each API has its own semantics (and bugs).
- Asynchronous/Synchronous
- Threaded/non-thread
- Different programming models in the same application.
- Scalability and Compatibility
- New devices or protocols
- Application must be modified to work with new devices
- Application must keep up with orthogonal research issues.

**Problems**

## Data Stream IO Abstraction
Many Grid IO needs can be treated as a streams of bytes.
open/close/read/write functionality satisfies most requirements.
Protocol details
Rarely does the application need (or want) to deal with protocol details.
Most needs can be satisfied at initialization time.

- Many Grid IO needs can be treated as a streams of bytes.
- open/close/read/write functionality satisfies most requirements.
- Protocol details
- Rarely does the application need (or want) to deal with protocol details.
- Most needs can be satisfied at initialization time.

**Observations**

## Solution

- Globus XIO user API
- Single API/Single Set of semantics.
- Simple open/close/read/write API
- Driver Abstraction
- Hides protocol details
- Allows for extensibility
- Drivers can be selected at runtime

## Globus XIO Approach

- Network Protocol

- Network Protocol

**Globus XIO Approach**

- Application

- Network Protocol

- Globus XIO

- Driver

- Driver

- Driver

## Example Application With Globus XIO

- Application

- Collaborative Display

- Video Input Stream

- Remote Data Store

- HPSS

- Globus XIO
- (GridFTP driver)

- Globus XIO
- (HPSS Driver)

- Globus XIO
- (video capture driver)

- Globus XIO
- (display driver)

## Drivers

- Make 1 API do many types of IO
- Specific drivers for specific protocols/devices
- Transform
- Manipulate or examine data
- Do not move data outside of process space
- Compression, Security, Logging
- Transport
- Moves data across a wire
- TCP, UDP, File IO, Device IO
- Typically move data outside of process space

## Stack

- Transport
- Exactly one per stack
- Must be on the bottom
- Transform
- Zero or many per stack
- Control flows from user to the top of the stack, to the transport driver.

- Example Driver Stack

- Compression

- Logging

- TCP

## Handles

- Associated with a single stack
- When a handle is opened it is bound to a specific immutable stack.
- Handle is bound to stack at runtime.
- Used in all user IO operations
- open/close/read/write
- Contains the state of the *connection*
- Initialization
- Driver specific options.

## Example Handle Use

- Driver Stack

- Compression

- Logging

- TCP

- User

- Handle

- Handle is bound to the stack.
- User performs data operations on the handle.
- The data operation is passed down the stack.
- The data is compressed by the first driver.
- The logging driver logs the exchange in syslog.
- The TCP driver sends the compressed data across the wire.

## Globus XIO Framework

- Moves the data from user to driver stack.
- Manages the interactions between drivers.
- Assist in the creation of drivers.
- Asynchronous support.
- Close and EOF Barriers.
- Error checking
- Internal API for passing operations down the stack.

- User API

- Framework

- Driver Stack

- Transform

- Transform

- Transport

**Notes:**
This lays out what we will talk about
Portable between different flavors of unix, windows, threads, system configurations, etc...

## Driver Development Warning

- For power users only
- Assumption is you know what you are doing
- Weaker error reporting
- For the sake of efficiency.
- Possible to trip assertions.
- If the internal API is misused.

**Notes:**
This lays out what we will talk about
Portable between different flavors of unix, windows, threads, system configurations, etc...

## Interface functions

- A set function signatures
- Implemented by the driver
- Registered with Globus XIO
- Semantics
- Globus XIO makes calls to these functions expecting specific behaviour
- Ex: the read() interface function should produce some data, and the write() interface function should consume data.

**Notes:**
This lays out what we will talk about
Portable between different flavors of unix, windows, threads, system configurations, etc...

## Existing Drivers

- Transport
- TCP, UDP, File
- Transform
- GSI, HTTP, GSSAPI_FTP
- Advanced
- UDT, MODE E, GridFTP
- Coming Soon
- Your driver?

**Notes:**
This lays out what we will talk about
Portable between different flavors of unix, windows, threads, system configurations, etc...

## Performance

- XIO was designed to be highly efficient.
- Comparisons of the old globus_io and the old GridFTP server show an improvement of maybe 10-15% (can’t guarantee it is statisitically significant)
- Had one individual tell us that his protocol implementation actually ran faster when he wrote a driver due to our efficient threading and event handling.

## XIO Summary

- User API
- Simple open/close/read/write
- An abstraction to drivers
- Runtime configurable stacks
- Drivers
- Drivers do all the work
- Transport and Transform
- More information:
- _[http://www.globus.org/toolkit/docs/4.0/common/xio](http://www.globus.org/toolkit/docs/4.0/common/xio)__[/](http://www.globus.org/toolkit/docs/4.0/common/xio)_

## Server Data Storage Interface (DSI)

## New Server Architecture

- GridFTP (and normal FTP) use (at least) two separate socket connections:
- A control channel for carrying the commands and responses
- A Data Channel for actually moving the data
- Control Channel and Data Channel can be (optionally) completely separate processes.
- A single Control Channel can have multiple data channels behind it.
- This is how a striped server works.
- In the future we would like to have a load balancing proxy server work with this.

## New Server Architecture

- Data Transport Process (Data Channel) is architecturally, 3 distinct pieces:
- The protocol handler.  This part talks to the network and understands the data channel protocol
- The Data Storage Interface (DSI). A well defined API that may be re-implemented to access things other than POSIX filesystems
- ERET/ESTO processing.  Ability to manipulate the data prior to transmission.
- currently handled via the DSI
- In V4.2 we to support XIO drivers as modules and chaining
- Working with several groups to on custom DSIs
- LANL / IBM for HPSS
- UWis / Condor for NeST
- SDSC for SRB

## The Data Storage Interface (DSI)

- Conceptually, the DSI is very simple.
- There are a few required functions (init, destroy)
- Most of the interface is optional, and you can only implement what is needed for your particular application.
- There are a set of API functions provided that allow the DSI to interact with the server itself.
- Note that the DSI could be given significant functionality, such as caching, proxy, backend allocation, etc..

## Reliable File Transfer:Third Party Transfer

**Reliable File Transfer:****Third Party Transfer**

- RFT Service

- RFT Client

- SOAP Messages

- Notifications(Optional)

- GridFTP Server

- GridFTP Server

- Fire-and-forget transfer
- Web services interface
- Many files & directories
- Integrated failure recovery
- Has transferred 900K files

## Globus Toolkit: Open Source Grid Infrastructure

**Data Mgmt**

**Security**

**Common****Runtime**

**Execution Mgmt**

**Info Services**

- GridFTP

- Authentication
- Authorization

- Reliable
- File
- Transfer

- Data Access
- & Integration

- Grid Resource
- Allocation &
-  Management

- Index

- CommunityAuthorization

- Data
- Replication

- Community
- Scheduling
- Framework

- Delegation

- Replica
- Location

- Trigger

- Java Runtime

- C Runtime

- Python Runtime

- WebMDS

- Workspace
- Management

- Grid Telecontrol Protocol

**Globus Toolkit v4**
**www.globus.org**

- CredentialMgmt

**Globus Toolkit:**** Open Source Grid Infrastructure**

## For More Information

- Bill Allcock
- _[allcock@mcs.anl.gov](mailto:allcock@mcs.anl.gov)_
- www.mcs.anl.gov/~allcock
- Jennifer Schopf
- _[jms@mcs.anl.gov](mailto:jms@mcs.anl.gov)_
- www.mcs.anl.gov/~jms
- Globus Alliance
- _[www.globus.org](http://www.globus.org/)_
- GlobusWORLD 2006
- September 10-14, Washington, D.C.
- Jointly located with GGF and GridWorld

## Testing Overview

- Nightly builds and tests
- TestGrid at USC/ISI
- Stand up services for several weeks
- Perform stress tests
- TestGrid at LBNL
- Focus on WS Core performance and interoperability tests
- Performance and reliability testing is a major focus
- Component-specific approaches mostly
- Calls for Community Testing near release time - we welcome new testing help!

## Documentation

**Documentation**

- Current document significantly more detailed than earlier versions
- _[http://www.globus.org/toolkit/docs/4.0/](http://www.globus.org/toolkit/docs/4.0/)_
- Tutorials available for those of you building a new service
- _[http://www-unix.globus.org/toolkit/tutorials/BAS/](http://www-unix.globus.org/toolkit/tutorials/BAS/)_

- Globus® Toolkit 4: Programming Java Services (The Morgan Kaufmann Series in Networking), by Borja Sotomayor, Lisa Childers (Available through Amazon, £19.99  or $20)

## Credits

- Globus Toolkit v4 is the work of many talented Globus Alliance members, at
- Argonne Natl. Lab & U.Chicago
- USC Information Sciences Corporation
- National Center for Supercomputing Applns
- U. Edinburgh
- Swedish PDC
- Univa Corporation
- Other contributors at other institutions
- Supported by DOE, NSF, UK EPSRC, and other sources

## Tested Platforms

- Debian
- Fedora Core
- FreeBSD
- HP/UX
- IBM AIX
- Red Hat
- Sun Solaris

- SGI Altix (IA64 running Red Hat)
- SuSE Linux
- Tru64 Unix
- Apple MacOS X (no binaries)
- Windows – Java components only

- List of binaries and known platform-specific install bugs at
- http://www.globus.org/toolkit/docs/4.0/admin/ docbook/ ch03.html

## WSRF vs XML/SOAP

- The definition of WSRF means that the Grid and Web services communities can move forward on a common base
- Why Not Just Use XML/SOAP?
- WSRF and WS-N *are *just XML and SOAP
- WSRF and WS-N are just Web services
- Benefits of following the specs:
- These patterns represent best practices that have been learned in many Grid applications
- There is a community behind them
- Why reinvent the wheel?
- Standards facilitate interoperability

## OGSA-DAI

- Web services interface for accessing structured data resources
- Highly extensible and customisable
- Supports:
- relational databases, xml:dbs, structured files
- SQL, XPath, XQuery, text matching
- Runs on GT4, OMII_2, Axis 1.2, *GRIA, UNICORE*
- single client side API, multiple service flavours

## OGSA-DAI functions

- Interact with data resources:
-  Queries and updates
-  Data transformation / compression
-  Data delivery
- Extensible set of activities
- 30 predefined, framework for writing your own
- Customize for your project using
- Additional Activities
- Client Toolkit APIs
- Data Resource handlers
- A base for higher-level services
- federation, mining, visualization,...
- http://www.ogsadai.org.uk/

## The OGSA-DAI Framework

**OGSA-DAI service**

**Engine**

**SQLQuery**

**JDBC**

***Data ***
***Resources***

***Activities***

**DB2**

**The OGSA-DAI Framework**

**GZip**

**GridFTP**

**XPath**

**XMLDB**

**XIndice**

**readFile**

**File**

**SWISS**
**PROT**

**XSLT**

**SQL**
**Server**

**Data-**
**bases**

**Application**

**Client Toolkit**

**Notes:**
Data resources used to be called connection managers