# Presentation: 658681

## Globus and PPDG

- Jennifer Schopf
- Argonne National Lab

## Outline

- Overview of what Globus is about
- What’s is new
- New users (many!), Packaging, Condor-G, GridFTP, Replica catalog, Replica management,
- Review of PPDG-related activities
- Future plans:
- CAS, Gram-2

## The Globus Project in a Nutshell

- Core group at ANL, UC, USC/ISI
- Close collaborators who contribute code
- Condor @ UW, LBNL, NCSA
- Large community who submit ports, bug fixes, etc.—and use the code
- Support: DARPA, DOE, NASA, NSF, Microsoft
- We do
- Research (Design, prototype, evaluate new Grid tech)
- Development (Protocols, services, APIs, SDKs, tools)
- Deployment (Work with groups to deploy Grid infrastructure)
- Applications (The driver for all of the above)

## Layered Grid Architecture(By Analogy to Internet Architecture)

**Notes:**
We define Grid architecture in terms of a layered collection of protocols. 
Fabric layer includes the protocols and interfaces that provide access to the resources that are being shared, including computers, storage systems, datasets, programs, and networks.  This layer is a logical view rather then a physical view.  For example, the view of a cluster with a local resource manager is defined by the local resource manger, and not the cluster hardware.  Likewise, the fabric provided by a storage system is defined by the file system that is available on that system, not the raw disk or tapes.
The connectivity layer defines core protocols required for Grid-specific network transactions.  This layer includes the IP protocol stack (system level application protocols [e.g. DNS, RSVP, Routing], transport and internet layers), as well as core Grid security protocols for authentication and authorization.
Resource layer defines protocols to initiate and control sharing of (local) resources.  Services defined at this level are gatekeeper, GRIS, along with some user oriented application protocols from the Internet protocol suite, such as file-transfer.
Collective layer defines protocols  that provide system oriented capabilities that are expected to be wide scale in deployment and generic in function.  This includes GIIS,  bandwidth brokers, resource brokers,....
Application layer defines protocols and services that are parochial  in nature, targeted towards a specific application domain or class of applications.  These are   are  are  ... arrgh

## Grid Services Architecture:Fabric Layer Protocols & Services

- Just what you would expect: the diverse mix of resources that may be shared
- Individual computers, Condor pools, file systems, archives, metadata catalogs, networks, sensors, etc., etc.
- Few constraints on low-level technology: connectivity and resource level protocols form the “neck in the hourglass”
- Defined by interfaces not physical characteristics

## Grid Services Architecture:Connectivity Layer Protocols & Services

- Communication
- Internet protocols: IP, DNS, routing, etc.
- Security: Grid Security Infrastructure (GSI)
- Uniform authentication & authorization mechanisms in multi-institutional setting
- Single sign-on, delegation, identity mapping
- Public key technology, SSL, X.509, GSS-API
- Supporting infrastructure: Certificate Authorities, key management, etc.

- GSI: www.globus.org

## Grid Services Architecture:Resource Layer Protocols & Services

- Grid Resource Access and Mgmt (GRAM)
- Remote allocation, reservation, monitoring, control of compute resources
- GridFTP protocol (FTP extensions)
- High-performance data access & transport
- Grid Resource Information Service (GRIS)
- Access to structure & state information
- Network reservation, monitoring, control
- All integrated with GSI: authentication, authorization, policy, delegation

## Grid Services Architecture:Collective Layer Protocols & Services

- Index servers aka metadirectory services
- Custom views on dynamic resource collections assembled by a community
- Resource brokers (e.g., Condor Matchmaker)
- Resource discovery and allocation
- Replica catalogs
- Co-reservation and co-allocation services
- Etc., etc.

## Globus Toolkit Components

- Core protocols and services
- Grid Security Infrastructure
- Grid Resource Access & Management
- MDS information & monitoring
- GridFTP data access & transfer
- Other services
- Community Authorization Service
- DUROC co-allocation service
- Other Data Grid technologies
- Replica catalog, replica management service

## What’s New

- _Lots_ of new users
- Condor-G (next talk)
- Packaging
- GridFTP
- Replica catalog
- Replica manager

## Globus Applications and Deployments

- Application projects include
- GriPhyN, PPDG, NEES, EU DataGrid, ESG, Fusion Collaboratory, etc., etc.
- Infrastructure deployments include
- DISCOM, NASA IPG, NSF TeraGrid, DOE Science Grid, EU DataGrid, etc., etc.
- UK Grid Center, U.S. GRIDS Center
- Technology projects include
- Data Grids, Access Grid, Portals, CORBA, MPICH-G2, Condor-G, GrADS, etc., etc.

## Packaging Work

- Enables modular binary and source distributions
- NCSA (Bletzinger, Blau) has developed Grid Packaging Toolkit (GPT) for packaging Grid software.
- Set of scripts and Perl modules which manage packaging metadata
- Includes scripts that package software
- A set of files + metadata that allow the files to be managed as a unit
- Metadata stored in ASCII files encoded via XML
- ANL and ISI have worked closely with NCSA on applying GPT to the Globus Toolkit.
- A (very large) set of Globus Toolkit “packages”
- Convenience scripts for building and installing
- Alpha 1, 2, 3 users have tested, provided feedback.

## GridFTP: Basic Approach

- FTP is defined by several IETF RFCs
- Start with most commonly used subset
- Standard FTP: get/put etc., 3rd-party transfer
- Implement standard but often unused features
- GSS binding, extended directory listing, simple restart
- Extend in various ways, while preserving interoperability with existing servers
- Parameter set/negotiate, parallel transfers (multiple TCP streams), striped transfers (multiple hosts), partial file transfers, automatic & manual TCP buffer setting, progress monitoring, extended restart (via plug-ins)

## Our Approach to Replica Management

- Identify _replica cataloging_ and _reliable replication_ as two fundamental services
- Layer on other Grid services: GSI, transport, information service
- Use LDAP as catalog format and protocol, for consistency
- Use as a building block for other tools
- Advantage
- These services can be used in a wide variety of situations

## Replica Manager Components

- Replica catalog definition
- LDAP object classes for representing logical-to-physical mappings in an LDAP catalog
- Low-level _replica catalog_ API
- globus_replica_catalog library
- Manipulates replica catalog: add, delete, etc.
- High-level _reliable replication_ API
- globus_replica_manager library
- Combines calls to file transfer operations and calls to low-level API functions: create, destroy, etc.

## Activities in Progress with PPDG

- GDMP knowledge building
- Replica Catalog Performance Analysis
- SC2001 Demo Coordination
- PPDG – GriPhyN Coordination

## GDMP/Globus Coordination

- Studying:
- How to support GDMP deployment in ATLAS testgrid, and RC integration
- How to integrate with demos and PPDG deployments
- How to integrate into DGRA
- How to integrate with reliable file transfer
- Now running on two test machines at ANL.

## Replica Catalog Performance Analysis

- Measuring performance at large capacities
- Measuring OpenLDAP, Netscape 4.13, Oracle ODS
- Will measure direct relational mapping
- Initial results show problems with current schema
- Will examine remedies in LDAP & SQL

## SC2001 Demo Coordination

- Help in coordinating sites
- Adding resources from ANL MCS
- Adding visualization facilities
- Adding Grid Info Services
- Coordinating themes and messages between PPDG and GriPhyN

## PPDG – GriPhyN Coordination

- Focal point is replica catalog work
- Joint effort to understand and catalog data
- Define data file types, quantities, usage
- Need recording process of data production to facilitate data reproduction
- Coordination of monitoring, resource management work
- Coordination of research and production integration
- get virtual data on PPDG horizon

## Globus Toolkit Status

- New release coming soon
- In alpha3 now, beta this month, 4Q01 release
- New packaging: enables modular binary and source distributions
- GRAM 1.5: enhanced robustness
- MDS-2.1: security, better performance, etc.
- GridFTP & Replica Mgt: Data management
- GSI extensions, Community Authorization Svc
- Prototype now, 4Q01 alpha, 1Q02 release.
- Java, other “Commodity Grid” toolkits

## Data Services to Come:

- More flexible catalog
- Arbitrary file mapping
- Hierarchical catalog
- Reliable File Transfer
- Performance
- Ease of installation and use
- Data replica selection
- Automatic data propagation

## Replica Catalog Servicesas Building Blocks: Examples

- Combine with information service to build _replica selection_ services
- E.g. “find best replica” using performance info from NWS and MDS
- Use of LDAP as common protocol for info and replica services makes this easier
- Combine with application managers to build _data distribution_ services
- E.g., build new replicas in response to frequent accesses

## GSI Futures:Community Authorization

- 2.  CAS reply, with

-      and resource CA info

- user/group

- membership

- resource/collective

- membership

- collective policy

- information

**CAS**

** **

*Does the *

*collective policy *

*authorize this *

*request for this *

*user?*

* *

**User**

** **

- 1.  CAS request, with

-      resource names

-      and operations

**Resource**

** **

*Is this request *

*authorized for *

*the CAS?*

* *

*Is this request *

*authorized by *

*the *

*capability?*

* *

- local policy

- information

- 3.  Resource request,

-      authenticated with

- capability

- 4.  Resource reply

- capability

## Globus Current Research Includes:

- Resource management
- GRAM-2, Reservation, co-allocation, policy, etc.
- Security
- (Restricted) delegation, CAS, PKI usability (online credential repositories, etc.)
- Data Grid technologies
- Replication, robust/fast transport, etc.
- Tools
- Workflow, resource discovery & selection

## More Information

- Globus
- http://www.globus.org
- Globus data grid work in general
- http://www.globus.org/datagrid/
- Interfacing GDMP and Globus
- http://www.globus.org/research/papers/gdmp_hpdc_final_version.pdf
- Getting started with the Globus Replica Cat
- http://www.globus.org/datagrid/deliverables/replicaGettingStarted.pdf