# Presentation: 002413

## CEDPS and CDIGS:A Tale of Two (Globus) Projects 

- Jennifer M. Schopf
- Argonne National Laboratory
- University of Chicago
- jms@mcs.anl.gov

## SciDAC Center for Enabling Distributed Petascale Science

- Argonne National Laboratory
- Fermi National Accelerator Laboratory
- Lawrence Berkeley National Laboratory
- University of Southern California
- University of Wisconsin
**www.cedps.net**
- PI: Ian Foster
- Director: Jennifer Schopf, ANL jms@mcs.anl.gov

## SciDAC in a Nutshell

- Advance scientific discovery esp. as related to HPC resources, PetaScale computing
- 3 institutes and 10 Centers for Enabling Technologies funded, as well as many Scientific Application Partnerships
- Center for Enabling Distributed Petascale Science
- CEDPS – “seds” (silent P)
- DOE SciDAC Center for Enabling Technology
- July 1, 2006 – June 30, 2011, $2.4M/yr
- Collaboration between 5 sites
- Argonne National Laboratory
- Fermi National Laboratory
- Lawrence Berkeley National Laboratory
- USC Information Sciences Institute
- University of Wisconsin Madison

## The Petascale Data Challenge

**The Petascale Data Challenge**

- DOE facilities generate**many** **petabytes** of data(2 petabytes = **all** U. S. academic research libraries!)

- Massive data

- U

- U

- U

- U

- U

- DOE
- facilities

**Remote users **(at labs  universities, industry)       need data!

**Rapid, reliable access**key to maximizingvalue of $B facilities

- U

- Remotedistributed users

- U

- U

## Bridging the Divide (1):Move Data to Users When & Where Needed

**Reliable**: recoverfrom many failures
**Predictable**: data arrives when scheduled
**Secure**: protect expensive resources & data
**Scalable**: deal with manyusers & much data

***Bridging the Divide *****(1):****Move Data to Users ****When & Where ****Needed**

- C

- B

- A

**Fast**: >10,000x faster thanusual Internet

- “Deliver this 100  Terabytes to locations A, B, C by 9am tomorrow”

## Bridging the Divide (2):Allow Users to Move ComputationNear Data

**Flexible**: easyintegration of functions
**Secure**: protect expensive resources & data
**Scalable**: deal with manyusers & much data

***Bridging the Divide *****(2):****Allow Users to Move ****Computation****Near Data**

- A

**Science services**:provide analysisfunctions neardata source

- “Perform mycomputation F ondatasets X, Y, Z”

- Y

- Z

- X

- F

## Bridging the Divide (3):Troubleshoot End-to-EndProblems

**Instrument**: includemonitoring points inall system components
**Monitor**: collect data inresponse to problems
**Diagnose**: identify thesource of problems

***Bridging the Divide *****(3):****Troubleshoot ****End-to-End****Problems**

- C

- B

- A

- “Why did my datatransfer (or remoteoperation) fail?”

**Identify** & **diagnose** failures & performanceproblems

## Data Services in CEDPS

- Ann Chervenak, ISI, is the CEDPS Data lead
- And these slides are adapted from hers
**Develop tools and techniques for *****reliable, high-performance, secure, and policy-driven placement of data within a distributed science environment***
- Data *placement* and *distribution services* that implement different data distribution and placement behaviors
*Managed Object Placement Service—*enhancement to today’s GridFTP—that allows for management of:
- Space
- Bandwidth
- Connections
- Other resources needed to endpoints of data transfers

## CEDPS Data Services

**CEDPS Data Services**

- Data Placement Strategy Layer
- Decide where to place objects and 			replicas in the Grid environment
- Policy-driven, based on needs of 		application
- Effectively creates a placement workflow that is then passed to the Reliable Distribution Service Layer for execution
- Reliable Distribution Service Layer
- Responsible for carrying out the distribution or placement “plan” generated by higher-level service
- Provide feedback to higher level placement services on the outcome of the placement workflow
- Call on lower-level services to coordinate

## Managed Object Placement Service

- Data Transfer Service (e.g., GridFTP)
- Disk Space Manager: for doing local storage allocation (NeST, D-Cache, SRM)
- Connection Management: incoming and outgoing
- Scheduler (C-Cache, Stork, RFT): includes queue
- Eventually interact with both endpoints of transfer

## Science Services in CEDPS

- Kate Keahey, ANL, is the CEDPS Science Services lead
**Develop tools and techniques for *****construction, operation, and provisioning of scalable science services***
*Service construction tools* that make it easy to take application code (whether simulation, data analysis, command line program, or library function) and wrap it as a remotely accessible service
*Service provisioning tools* that allow dynamic management of the computing, storage, and networking resources required to execute a service, and the configuration of those resources to meet application requirements.
- Services to move computation to data

## Scalable Science Services

- Service-enabling applications is too hard for application developers
- Community may already have a required data analysis function, typically implemented as a standalone (parallel or sequential) program or library
- Turning this existing implementation into a service is arduous and time-consuming
- Process involves knowledge about the mechanics of service container implementation and Grid mechanisms
- Solution: Formalize and automate this process in service wrapping tools
- Automate the process of embedding application code into an Application Hosting Service (AHS)

## Service Construction:Application Hosting Environment

- An AHS involves
- An application-specific service interface to analysis code
- A management interface that allows the service provider to monitor and control the AHSs execution
- The AHS interacts with external policy decision points (PDPs) for policy enforcement and provisioners for service scalability

## Service Provisioning with Variance

- If the load on a service varies significantly over time, then the number of resources allocated to the service must also vary
- Solution: Introduce provisioning tools that
- Allow a service provider to specify desired performance levels, while other system components
- Monitor service behavior
- Dynamically add and remove resources applied to service activity, so as to adapt to varying service load

## Troubleshooting in CEDPS

- Brian Tierney, LBNL, is the CEDPS TS lead
**Develop tools and techniques for *****failure detection and diagnosis***** in distributed systems**
- Better logs and logging services to understand application and service behavior
- Better diagnostic tools to discover failures and performance faults, and for notification of these errors

## Logs and Log Management

- Logging and monitoring data is hard to find and manage
- Data scattered across sites and within a site
- No agreed on “standards” for what logs should look like
- Large volumes of data are possible
- A heavily loaded GridFTP server with all network and disk I/O instrumented generates 1.1 GB of log data per hour
- Introduce “Best Practices” for logging
- Then implement for GT, Condor, and others
- Log collection service
- Work to deploy on OSG as first case
- Log management functions
- Provide for turning logging on and off, moving log data back to the user for analysis, deleting old log data, etc.

## Automatic Failure Detection

- Automated failure detection for infrastructure services
- Distributed systems often run many services with little or no 24-7 support
- Failures are discovered only when user tasks fail
- Solution: Deploy monitoring information providers to gather behavior data on resources and services
- Use this data to warn system administrators of faults and to study how fault behaviors change over time.
- Extend MDS4 Trigger Service
- Create a NetLogger-based event-driven monitoring system to gather runtime data from services and applications

## Performance Degradation Detection

- Performance degradation is often overlooked
- Many systems have long running services used by different users at different times, and no single group tracking behavior
- Solution: Develop and apply analysis functions to archived background monitoring and event-driven log data
- Track executions dynamically and compare them to past behaviors and service guarantees
- When service behavior degrades or deadlines are threatened the proper services or users can be notified to take action
- Develop analysis components to examine end-to-end bottleneck analysis and detection, trend analysis, and alarm generation

## Work with Applications 

-    Strong collaborations with DOE applications, SciDAC software centers, and DOE facilities

## Work with Applications

- Earth Systems Grid
- Data work
- Error and warning alpha tester
- Open Science Grid
- Data services for ATLAS and CMS
- Services work with STAR
- Logging service Alpha tester
- Second Wave
- LIGO (OSG) – Data focus
- Fusion- FACETS – Scalable services focus
- DANCE – Scalable services focus
- Combustion – Troubleshooting focus
- GADU – Scalable services and data focus

## Community Driven Improvement of Globus Software (CDIGS)

- University of Chicago
- University of Southern California
**www.cdigs.org**
- PIs: Ian Foster, Carl Kesselman
- Director: Dan Fraser, UC/ANL
- _[fraser@mcs.anl.gov](mailto:fraser@mcs.anl.gov)_
- (until recently, Lisa Childers, UC/ANL)

## Community Driven Improvement of Globus Software (CDIGS)

- A National Science Foundation Office of Cyberinfrastructure Project
- Awarded Dec 2005, 5 years
- Collaboration between
- USC Information Sciences Institute
- University Chicago
- Four high level goals
- Evolve and enhance Globus functionality, performance, scalability and robustness
- Improve usability and manageability so as to decrease the cost and complexity of deploying, operating and using Globus infrastructure
- Support major NSF users and communities
- Expand the Globus community

## Goal #1Evolving and Enhancing Globus

- Enhancements to software components to address user requirements for functionality, performance, and robustness
- Across all areas of the toolkit
- Improvements in GRAM performance and robustnes
- Creation of a new attribute-based authorization framework
- Release of the workspace service for virtual machine management
- Enhancements to the performance and functionality of information services
- And of course bug fixes and basic support
- In 2005, we resolved 667 bugzilla issues, including both feature requests and error reports.

## Goal #2: Improving Usability

- Target on two end user communities
- System administrator- responsible for deploying and maintaining a Globus based Grid infrastructure
- Overhauled the software packaging- simplifying the installation and configuration process of the toolkit
- Streamlining the installation requirements for individual components
- Greatly improved documentation targeted towards the system administrator
- End user – working with Globus services or developing aps
- Continued improvement of content, coverage, and readability of Globus documentation
- Availability of a number of user-centric tools that have significantly reduced the barrier to entry for new and existing users and developers
- Introduce service development environment, part of Incubator projects

## Goal #3: Supporting Users

- Work with key NSF user communities and technical meetings
- BIRN, GEON, LEAD, LIGO, OSG, TeraGrid, etc
- Presentation of numerous talks and tutorials to other groups
- DOE (ESG, Fusion, etc)
- International (EGEE, UK eScience, etc)
- Large community meetings (GGF/OGF, SC, etc)
- Authoring of many technical articles and much documentation
- Creation and presentation of the Globus roadmap
- Globus software releases, as well as contributions to other releases such as VDT

## Outreach

- Over 100 letter writers supported the CDIGS proposal
- To date, outreach has been somewhat ad hoc
- “Intro to Globus” talks
- Half-day and full day overviews
- Hands on tutorials
- Component-centric work as well
- Currently developing a more coordinated plan to interact with outreach for many large projects and centers, and better focus our efforts

## Upcoming Outreach To Note

- RoadMap Meeting2 in the process of being planned
- Wider scope of attendees
- Looking at ways to get more detailed feedback
- Tutorials and presentations planned for
- OGF 20 (May, Manchester)
- TG 2007* (June, Wisc)
- CHEP (Sept, Canada)
- OGF21 (Oct, West Coast),
- SC2007* (Nov, Reno) and others
- Currently taking visit requests at outreach@globus.org
- *proposed

## Goal #4: Expanding the Communityof Globus Contributors

- Creation of the dev.globus community process
- Provides an open forum for discussion and enhancement of current Globus
- Enabling the integration of ~20 new components from the US and Europe as “incubators”

## Globus Development Environment

- Based on Apache Jakarta
- Individual development efforts organized as projects
- Consensus-based decision making
- Control over each project in the hands of its most active and respected contributors (committers)
- Globus Management Committee (GMC) providing overall guidance and conflict resolution

## Common Infrastructure

- Code repositories (CVS, SVN)
- Mailing lists
- *-dev, *-user, *-announce, *-commit
- Issue tracking (bugzilla)
- Including roadmap info for future development
- Wikis
- License (Apache 2)
- Known interactions for people accessing your project

## Current Technology Projects

- Common runtime projects
- C Core Utilities, C WS Core, CoG jglobus, Core WS Schema, Java WS Core, Python Core, XIO
- Data projects
- Data Replication, GridFTP, OGSA-DAI, Reliable File Transfer, Replica Location
- Execution projects
- GRAM, GridWay, MPICH-G2
- Information services projects
- MDS4
- Security Projects
- C Security, CAS/SAML Utilities, Delegation Service, GSI-OpenSSH, MyProxy

## Non-Technology Projects

- Distribution Projects
- Globus Toolkit Distribution
- Process was used for April 4.0.2 & 4.0.3 releases
- Documentation Projects
- GT Release Manuals
- Incubation Projects
- Incubation management project
- And any new projects wanting to join

## Incubator Process in dev.globus

- Entry point for new Globus projects
- Incubator Management Project (IMP)
- Oversees incubator process form first contact to becoming a Globus project
- Quarterly reviews of current projects
- Process being debugged by “Incubator Pioneers”
- http://dev.globus.org/wiki/Incubator/ Incubator_Process

## Incubator Projects

- “Escalation”

- B

- A

- Proposal

- Any Grid Project

## Current Incubator Projectsdev.globus.org/wiki/Welcome# Incubator_Projects

- Distributed Data Management (DDM)
- Dynamic Accounts
- Grid Authentication and Authorization with Reliably Distributed Services (GAARDS)
- Grid Development Tools for Eclipse (GDTE)
- GridShib
- Grid Toolkit Handle System (gt-hs)
- Higher Order Component Service Architecture (HOC-SA)
- Introduce

- Local Resource Manager Adaptors (LRMA)
- Metrics
- MEDICUS
- OGCE
- Portal-based User Registration Service (PURSe)
- ServMark
- UCLA Grid Portal Software (UGP)
- Workflow Enactment Engine Project (WEEP)
- Cog Workflow
- Virtual Workspaces

## We’ve Just had Our First Escalation!

- GridWay Meta Scheduling Project
- Iganacio Llorente, Universided Complutense de Madrid
- Provides scheduling functionality similar to that found on local DRM (Distributed Resource Management) systems
- Advanced scheduling capabilities on a Grid consisting of Globus services
- Dynamic discovery & selection
- Opportunistic migration
- Support for the definition of new scheduling policies
- Detection and recovery from remote and local failures
- Straightforward deployment that does not require new services apart from those provided by the Globus Toolkit: MDS, GRAM, GridFTP and RFT

## CDIGS Summary

- Evolve and enhance Globus functionality, performance, scalability and robustness
- Improve usability and manageability so as to decrease the cost and complexity of deploying, operating and using Globus infrastructure
- Support major NSF users and communities
- Expand the Globus community

## Key CDIGS and CEDPS Overlap

- A large portion of CEDPS participants are also funded under CDIGS, and vice versa
- Goal is complementary programs
- Actively working on cross-over discussions
- Management
- Ian Foster is PI on both and provides general direction
- I’m director of CEDPS and lead CDIGS outreach
- Dan is director of CDIGS and a participant in CEDPS data area
- Ann Chervenak is CEDPS data lead, and oversees several CDIGS components, many others like this

## Cross Over: Functionality

- CEDPS is a “traditional” project with pre- defined deliverables
- http://cedps.net/wiki/images/2/20/ CEDPS_management_plan.pdf
- CDIGS is focusing on community and support
- New functionality funded by CDIGS comes from published roadmaps and community requests - _[http://snipurl.com/18c0q](http://snipurl.com/18c0q)_
- Deliverables are community defined
- Much new functionality is released to the broader community form a variety of funding sources

## Cross Over: Outreach

- CEDPS will heavily leverage CDIGS outreach
- Also working with SciDAC outreach center

## For More Information

- Jennifer Schopf
- jms@mcs.anl.gov
- http://www.mcs.anl.gov/~jms
- CEDPS
- _[http://www.cedps.net](http://www.cedps.net/)_
- CDIGS
- www.cdigs.org
- Dev.globus
- _[http://dev.globus.org](http://dev.globus.org/)_
- Globus Main Website
- _[http://www.globus.org](http://www.globus.org/)_