# Presentation: 083319

## dev.globusa framework for open Globus development

- UK National eScience Centre
- Argonne National Laboratory

- dev.globus*a framework for open **Globus development*

## Globus Open Source History

- 1996 – First established as an open source software project
- Globus development team limited to ANL/UC and ISI
- 2003 – Globus Alliance formed
- EPCC (Edinburgh, UK), NCSA (IL, USA), and KTH (Sweden)
- 2005 – dev.globus.org formed
- source code development *infrastructure*
- meritocratic *governance model*

## Outline

- What is Globus
- A little scene setting
- Some technology highlights
- The dev.globus environment
- A little on the governance and infrastructure
- Kinds of projects
- Incubator projects

## What is a Grid?

- Resource sharing
- Computers, storage, sensors, networks, ...
- Sharing always conditional: issues of trust, policy, negotiation, payment, ...
- Coordinated problem solving
- Beyond client-server: distributed data analysis, computation, collaboration, ...
- Dynamic, multi-institutional virtual orgs
- Community overlays on classic org structures
- Large or small, static or dynamic
- Hard because of lack of central control, shared resources, crossing of administrative domains

## The Role of the Globus Toolkit

- A collection of solutions to problems that come up frequently when building collaborative distributed applications
- Heterogeneity
- A focus, in particular, on overcoming heterogeneity for application developers
- Standards
- We capitalize on and encourage use of existing standards (IETF, W3C, OASIS, GGF)
- GT also includes reference implementations of new/proposed standards in these organizations

**Notes:**
Heterogenity: 

Trying to support heterogenity:  no preconditions:
	- different hardware (types of machines, different configurations, different “data producers”, etc.)
	- different software (different applications, different languages, different operating systems, etc.)

Standards:
	Standards enable interoperability.  An example:
		- If the community agrees to standardize certain patterns for interacting with a type of service, then services
can interact with each other
		- WSRF

## Heterogeniety:Globus is an Hour Glass

- Local sites have an their own policies, installs – heterogeneity!
- Queuing systems, monitors, network protocols, etc
- Globus unifies
- Build on Web services
- Use WS-RF, WS-Notification to represent/access state
- Common management abstractions & interfaces

- Local heterogeneity

- Higher-Level Services
- and Users

- Standard GT4
- Interfaces

## Standards:What does WSRF give Globus

- Reference implementation of WSRF and WS-N functions
- Naming and bindings (basis for virtualization)
- Every resource can be _uniquely referenced_ and has one or more _associated services_ for interacting
- Lifecycle (basis for resilient state management)
- Resources created by svcs following a _factory_ pattern
- Resource destroyed _immediately_ or _scheduled_
- Information model (basis for monitoring & discovery)
- _Resource properties_ associated with resources
- Operations for _querying_ and _setting_ this info
- Asynchronous _notification_ of changes to properties
- Service groups (basis for registries & collective svcs)
- Group membership rules and membership management
- Base fault type

## Standards: WSRF vs XML/SOAP

- The definition of WSRF means that the Grid and Web services communities can move forward on a common base
- Why Not Just Use XML/SOAP?
- WSRF and WS-N *are *just XML and SOAP
- WSRF and WS-N are just Web services
- Benefits of following the specs:
- These patterns represent best practices that have been learned in many Grid applications
- There is a community behind them
- Why reinvent the wheel?
- Standards facilitate interoperability

## Globus is a Building Block

- Basic components for grid functionality
- Highest-level services are often application specific, we let applications concentrate there
- Easier to reuse than to reinvent
- Compatibility with other Grid systems comes for free
- We provide basic infrastructure to get you one step closer

## Globus Toolkit: Open Source Grid Infrastructure

**Data Mgmt**

**Security**

**Common****Runtime**

**Execution Mgmt**

**Info Services**

- GridFTP

- C Security

- Reliable
- File
- Transfer

- Replica
- Location

- Grid Resource
- Allocation &
-  Management

- MDS4

- CAS/SAML
- Utilities

- Virtual
- Workspaces

- Delegation

- Data
- Replication

- C Core
- Utilities

- C WS
- Core

- COG
- jGlobus

- Dynamic
- Accounts

**Globus Projects**
**dev.globus.org**

- Globus Toolkit: Open Source Grid Infrastructure

- Java WS
- Core

- Core WS
- schema

**Other**
**Projects**

- GT Release
- Manuals

- GT
- Distribution

- Incubator
- Management

- XIO

**Notes:**
c core utilities, c ws core, cog jglobus, core ws schema, java ws core,> >xio, gridftp, rft, replica location, data replication, gram, dynamic> >accounts, virtual workspaces, mds4, c security, cas/saml utilities,> >delegation service, gt release manuals, globus toolkit.

## Data Mgmt:GridFTP

- Data Mgmt:GridFTP

- A high-performance,
- 	secure, reliable trans.
- 	protocol optimized  for
- 	high-bandwidth WANs
- IPv6 Support
- XIO for different transports
- Striping  multi-Gb/sec wide area transport
- Pluggable
- Front-end: e.g., future WS control channel
- Back-end: e.g., HPSS, cluster file systems
- Transfer: e.g., UDP, NetBLT transport

- Disk-to-disk onTeraGrid

**Notes:**
Future: integration of provisioning
E.g., storage reservation

## Data Mgmt: Reliable File Transfer

- A WSRF service for queuing file transfer requests
- Server-to-server transfers
- Checkpointing for restarts
- Database back-end for failovers
- Integrated failure recovery
- Allows clients to requests transfers and then “disappear”
- No need to manage the transfer
- Status monitoring available if desired

## Data Management:Replica Location Service

- Data Management:Replica Location Service

- Identify location of files via logical to physical name map
- Distributed indexing of names, fault tolerant update protocols
- GT4 version scalable & stable
- Managing ~40 million files across ~10 sites

- Index

- Index

| Local DB | Update send (secs) | Bloom filter  (secs)  | Bloom filter (bits) |
| --- | --- | --- | --- |
| 10K | <1 | 2 | 1 M |
| 1 M | 2 | 24 | 10 M |
| 5 M | 7 | 175 | 50 M |

## Execution Management: GRAM

- Common WS interface to schedulers
- Unix, Condor, LSF, PBS, SGE, ...
- More generally: interface for process execution management
- Lay down execution environment
- Stage data
- Monitor & manage lifecycle
- Kill it, clean up
- Not a scheduler/broker, but an uniform, web-service interface to local resources

## Information Services: MDS4

- Monitoring and Discovery System
- Set of information providers
- Interfaces to data sources: any web service, cluster systems, queuing systems, archives, file scraping, etc
- Higher level tools
- Index Service – registry with cache
- Trigger Service – warning system
- GUI interface - WebMDS

## Globus Toolkit: Open Source Grid Infrastructure

**Data Mgmt**

**Security**

**Common****Runtime**

**Execution Mgmt**

**Info Services**

- GridFTP

- C Security

- Reliable
- File
- Transfer

- Replica
- Location

- Grid Resource
- Allocation &
-  Management

- MDS4

- CAS/SAML
- Utilities

- Virtual
- Workspaces

- Delegation

- Data
- Replication

- C Core
- Utilities

- C WS
- Core

- COG
- jGlobus

- Dynamic
- Accounts

**Globus Projects**
**dev.globus.org**

- Globus Toolkit: Open Source Grid Infrastructure

- Java WS
- Core

- Core WS
- schema

**Other**
**Projects**

- GT Release
- Manuals

- GT
- Distribution

- Incubator
- Management

- XIO

**Notes:**
c core utilities, c ws core, cog jglobus, core ws schema, java ws core,> >xio, gridftp, rft, replica location, data replication, gram, dynamic> >accounts, virtual workspaces, mds4, c security, cas/saml utilities,> >delegation service, gt release manuals, globus toolkit.

## Outline

- What is Globus
- Some technology highlights
- The dev.globus environment
- A little on the governance and infrastructure
- Kinds of projects
- Incubator projects

## Why is Globus Software Open Source?

- To allow for inspection
- For consideration in standardization processes
- To encourage adoption
- In pursuit of ubiquity and interoperability
- To encourage contributions
- Harness the expertise of the community

## Open Contribution

- But distributing code under an open source license does not guarantee open development!
- Open development requires open processes
- So we have created dev.globus to facilitate contributions
- http://dev.globus.org/

## Governance Model

- Based on Apache Jakarta
- Individual development efforts organized as projects
- Consensus-based decision making
- Control over each project in the hands of its most active and respected contributors (committers)
- Globus Management Committee (GMC) providing overall guidance and conflict resolution

## Common Infrastructure

- Code repositories (CVS, SVN)
- Mailing lists
- *-dev, *-user, *-announce, *-commit for every project
- Issue tracking (bugzilla)
- Including roadmap info for future development
- Wikis
- Known interactions for people accessing your project

## Sample

- http://dev.globus.org/wiki/GRAM

## Technology Projects

- Common runtime projects
- C Core Utilities, C WS Core, CoG jglobus, Core WS Schema, Java WS Core, XIO
- Data projects
- GridFTP, Reliable File Transfer, Replica Location, Data Replication
- Execution projects
- GRAM, Dynamic accounts, Virtual workspaces
- Information services projects
- MDS4
- Security Projects
- C Security, CAS/SAML Utilities, Delegation Service, MyProxy

## Non-Technolgy Projects

- Distribution Projects
- Globus Toolkit Distribution
- Process was used for April 4.0.2 release
- Documentation Projects
- GT Release Manuals
- Incubation Projects
- Incubation management project
- And any new projects wanting to join

## Incubator Process in dev.globus

- Entry point for new Globus projects
- Incubator Management Project (IMP)
- Oversees incubator process form first contact to becoming a Globus project
- Quarterly reviews of current projects
- Process being debugged by “Incubator Pioneers”
- http://dev.globus.org/wiki/IncubatorDraft

## Process (1 of 3)

- Project proposes itself as a *Candidate*
- A proposed name for the project;
- A proposed project chair, with contact info;
- A list of the proposed committers for the project;
- An overview of the aims of the project;
- An overview of any current user base or user community, if applicable;
- An overview of how the project relates to other parts of Globus;
- A summary of why the project would enhance and benefit Globus.

## Process (2 of 3)

- IMP meet, discuss, and accept project as a *ProtoProject*
- ProtoProject now part of the Incubator framework
- Get assigned a Mentor to help
- Member of IMP
- Bridge between Globus and new ProtoProject
- Opportunity to get up to speed on Globus Development process

## Process (3 of 3)

- Quarterly reviews by IMP determine
- Stay a ProtoProject
- Retire
- Escalate to a full Globus project
- Escalation when ProtoProject passes checklist
- Legal
- Meritocracy
- Alignment/Synergy
- Infrastructure

## Current List of “Incubator Pioneers”

- Metrics
- Project for usage stats and usage metrics for GT4
- Lee Liming, University Chicago
- GridWay
- MetaScheduler that works over GT2 and GT4 GRAM, MDS
- Ignacio Lorente, University Madrid
- GridShib
- Integration of GSI and Shibboleth security
- Von Welch, NCSA
- Send email to incubator-committers@globus.org to discuss new project ideas!

## You Can Begin Participating in Globus Development Today!

- Monitor and comment on Globus development discussions; recent threads include:
- GT Backward Compatibility (gt-dev@globus.org)
- 4.2 Wish List for GRAM (gram-dev@globus.org)
- Submit bug fixes and other contributions
- Start your own Globus project!

## Globus Toolkit: Open Source Grid Infrastructure

**Data Mgmt**

**Security**

**Common****Runtime**

**Execution Mgmt**

**Info Services**

- GridFTP

- C Security

- Reliable
- File
- Transfer

- Replica
- Location

- Grid Resource
- Allocation &
-  Management

- MDS4

- CAS/SAML
- Utilities

- Virtual
- Workspaces

- Delegation

- Data
- Replication

- C Core
- Utilities

- C WS
- Core

- COG
- jGlobus

- Dynamic
- Accounts

**Globus Projects**
**dev.globus.org**

- Globus Toolkit: Open Source Grid Infrastructure

- Java WS
- Core

- Core WS
- schema

**Other**
**Projects**

- GT Release
- Manuals

- GT
- Distribution

- Incubator
- Management

- XIO

**Notes:**
c core utilities, c ws core, cog jglobus, core ws schema, java ws core,> >xio, gridftp, rft, replica location, data replication, gram, dynamic> >accounts, virtual workspaces, mds4, c security, cas/saml utilities,> >delegation service, gt release manuals, globus toolkit.

## Summary

- Globus has been an open source toolkit for 10 years
- Our recent move to dev.globus.org will make it much easier for outside contributors and projects
- Please consider helping out!

## Credits

- Our guidelines are derived directly from those of the Jakarta project and the Apache project within which Jakarta lives (with modifications to reflect specific properties of the Globus community)
- Thanks to Jakarta and Apache for the effort they have put into developing and documenting their processes!

## More Information

- Jennifer M. Schopf
- jms@mcs.anl.gov
- Globus Development Environment
- dev.globus.org
- Globus WORLD 2006
- http://www.globusworld.org/
- Washington D.C. Convention Center
- September 11 - 15, 2006
- Tenth Birthday Party – you’re invited!