# Presentation: 170385

## Distributed Systems DepartmentCRD Division

- Department Head – William E. Johnston (WEJohnston@lbl.gov)
- Deputy – Deborah A. Agarwal (DAAgarwal@lbl.gov)
- web site:  http://www-itg.lbl.gov/

## Mission Statement

- Enable scientists to address complex and large-scale computing and data analysis problems beyond what is possible today.
- The software developed falls roughly into two categories:
- Middleware providing basic services and capabilities in the DOE Science Grid
- Applications and middleware supporting more informal real-time collaboration between geographically remote researchers

## Environment

- Large-scale science and engineering projects
- Heterogeneous compute resources
- Multiple information and storage systems
- Heterogeneous communication mechanisms
- Instruments
- Large number of people
- Geographically dispersed
- Goal
- Enable the routine interaction of these resources to facilitate large-scale science and engineering

## Distributed Systems Department Projects

- Secure Grid Technologies
- DOE Science Grid
- Grid Services
- Python Cog Kit
- Akenti Distributed Authorization
- Cryptographic Aspects of the Grid Security Architecture
- Network Monitoring
- Net 100 - Netlogger toolkit
- Distributed Monitoring Framework
- Bandwidth Estimation and its Applications
- Scientific collaboration tools
- Secure and reliable group communication
- Peer-to-peer information sharing

## DOEScience Grid

- Initial Science Grid Configuration

**NERSC****Supercomputing****& Large-Scale Storage**

**PNNL**

**LBNL**

**ANL**

**ESnet**

**Europe**

**DOE****Science Grid**

**ORNL**

**ESNet**

**MDS**
**CA**

**Grid Managed Resources**

**User Interfaces**
**Application Frameworks**
**Applications**

**Grid Services:**** Uniform access to distributed resources**

**Asia-Pacific**

**Funded by the U.S. Dept. of Energy, Office of Science,****Office of Advanced Scientific Computing Research,****Mathematical, Information, and Computational Sciences Division**

## DOE Science Grid:Three Strongly Linked Challenges

- How do we reliably and effectively deploy and operate a DOE Science Grid?
- Requires coordinated effort by multiple labs
- ESNet for identity certificate and directory services
- Manage basic software plus import other R&D work
- Distributed system monitoring and management
- What else? Will see.
- Application partnerships linking computer scientists and application groups
- How do we exploit Grid infrastructure to facilitate DOE applications?
- Enabling R&D
- Extending technology base for Grids
- Packaging Grid software for deployment
- Developing application toolkits
- Web services for science portals

## Roadmap for the Science Grid

**Roadmap for the Science Grid**

**Grid compute and data resources****federated across partner sites using Globus**

**Pilot simulation and collaboratory users**

**Production Grid Info. Services and Certificate Authority **

**Auditing and monitoring services**

**Grid system administration**

**Help desk, tutorials, application****integration support**

**Integrate R&D from other projects**

**Scalable****Science****Grid**

## Current resources and services

- All registered in MDS at all sites
- 3 Sun SMPs (4 and 8 CPU) (LBNL)
- Several dozen Linux systems
- Visualization server
- NERSC compute and HPSS systems (on track)
- 256 node Itanium cluster (PNNL)
- Data Grid cluster (ANL)
- -----------------------
- Services
- myProxy server
- DOE Science Grids CA and Ras
***PI: Bill Johnston, ***
***    ******Keith Jackson, Josh Boverhof, Matt Rodriguez***

## Grid Services Project

- Develop Open Grid Services Architecture
- Grid Services Specifications
- Open Grid Services Infrastructure
- Implementations in multiple languages and protocols
- Higher-level Services
- Application services and infrastructure
- Structure of the proposal
- Participants: LBNL and ANL
- 3 year project
- Started last May
***PI: Keith Jackson***
***Josh Boverhof, Monte Goode, Keith Beattie, Wolfgang Hosheck***

## Emphasis

- Addressing the needs of scientific computing community
- Language bindings and infrastructure implementation
- Python
- C/C++
- Performance aspects
- Investigation of fast communication protocols
- Encodings
- Overall performance
- Application-level services and software

## CoG Kits

- Commodity Grid toolkits are being developed in Java and Python to:
- Use and leverage existing technologies for Grid programming
- Provide access to Grid Services from high-level component-based frameworks
- Enable the routine construction of Problem Solving Environments, Science Portals, and Grid Applications
- Reduce development and maintenance costs
- Joint project with Argonne
*PI: Keith R. Jackson*
*Matt Rodriguez, Joshua Boverhof*

## Python CoG Kit

- Wraps the Globus ToolkitTM
- Use and leverage existing Python technologies for Grid programming
- Support for high-level constructs, e.g. classes, modules, packages, components
- XML support
- Web Services support
- SOAP
- WSDL
- UDDI
- Platform independent GUI’s, open source IDE’s
- Maximize software flexibility, extensibility, and reusability by:
-  Developing reusable components that access Grid services
- Leveraging a familiar development environment

- Globus Project and Globus Toolkit are trademarks held by the University of Chicago  of Chicago

## Python CoG Kit Layering

## Accomplishments & Interactions

- Released versions of the SOAP protocol, for both Java and Python, that use GSI security
- Developed the Reliable File Transfer Service using the Python CoG Kit
- The Community Authorization Server from ANL is built using the Python CoG Kit
- The Access Grid controllers are developed using the Python CoG Kit.
- Win32 support was added
- The Atlas project is using the Python CoG Kit for accessing Grid resources

## Akenti Distributed Authorization Service

- Targeted at distributed environments
- Access based on policy statements made by stakeholders
- Handle multiple independent stakeholders for a single resource
- Use Public Key Infrastructure standards to identify users and create digitally signed certificates
- Ease of use
- GUI interfaces to generate certificates
- Simple API for PolicyEnforcementPoints to call
- Interoperate with Web and Grid authentication mechanisms.

## Flexible, rich policy 

- Supports a relational policy language using
- components of Distinguished Name
- stakeholder defined attributes - group, role
- runtime constraints - time of day, load factor
- Supports delegated certificates
- will support restricted delegation
***PI: Mary Thompson***
***	******Abdelilah Essiari, Keith Beattie******	***

## Cryptographic Aspects of the Grid Security Architecture Concept

- Grid-enabled applications have security requirements
- authentication and authorization of individuals accessing the resources
- protection from hackers and viruses, privacy of data
- Security is a critical component of the grid middleware
- current Grid Security Architecture (GSI) is inadequate
- The Grid middleware must address the security needs
- an architecture enabling secure communications between the Grid-enabled application components spread across the Internet
- an architecture supporting the security services of access control, confidentiality, authenticity, and integrity (e.g, SSL protocol)
- an architecture with a strong cryptographic foundation to accurately define the level of security achieved
- a security architecture which is deployable and inter-operable

## Tasks/Milestones

- Study the cryptographic requirements of the Grid Security and Open Grid Services Architectures (GSI, OGSA)
- Provide a cryptographic analysis of the Grid Security and Open Grid Services Architectures (GSI, OGSA)
- Implement our results within the Globus software toolkit
- 	***PI: Olivier Chevassut (LBNL)  ***
***   ******D. Agarwal, K. Jackson (LBNL),  F. ******Siebenlist******, V. Welch (ANL)***

## NetLogger

- Methodology for Monitoring and Analysis of   Distributed Systems
- Enables the real-time diagnosis of performance problems
- Includes tools for generating precision event logs
- Allows detailed end-to-end application and system level monitoring
- Includes tools for visualizing log data to view the state of the distributed system in real time
- Combines network, host, and application-level monitoring
*Contact:  Brian L. Tierney (BLTierney@lbl.gov)*

## Net100: Developing Network-aware Operating Systems

- Principal investigators
- Matt Mathis, PSC (*mathis@psc.edu*)
- Brian Tierney, LBNL (*bltierney@lbl.gov*)
- Tom Dunigan, ORNL (*thd@ornl.gov*)
- Objective:
- measure  and understand end-to-end network and application performance
- autotune network applications (Data Grid / bulk data transfer)
- Build upon web100 project
- Project Components
- active network probes and passive OS sensors
- Web100 verification
- network measurements archive
- tuning daemon (WAD) to tune network flows based on network measurements
- analyze various proposed protocol enhancements

## Network Tool Analysis Framework (NTAF)

## Major Accomplishments

- Designed and implemented NTAF servers
- Integrated NTAF with NetLogger archive
- Web interface to the archive for interactive analysis
- Instrumented many tools with NetLogger
- Designed and Implemented the Work Around Daemon (WAD)
- Auto-tuning
- Auto-monitoring
- HS-TCP testing
- Found Linux TCP Bugs, learned new tuning tricks (e.g.: txqueuelen)
- Used NTAF to test new tools (netest, ncs, etc.)
- .

## NetLogger + Web100

**NetLogger + Web100**

- Oakland to SC02

## Distributed Monitoring Framework

- When building distributed systems, we often observe unexpectedly low performance
- The reasons for which are usually not obvious
- Bottlenecks can be in any of the following components:
- Software: applications and operating systems
- the disks, network adapters, bus, memory, etc. on either the sending or receiving host
- the network switches and routers, and so on
- There is a fault (error, crash, etc) in some grid middleware component.
- How does Grid job owner or the Grid sysadmin determine which component caused the job to fail?
- How does the developer of the failed component find out what exactly what was happening when the fault occurred?
- This includes knowing the state of multiple distributed components
- Solution must be useful to Grid Users, Grid Administrators, and Grid software developers

## Solution

- A complete **End-to-End** monitoring framework that includes the following components:
- instrumentation tools (application, middleware, and OS monitoring)
- host and network sensors  (host and network monitoring)
- sensor management tools  (sensor control system)
- monitoring event publication service
- monitoring event archive service
- event analysis and visualization tools
- a common set of protocols for describing, exchanging and locating monitoring data

## More uses for Monitoring Data

- Monitoring Data not just for end-to-end performance analysis and troubleshooting
- Lots of “middleware services” need Monitoring data too:
- Grid Schedulers
**find the best match of CPUs and data sets for a given job**
- Grid Replica Selection
**find the “best” copy of a data set to use**
- Reliable File copy service
**detect failures and recover**
- Network-aware applications
**TCP buffer size tuning, number of parallel streams, etc.**

## DMF Goals

- Provide a unifying view to a wide range of sensor data, from network to host to application
- DMF is based on Web services
- XML, SOAP, WSDL
- Define common protocols and data formats
- we are leading a GGF effort in this area
- Work with others to integrate existing components using this framework
- e.g.: PingER, NWS, MDS
- develop missing pieces
- e.g.: event archives

## Grid Monitoring Architecture (GMA): Terminology and Architecture

- (Performance) Event:
- Typed collection of data with a specific structure
- Producer Interface:
- makes performance data (events) available
- Consumer Interface:
- receives performance data (events)
- Directory Service:
- supports information publication and discovery
- must be distributed and/or replicated

## Major Accomplishments

- NetLogger Improvements
- Binary format
- Reliability
- Trigger mechanism
- Netarchd: mySQL-based archive for NetLogger data
- pyGMA
- Released v1.0
- Activation Service
- SC02 Demo with Atlas software
- Standardization efforts
- PPDG
- Grid Laboratory Universal Environment (GLUE) Schema
- Global Grid Forum (GGF) work
- Discovery and Monitoring Event Descriptions WG
- Network Measurement WG

## Bandwidth Estimation

- Developing a formula to accurately measure available bandwidth and capacity
- Need to implement a tool for hop-by-hop measurement
***PI: Jin Guojan***

## The InterGroup Protocol

- Scalable group (many-to-many) communication
- Based on experience building previous reliable multicast protocols
- Resolve inherent conflict between strict delivery guarantees and scale of the group
- Scale by splitting group into a sender and receiver group
- Flexible receiver-based delivery guarantee decision
- Receivers only pay the delay of the level of ordering and reliability they require
***Deb Agarwal ,Karlo Berket***

## InterGroup Schematic

## Secure Reliable Group Communication

- Provide an integrated prototype of a general-purpose secure, reliable group communication system that scales to the Internet
- Define a secure group management layer and reliable multicast protocol :
- Reliable multicast protocol
- Group key agreement
- Access control
- Support multi-sender groups that scale to the Internet while providing strong delivery guarantees
- Enforce admission policies for dynamic peer groups

## Concept

- Develop, implement, and deploy the InterGroup protocol which is a reliable multicast protocol intended to scale to large group sizes and to the Internet
- Develop, implement, and deploy a Secure Group Layer that provides a secure channel abstraction similar to Secure Sockets Layer (SSL), but for multicast

## Tasks/Milestones

- Deploy beta release of InterGroup (Java and C++ interfaces)
- Design and implementation of further InterGroup features (e.g. send proxy, flow control, additional ordering and reliability level, etc)
- Design and implementation of the Secure Group Layer
- Analysis and testing of design and implementation
- Publication of results and theory
- Support of early adopters
***PI: Deb Agawal***
***Karlo Berket, Olivier Chevassut***

## Develop a peer-to-peer system to support secure, location independent information sharing in the scientific community
Group communication mechanisms for scalable on-demand search
Security mechanisms and policies based on Secure Group Layer and Akenti

- Group communication mechanisms for scalable on-demand search
- Security mechanisms and policies based on Secure Group Layer and Akenti

**Peer-to-peer Information Sharing**

## Tasks/Milestones (’03-’04)

- Design and implement resource discovery messaging framework and information sharing application
- Design and implement security mechanisms and policies
- Deploy system (initial prototype Q2 ’03)
- Establish initial user base
- Publication of ideas and design
***PI: Karlo Berket,***
***Guillaume Egles, Abdelilah Essiari***

## Collaboratory tools

- Early work focused on access to scientific instruments
- remote instrument read-out and control
- videoconferencing
- telepresence
- Highly interactive tools are needed only a small percentage of the time
- Videoconferencing support has a high cost
- Need tools that provide a continuum of interaction

## Pervasive Collaborative Computing Environment

- Provide tools that support the day-to-day collaboration needs of a group
- Support the collaboration continuum
- Provide tools that directly support work processes
- Web-based interface
- Leverage off of the Grid services
- Leverage off of existing standards and tools
- Work with collaboratory users

## Secure Chat tool

## Tasks/Milestones

- Deploy a release of the IRC and group communication-based secure messaging service and base PCCE framework
- Design and Implement a collaborative workflow definition, submission, and tracking tool
- Integration of Grid services (security and directory initially)
- Integration of an electronic notebook
- Development of prototype incremental trust implementation
***PI: Deb Agarwal***
***	******Charles McParland and Marcia Perry***