# Presentation: 589187

## Monitoring and Discovery in a Web Services Framework:           Functionality and Performance of Globus Toolkit MDS4

- Jennifer M. Schopf
- Argonne National Laboratory
- UK National eScience Centre (NeSC)
- Sept 11, 2006

## What is a Grid

- Resource sharing
- Computers, storage, sensors, networks, ...
- Sharing always conditional: issues of trust, policy, negotiation, payment, ...
- Coordinated problem solving
- Beyond client-server: distributed data analysis, computation, collaboration, ...
- Dynamic, multi-institutional virtual orgs
- Community overlays on classic org structures
- Large or small, static or dynamic

## Why is this hard/different?

- Lack of central control
- Where things run
- When they run
- Shared resources
- Contention, variability
- Communication
- Different sites implies different sys admins, users, institutional goals, and often “strong personalities”

## So why do it?

- Computations that need to be done with a time limit
- Data that can’t fit on one site
- Data owned by multiple sites
- Applications that need to be run bigger, faster, more

## What Is Grid Monitoring?

- Sharing of community data between sites using a standard interface for querying and notification
- Data of interest to more than one site
- Data of interest to more than one person
- Summary data is possible to help scalability
- Must deal with failures
- Both of information sources and servers
- Data likely to be inaccurate
- Generally needs to be acceptable for data to be dated

## Common Use Cases

- Decide what resource to submit a job to, or to transfer a file from
- Keep track of services and be warned of failures
- Run common actions to track performance behavior
- Validate sites meet a (configuration) guideline

## What is MDS4?

- Grid-level monitoring system used most often for resource selection and error notification
- Aid user/agent to identify host(s) on which to run an application
- Make sure that they are up and running correctly
- Uses standard interfaces to provide publishing of data, discovery, and data access, including subscription/notification
- WS-ResourceProperties, WS-BaseNotification, WS-ServiceGroup
- Functions as an hourglass to provide a common interface to lower-level monitoring tools

## Standard Schemas

- (GLUE schema, eg)

- Information Users :
- Schedulers, Portals, Warning Systems, etc.

- WS standard interfaces for subscription, registration, notification

## Web ServiceResource Framework (WS-RF)

- Defines standard interfaces and behaviors for distributed system integration, especially (for us):
- Standard XML-based service information model
- Standard interfaces for push and pull mode access to service data
- Notification and subscription

## MDS4 UsesWeb Service Standards

- WS-ResourceProperties
- Defines a mechanism by which Web Services can describe and publish resource properties, or sets of information about a resource
- Resource property types defined in service’s WSDL
- Resource properties can be retrieved using WS-ResourceProperties query operations
- WS-BaseNotification
- Defines a subscription/notification interface for accessing resource property information
- WS-ServiceGroup
- Defines a mechanism for grouping related resources and/or services together as service groups

## WSRF vs XML/SOAP

- The definition of WSRF means that the Grid and Web services communities can move forward on a common base
- Why Not Just Use XML/SOAP?
- WSRF and WS-N *are *just XML and SOAP
- WSRF and WS-N are just Web services
- Benefits of following the specs:
- These patterns represent best practices that have been learned in many Grid applications
- There is a community behind them
- Why reinvent the wheel?
- Standards facilitate interoperability

## MDS4 Components

- Information providers
- Monitoring is a part of every WSRF service
- Non-WS services are also be used
- Higher level services
- Index Service – a way to aggregate data
- Trigger Service – a way to be notified of changes
- Both built on common aggregator framework
- Clients
- WebMDS
- All of the tool are schema-agnostic, but interoperability needs a well-understood common language

## Information Providers

- Data sources for the higher-level services
- Some are built into services
- Any WSRF-compliant service publishes some data automatically
- WS-RF gives us standard Query/Subscribe/Notify interfaces
- GT4 services: ServiceMetaDataInfo element includes start time, version, and service type name
- Most of them also publish additional useful information as resource properties

## Information Providers:GT4 Services

- Information Providers:GT4 Services

- Reliable File Transfer Service (RFT)
- Service status data, number of active transfers, transfer status, information about the resource running the service
- Community Authorization Service (CAS)
- Identifies the VO served by the service instance
- Replica Location Service (RLS)
- Note: not a WS
- Location of replicas on physical storage systems (based on user registrations) for later queries

## Information Providers (2)

- Other sources of data
- Any executables
- Other (non-WS) services
- Interface to another archive or data store
- File scraping
- Just need to produce a valid XML document

## Information Providers:Cluster and Queue Data

- Information Providers:Cluster and Queue Data

- Interfaces to Hawkeye, Ganglia, CluMon, Nagios
- Basic host data (name, ID), processor information, memory size, OS name and version, file system data, processor load data
- Some condor/cluster specific data
- This can also be done for sub-clusters, not just at the host level
- Interfaces to PBS, Torque, LSF
- Queue information, number of CPUs available and free, job count information, some memory statistics and host info for head node of cluster

## Other Information Providers

- File Scraping
- Mostly used for data you can’t find programmatically
- System downtime, contact info for sys admins, online help web pages, etc.
- Others as contributed by the community!

## Higher-Level Services

- Index Service
- Caching registry
- Trigger Service
- Warn on error conditions
- All of these have common needs, and are built on a common framework

## MDS4 Index Service

- Index Service is both registry and cache
- Datatype and data provider info, like a registry (UDDI)
- Last value of data, like a cache
- Subscribes to information providers
- In memory default approach
- DB backing store currently being discussed to allow for very large indexes
- Can be set up for a site or set of sites, a specific set of project data, or for user-specific data only
- Can be a multi-rooted hierarchy
- No *global* index

## MDS4 Trigger Service

- Subscribe to a set of resource properties
- Evaluate that data against a set of pre-configured conditions (triggers)
- When a condition matches, action occurs
- Email is sent to pre-defined address
- Website updated

## Common Aspects

- 1) Collect information from information providers
- Java class that implements an interface to collect XML-formatted data
*“**Query”* uses WS-ResourceProperty mechanisms to poll a WSRF service
*“**Subscription” *uses WS-Notification subscription/notification
*“**Execution” *executes an administrator-supplied program to collect information
- 2) Common interfaces to external services
- These should all have the standard WS-RF service interfaces

## Common Aspects (2)

- 3) Common configuration mechanism
- Maintain information about which information providers to use and their associated parameters
- Specify what data to get, and from where
- 4) Services are self-cleaning
- Each registration has a lifetime
- If a registration expires without being refreshed, it and its associated data are removed from the server
- 5) Soft consistency model
- Flexible update rates from different IPs
- Published information is recent, but not guaranteed to be the absolute latest
- Load caused by information updates is reduced at the expense of having slightly older information
- Free disk space on a system 5 minutes ago rather than 2 seconds ago

## Aggregator Framework

## Aggregator Frameworkis a General Service

- This can be used for other higher-level services that want to
- Subscribe to Information Provider
- Do some action
- Present standard interfaces
- Archive Service
- Subscribe to data, put it in a database, query to retrieve, currently in discussion for development
- Prediction Service
- Subscribe to data, run a predictor on it, publish results
- Compliance Service
- Subscribe to data, verify a software stack match to definition, publish yes or no

## WebMDS User Interface

- Web-based interface to WSRF resource property information
- User-friendly front-end to Index Service
- Uses standard resource property requests to query resource property data
- XSLT transforms to format and display them
- Customized pages are simply done by using HTML form options and creating your own XSLT transforms
- Sample page:
- http://mds.globus.org:8080/webmds/webmds?info=indexinfo&xsl=servicegroupxsl

## WebMDS Service

## Site 3

- WebMDS

- App B

- Index

- App B

- Index

- Index

- Index

- Rsc

- 3.a

- RLS

- Rsc

- 3.b

- RLS

- I

- I

- Rsc

- 3.b

- Site 1

- West Coast

- Index

- West Coast

- Index

- Trigger

- Service

- Rsc

- 2.a

- Hawkeye

- Hawkeye

- Rsc

- 2.b

- GRAM

- GRAM

- I

- I

- Site 2

- Index

- Site 2

- Index

- Rsc

- 1.c

- Ganglia/LSF

- Ganglia/LSF

- Rsc

- 1.c

- GRAM

- (LSF)

- GRAM

- (LSF)

- I

- I

- Rsc

- 1.a

- Rsc

- 1.b

- Ganglia/PBS

- Ganglia/PBS

- Rsc

- 1.b

- GRAM

- (PBS)

- GRAM

- (PBS)

- I

- I

- Site 1

- Index

- Site 1

- Index

- RFT

- RFT

- Rsc

- 1.d

- I

- I

**A**

**A**

**B**

**B**

**C**

**C**

**D**

**D**

**E**

**E**

- VO Index

**F**

**F**

- Trigger action

## Index

- Site 1

- Rsc

- 1.c

- Ganglia/LSF

- Ganglia/LSF

- Rsc

- 1.c

- GRAM

- (LSF)

- GRAM

- (LSF)

- I

- I

- Rsc

- 1.a

- Rsc

- 1.b

- Ganglia/PBS

- Ganglia/PBS

- Rsc

- 1.b

- GRAM

- (PBS)

- GRAM

- (PBS)

- I

- I

- Site 1

- Site 1

- RFT

- RFT

- Rsc

- 1.d

- I

- I

**A**

**A**

- Container

- Service

- Registration

## Slide 33

## Slide 34

## Site 3

- WebMDS

- App B

- Index

- App B

- Index

- Index

- Index

- Rsc

- 3.a

- RLS

- Rsc

- 3.b

- RLS

- I

- I

- Rsc

- 3.b

- Site 1

- West Coast

- Index

- West Coast

- Index

- Trigger

- Service

- Rsc

- 2.a

- Hawkeye

- Hawkeye

- Rsc

- 2.b

- GRAM

- GRAM

- I

- I

- Site 2

- Index

- Site 2

- Index

- Rsc

- 1.c

- Ganglia/LSF

- Ganglia/LSF

- Rsc

- 1.c

- GRAM

- (LSF)

- GRAM

- (LSF)

- I

- I

- Rsc

- 1.a

- Rsc

- 1.b

- Ganglia/PBS

- Ganglia/PBS

- Rsc

- 1.b

- GRAM

- (PBS)

- GRAM

- (PBS)

- I

- I

- Site 1

- Index

- Site 1

- Index

- RFT

- RFT

- Rsc

- 1.d

- I

- I

**A**

**A**

**B**

**B**

**C**

**C**

**D**

**D**

**E**

**E**

- VO Index

**F**

**F**

- Trigger action

## Any questions before I walk through two current deployments?

- Grid Monitoring and Use Cases
- MDS4
- Information Providers
- Higher-level services
- WebMDS
- Deployments
- Metascheduling Data for TeraGrid
- Service Failure warning for ESG
- Performance Numbers
- MDS for You!

## Working with TeraGrid

- Large US project across 9 different sites
- Different hardware, queuing systems and lower level monitoring packages
- Starting to explore MetaScheduling approaches
- Currently evaluating almost 20 approaches
- Need a common source of data with a standard interface for basic scheduling info

## Cluster Data

- Provide data at the subcluster level
- Sys admin defines a subcluster, we query one node of it to dynamically retrieve relevant data
- Can also list per-host details
- Interfaces to Ganglia, Hawkeye, CluMon, and Nagios available now
- Other cluster monitoring systems can write into a .html file that we then scrape

## Cluster Info

- UniqueID
- Benchmark/Clock speed
- Processor
- MainMemory
- OperatingSystem
- Architecture

- Number of nodes in a cluster/subcluster
- StorageDevice
- Disk names, mount point, space available
- TG specific Node properties

## Data to collect: Queue info

- LRMSType
- LRMSVersion
- DefaultGRAMVersion and port and host
- TotalCPUs
- Status (up/down)
- TotalJobs (in the queue)

- RunningJobs
- WaitingJobs
- FreeCPUs
- MaxWallClockTime
- MaxCPUTime
- MaxTotalJobs
- MaxRunningJobs

- Interface to PBS (Pro, Open, Torque), LSF

## How will the data be accessed?

- Java and command line APIs to a common TG-wide Index server
- Alternatively each site can be queried directly
- One common web page for TG
- http://mds.teragrid.org
- Query page is next!

## Status

- Demo system running since Autumn ‘05
- Queuing data from SDSC and NCSA
- Cluster data using CluMon interface
- All sites in process of deployment
- Queue data from 7 sites reporting in
- Cluster data still coming online

## Earth Systems Grid Deployment

- Supports the next generation of climate modeling research
- Provides the infrastructure and services that allow climate scientists to publish and access key data sets generated from climate simulation models
- Datasets including  simulations generated using the Community Climate System Model (CCSM) and the Parallel Climate Model (PCM
- Accessed by scientists throughout the world.

## Who uses ESG?

- In 2005
- ESG web portal issued 37,285 requests to download 10.25 terabytes of data
- By the fourth quarter of 2005
- Approximately two terabytes of data downloaded per month
- 1881 registered users in 2005
- Currently adding users at a rate of more than 150 per month

## What are the ESG resources?

- Resources at seven sites
- Argonne National Laboratory (ANL)
- Lawrence Berkeley National Laboratory (LBNL)
- Lawrence Livermore National Laboratory (LLNL)
- Los Alamos National Laboratory (LANL)
- National Center for Atmospheric Research (NCAR)
- Oak Ridge National Laboratory (ORNL)
- USC Information Sciences Institute (ISI)
- Resources include
- Web portal
- HTTP data servers
- Hierarchical mass storage systems
- OPeNDAP system
- Storage Resource Manager (SRM)
- GridFTP data transfer service [
- Metadata and replica management catalogs

## Slide 47

## The Problem:

- Users are 24/7
- Administrative support was not!
- Any failure of ESG components or services can severely disrupt the work of many scientists
- The Solution
- Detect failures quickly and minimize infrastructure downtime by deploying MDS4 for error notification

## ESG Services Being Monitored

| Service Being Monitored | ESG Location |
| --- | --- |
| GridFTP server | NCAR |
| OPeNDAP server | NCAR |
| Web Portal | NCAR |
| HTTP Dataserver | LANL, NCAR |
| Replica Location Service (RLS) servers | LANL , LBNL, LLNL, NCAR, ORNL |
| Storage Resource Managers | LANL, LBNL,  NCAR, ORNL |
| Hierarchical Mass Storage Systems | LBNL, NCAR, ORNL |

## Index Service

- Site-wide index service is queried by the ESG web portal
- Generate an overall picture of the state of ESG resources displayed on the Web

## Slide 51

## Trigger Service

- Site-wide trigger service collects data and sends email upon errors
- Information providers are polled at pre-defined services
- Value must be matched for set number of intervals for trigger to occur to avoid false positives
- Trigger has a delay associated for vacillating values
- Used for offline debugging as well

## 1 Month of Error Messages

| Total error messages for May 2006 | 47 |
| --- | --- |
| Messages related to certificate and configuration problems at LANL | 38 |
| Failure messages due to brief interruption in network service at ORNL on 5/13 | 2 |
| HTTP data server failure at NCAR 5/17 | 1 |
| RLS failure at LLNL 5/22 | 1 |
| Simultaneous error messages for SRM services at NCAR, ORNL, LBNL on 5/23 | 3 |
| RLS failure at ORNL 5/24 | 1 |
| RLS failure at LBNL 5/31 | 1 |

## 1 Month of Error Messages

| Total error messages for May 2006 | 47 |
| --- | --- |
| Messages related to certificate and configuration problems at LANL | 38 |
| Failure messages due to brief interruption in network service at ORNL on 5/13 | 2 |
| HTTP data server failure at NCAR 5/17 | 1 |
| RLS failure at LLNL 5/22 | 1 |
| Simultaneous error messages for SRM services at NCAR, ORNL, LBNL on 5/23 | 3 |
| RLS failure at ORNL 5/24 | 1 |
| RLS failure at LBNL 5/31 | 1 |

- For the month of May 2006, ESG’s deployment of MDS4 generated 47 failure messages that were sent to ESG system administrators. These are summarized in Table 3. The majority of these failure messages were caused by downtime throughout the month of services at LANL due to certificate expiration and service configuration problems. During this period, LANL staff members were not available to address these issues. Additional LANL failure messages would have been generated, except that we disabled the triggers from LANL during a two-week period when the responsible staff person was on vacation.
- The remaining error messages indicate short-term service failures. Two failure messages were generated due to a network outage at ORNL on May 13th. Three error messages for Storage Resource Managers (SRMs) at different sites were generated on May 23rd. Since it is unlikely that all three of these services failed simultaneously, these messages were more likely due to a problem with the network, the monitoring services, or the client that checks the status of SRM services.

## Benefits

- Overview of current system state for users and system administrators
- At a glance info on resources and services availability
- Uniform interface to monitoring data
- Failure notification
- System admins can identify and quickly address failed components and services
- Before this deployment, services would fail and might not be detected until a user tried to access an ESG dataset
- Validation of new deployments
- Verify the correctness of the service configurations and deployment with the common trigger tests
- Failure deduction
- A failure examined in isolation may not accurately reflect the state of the system or the actual cause of a failure
- System-wide monitoring data can show a pattern of failure messages that occur close together in time can be used to deduce a problem at a different level of the system
- Eg. 3 SRM failures
- EG. Use of MDS4 to evaluate file descriptor leak

## OUTLINE

- Grid Monitoring and Use Cases
- MDS4
- Index Service
- Trigger Service
- Information Providers
- Deployments
- Metascheduling Data for TeraGrid
- Service Failure warning for ESG
- Performance Numbers
- MDS for You!

## Scalability Experiments

- MDS index
- Dual 2.4GHz Xeon processors, 3.5 GB RAM
- Sizes: 1, 10, 25, 50, 100
- Clients
- 20 nodes also dual 2.6 GHz Xeon, 3.5 GB RAM
- 1, 2, 3, 4, 5, 6, 7, 8, 16, 32, 64, 128, 256, 384, 512, 640, 768, 800
- Nodes connected via 1Gb/s network
- Each data point is average of 8 minutes
- Ran for 10 mins but first 2 spent getting clients up and running
- Error bars are SD over 8 mins
- Experiments by Ioan Raicu, U of Chicago, using DiPerf

## Size Comparison

-  In our current TeraGrid demo
- 17 attributes from 10 queues at SDSC and NCSA
- Host data - 3 attributes for approx 900 nodes
- 12 attributes of sub-cluster data for 7 subclusters
- ~3,000 attributes, ~1900 XML elements, ~192KB.
- Tests here- 50 sample entries
- element count of 1113
- ~94KB in size

## Slide 60

## Slide 61

## MDS4 Stability

| Vers. | Index
Size | Time up
(Days) | Queries
Processed | Query
Per
Sec. | Round-trip
Time (ms) |
| --- | --- | --- | --- | --- | --- |
| 4.0.1 | 25 | 66+ | 81,701,925 | 14 | 69 |
| 4.0.1 | 50 | 66+ | 49,306,104 | 8 | 115 |
| 4.0.1 | 100 | 33 | 14,686,638 | 5 | 194 |
| 4.0.0 | 1 | 14 | 93,890,248 | 76 | 13 |
| 4.0.0 | 1 | 96 | 623,395,877 | 74 | 13 |

## Index Maximum Size

| Heap
Size (MB) | Approx. Max.
Index Entries | Index
Size (MB) |
| --- | --- | --- |
| 64 | 600 | 1.0 |
| 128 | 1275 | 2.2 |
| 256 | 2650 | 4.5 |
| 512 | 5400 | 9.1 |
| 1024 | 10800 | 17.7 |
| 1536 | 16200 | 26.18 |

**Limited by Java Heap Size**

## Performance

- Is this enough?
- We don’t know!
- Currently gathering up usage statistics to find out what people need
- Bottleneck examination
- In the process of doing in depth performance analysis of what happens during a query
- MDS code, implementation of WS-N, WS-RP, etc

## MDS For You

- Grid Monitoring and Use Cases
- MDS4
- Information Providers
- Higher-level services
- WebMDS
- Deployments
- Metascheduling Data for TeraGrid
- Service Failure warning for ESG
- Performance Numbers
- MDS for You!

## How Should You Deploy MDS4?

- Ask: Do you need a Grid monitoring system?
- Sharing of community data between sites using a standard interface for querying and notification
- Data of interest to more than one site
- Data of interest to more than one person
- Summary data is possible to help scalability

## What does your projectmean by monitoring?

- What does your projectmean by monitoring?

- Display site data to make resource selection decisions
- Job tracking
- Error notification
- Site validation
- Utilization statistics
- Accounting data

## What does your projectmean by monitoring?

- What does your projectmean by monitoring?

- Display site data to make resource selection decisions
- Job tracking
- Error notification
- Site validation
- Utilization statistics
- Accounting data

**MDS4 a Good Choice!**

## What does your projectmean by monitoring?

- What does your projectmean by monitoring?

- Display site data to make resource selection decisions
- Job tracking – generally application specific
- Error notification
- Site validation
- Utilization statistics – use local info
- Accounting data- use local info and reliable messaging – AMIE from TG is one option

**Think about**
**other tools**

## What data do you need

- There is no generally agreed upon list of data every site should collect
- Two possible examples
- What TG is deploying
- http://mds.teragrid.org/docs/mds4-TG-overview.pdf
- What GIN-Info is collecting
- _[http://forge.gridforum.org/sf/wiki/do/viewPage/projects.gin/wiki/GINInfoWiki](http://forge.gridforum.org/sf/wiki/do/viewPage/projects.gin/wiki/GINInfoWiki)_
- Make sure the data you want is actually theoretically possible to collect!
- Worry about the schema later

## Building your own info providers

- See the developer session!
- Some pointers...
- List of new providers
- http://www.globus.org/toolkit/docs/development/4.2-drafts/info/providers/index.html
- How to write info providers:
- http://www.globus.org/toolkit/docs/4.0/info/usefulrp/rpprovider-overview.html
- http://www-unix.mcs.anl.gov/~neillm/mds/rp-provider-documentation.html
- http://globus.org/toolkit/docs/4.0/info/index/WS_MDS_Index_HOWTO_Execution_Aggregator.html

## How many Index Servers?

- Generally one at each site, one for full project
- Can be cross referenced and duplicated
- Can also set them up for an application group or any subset

## What Triggers?

- What are your critical services?

## What Interfaces?

- Command line, Java, C, and Python come for free
- WebMDS give you the simple one out of the box
- Can stylize- like TG and ESG did – very straight forward

## What will you be able to do?

- Decide what resource to submit a job to, or to transfer a file from
- Keep track of services and be warned of failures
- Run common actions to track performance behavior
- Validate sites meet a (configuration) guideline

## Summary

- MDS4 is a WS-based Grid monitoring system that uses current standards for interfaces and mechanisms
- Available as part of the GT4 release
- Currently in use for resource selection and fault notification
- Initial performance results aren’t awful – we need to do more work to determine bottlenecks

## Where do we go next?

- Extend MDS4 information providers
- More data from GT4 services
- Interface to other data sources
- Inca, GRASP, PinGER Archive, NetLogger
- Additional deployments
- Additional scalability testing and development
- Database backend to Index service to allow for very large indexes
- Performance improvements to queries – partial result return

## Other Possible HigherLevel Services

- Other Possible HigherLevel Services

- Archiving service
- The next high level service we’ll build
- Currently a design document internally, should be made external shortly
- Site Validation Service (ala Inca)
- Prediction service (ala NWS)
- What else do you think we need? Contribute to the roadmap!
- http://bugzilla.globus.org

## Other Ways To Contribute

- Join the mailing lists and offer your thoughts!
- mds-dev@globus.org
- mds-user@globus.org
- mds-announce@globus.org
- Offer to contribute your information providers, higher level service, or visualization system
- If you’ve got a complementary monitoring system – think about being an Incubator project (contact incubator-commiters@globus.org, or come to the talk on Thursday)

## Thanks

- MDS4 Core Team: Mike D’Arcy (ISI), Laura Pearlman (ISI), Neill Miller (UC), Jennifer Schopf (ANL)
- MDS4 Additional Development help: Eric Blau, John Bresnahan, Mike Link, Ioan Raicu, Xuehai Zhang
- This work was supported in part by the Mathematical, Information, and Computational Sciences Division subprogram of the Office of Advanced Scientific Computing Research, U.S. Department of Energy, under contract W-31-109-Eng-38, and NSF NMI Award SCI-0438372. ESG work was supported by U.S. Depart­ment of Energy under the Scientific Discovery Through Advanced Computation (SciDAC) Program Grant DE-FC02-01ER25453. This work also supported by DOESG SciDAC Grant, iVDGL from NSF, and others.

## Slide 81

## For More Information

- Jennifer Schopf
- jms@mcs.anl.gov
- http://www.mcs.anl.gov/~jms
- Globus Toolkit MDS4
- http://www.globus.org/toolkit/mds
- MDS-related events at GlobusWorld
- MDS for Developers
- Monday 4:00-5:30, 149 A/B
- MDS “Meet the Developers” session
- Tuesday 12:30-1:30, Globus Booth