# Presentation: 125880

## MDS4 – The Globus Toolkit’s Monitoring and Discovery System

- Jennifer M. Schopf
- Argonne National Laboratory
- NeSC
- November, 2005

## What Is Grid Monitoring?

- A way to discover what services and resources are available to use
- A way to understand the status/attributes of those services
- A system to warn you when things fail
- Sharing of community data between sites using a standard interface for querying and notification

## Why Grid Monitoring Hard?

- Lack of central control
- Different local systems according to local policy
- Different interfaces and monitoring requirements
- Shared resources
- Contention, variability
- Communication
- Different sites implies different sys admins, users, institutional goals

## MDS4Monitoring and Discovery System

- Grid-level monitoring system used most often for resource selection
- Aid user/agent to identify host(s) on which to run an application
- Uses standard interfaces to provide publishing of data, discovery, and data access, including subscription/notification
- WS-ResourceProperties, WS-BaseNotification, WS-ServiceGroup
- Functions as an hourglass to provide a common interface to lower-level monitoring tools

## GLUE Schema Attributes

- (cluster info,
- queue info, FS info)

- Information Users :
- Schedulers, Portals, etc.

- WS standard interfaces for subscription, registration, notification

## MDS4 Components

- Higher level services
- Index Service – a way to aggregate data
- Trigger Service – a way to be notified of changes
- Both built on common aggregator framework
- Information providers
- Monitoring is a part of every WSRF service
- Non-WS services can also be used
- Clients
- WebMDS
- All of the tool are schema-agnostic, but interoperability needs a well-understood common language

## MDS4 Index Service

- Index Service is both registry and cache
- Subscribes to information providers
- Data, datatype, data provider information
- Caches last value of all data
- In memory default approach
- Soft-state registration
- Can be set up for a site or set of sites, a specific set of project data, or for user-specific data only

## MDS4 Trigger Service

- Subscribe to a set of resource properties
- Evaluate that data against a set of pre-configured conditions (triggers)
- When a condition matches, email is sent to pre-defined address
- Similar functionality in Hawkeye
- Currently in use by ESG

## Information Providers

- Data sources for the higher level services (eg. Index, Trigger)
- WSRF-compliant service
- WS-ResourceProperty for Query source
- WS-Notification mechanism for Subscription source
- Other services/data sources
- Executable program that obtains data via some domain-specific mechanism for Execution source.

## Information Providers:Cluster and Queue Data

- Information Providers:Cluster and Queue Data

- Interfaces to Hawkeye, Ganglia, CluMon (and Nagios Soon!)
- Basic host data (name, ID), processor information, memory size, OS name and version, file system data, processor load data
- Some condor/cluster specific data
- Interfaces to PBS, Torque, and LSF queue systems
- Queue information, number of CPUs available and free, job count information, some memory statistics and host info for head node of cluster

## Information Providers:GT4 Services

- Information Providers:GT4 Services

- Every WS built using GT4 core
- ServiceMetaDataInfo element includes start time, version, and service type name
- Reliable File Transfer Service (RFT)
- Service status data, number of active transfers, transfer status, information about the resource running the service
- Community Authorization Service (CAS)
- Identifies the VO served by the service instance
- Replica Location Service (RLS)
- Note: not a WS
- Location of replicas on physical storage systems (based on user registrations) for later queries

## Sample Deployment

## WebMDS User Interface

- Web-based interface to WSRF resource property information
- User-friendly front-end to the Index Service
- Uses standard resource property requests to query resource property data
- XSLT transforms to format and display them
- Customized pages are simply done by using HTML form options and creating your own XSLT transforms
- Sample page:
- http://mds.globus.org:8080/webmds/webmds?info=indexinfo&xsl=servicegroupxsl

## WebMDS Service

## Working with TeraGrid

- Large US project across 9 different sites
- Different hardware, queuing systems and lower level monitoring packages
- Starting to explore MetaScheduling approaches
- GRMS (Poznan)
- W. Smith (TACC)
- K. Yashimoto (SDSC)
- User Portal
- Need a common source of data with a standard interface for basic scheduling info

## What TG Resource Should I Use?

- Collecting up cluster data from Ganglia, CluMon, Hawkeye (and soon Nagios)
- Collecting Queue data from PBS, Torque, and LSF
- One common interface to access this programatically
- One common web page
- http://snipurl.com/j24r
- Query page is next!

## Status

- Currently have a demo system up
- Queueing data from SDSC and NCSA
- Cluster data using CluMon interface at NCSA
- Basic WebMDS interface
- Getting user feedback
- Will be available as a patch download in ~3 weeks – let me know if you want to try it out!

## ESG use of MDS4 Trigger Service

- Monitoring the states of integral service components
- RLS
- SRM
- OpenDAP
- HTTP
- GridFTP fileservers
- The Trigger service periodically checks to see if services are up and running
- If a service is gone down or is unavailable for any reason, an action script is executed
- Sends email to administrators
- Update portal status page

## Slide 23

## Where do we go next?

- Extend MDS4 information providers
- More data from GT4 components
- GRAM, RFT, CAS, RLS, GridFTP
- Interface to other data sources
- Inca, GRASP
- Interface to archivers
- PinGER, NetLogger
- Additional scalability testing and development
- Additional clients
- Higher level services
- Archiving, site validation services

## Thanks

- MDS4 Team: Mike D’Arcy (ISI), Laura Pearlman (ISI), Neill Miller (UC), Jennifer Schopf (ANL)
- Students: Ioan Raicu
- This work was supported in part by the Mathematical, Information, and Computational Sciences Division subprogram of the Office of Advanced Scientific Computing Research, U.S. Department of Energy, under contract W-31-109-Eng-38, and NSF NMI Award SCI-0438372. This work also supported by DOESG SciDAC Grant, iVDGL from NSF, and others.

## For More MDS4 Information

- Jennifer Schopf
- Jms@mcs.anl.gov
- http://www.mcs.anl.gov/~jms
- Globus Toolkit MDS4
- http://www.globus.org/toolkit/mds
- Monitoring and Discovery in a Web Services Framework: Functionality and Performance of the Globus Toolkit's MDS4
- www.mcs.anl.gov/~jms/Pubs/mds-sc05.pdf

## Some Performance Data

## Index Server Stability 4.0.0

- Zero-entry index on same server
- Ran queries against it for 8,338,435 seconds (just over 96 days)
- Server machine needed to be rebuilt for patches
- Processed 623,395,877requests
- Avg 74 per second
- Average query round-trip time of 13ms
- No noticeable performance or usability degradation over the entire duration of the test

## Index Server Scalability 4.0.1

- 100-entry index on same server, running just over 47 days
- ~190K of data  has been retireved
- Processed over 20 million requests, averaging 5 per second
- No noticeable performance or usability degradation.

## Scalability Experiments

- MDS index
- Dual 2.4GHz Xeon processors, 3.5 GB RAM
- Sizes: 1, 10, 25, 50, 100
- Clients
- 20 nodes also dual 2.6 GHz Xeon, 3.5 GB RAM
- 1, 2, 3, 4, 5, 6, 7, 8, 16, 32, 64, 128, 256, 384, 512, 640, 768, 800
- Nodes connected via 1Gb/s network
- Each data point is average of 8 minutes
- Ran for 10 mins but first 2 spent getting clients up and running
- Error bars are SD over 8 mins
- Experiments by Ioan Raicu, U of Chicago, using DiPerf

## Slide 31

## Slide 32