# Presentation: 184017

## Distributed Monitoring and Information Services for the Grid

- Jennifer M. Schopf
- Argonne National Laboratory
- NeSC
- Feb 20, 2006

## What is a Grid

- Resource sharing
- Computers, storage, sensors, networks, ...
- Sharing always conditional: issues of trust, policy, negotiation, payment, ...
- Coordinated problem solving
- Beyond client-server: distributed data analysis, computation, collaboration, ...
- Dynamic, multi-institutional virtual orgs
- Community overlays on classic org structures
- Large or small, static or dynamic

## Why is this hard/different?

- Lack of central control
- Where things run
- When they run
- Shared resources
- Contention, variability
- Communication
- Different sites implies different sys admins, users, institutional goals, and often “strong personalities”

## So why do it?

- Computations that need to be done with a time limit
- Data that can’t fit on one site
- Data owned by multiple sites
- Applications that need to be run bigger, faster, more

## What Is Grid Monitoring?

- Sharing of community data between sites using a standard interface for querying and notification
- A way to discover what services and resources are available to use
- A way to understand the status/attributes of those services
- A system to warn you when things fail

## Monitoring Use cases

- PPGD/GriPhyN/iVDGL monitoring group (2002-2004) found roughly 4 categories
- Health of system (NW, servers, cpus, etc)
- Resource selection
- System upgrade evaluation (have systems reached capacity)
- Application-specific progress tracking
- First three types need roughly the same information
- Fourth is user-specific and application specific – no general solution yet
- http://www.mcs.anl.gov/~jms/pg-monitoring

## Health of the System“Is the Grid up?”

- Health of the System“Is the Grid up?”

- Brief Description
- User of a grid replication service finds actions are much slower than normal
- Not sure if problem is with network, disk, CPU end points, or something inbetween
- Need archive data for historical, current streaming for comparison
- Performance events/sensors required
- Host monitoring - CPU,memory, disk
- Network path monitoring - bw, lat., traceroute
- GridFTP monitoring
- TCP stack monitoring (web 100)
- Possibly switch/router monitoring
- May want different data for user vs sys admins

## Resource Selection

- Brief Description
- User/Broker wants to decide where to run a job
- Sites advertise cluster information for grid-level scheduling decisions
- Also need data about storage locations and access speeds
- Information must be summarized for advertising to Grid, scalability is key issue
- Performance events/sensors required
- Static: number of compute nodes, cpu type and speed, OS, installed sw, available storage systems
- Dynamic:Queue lengths, large file transfer times

## What shouldmonitoring systems look like?

- All sensors must be non-intrusive
- All data is small, and must be “as timely as possible”
- All data must be kept for a long time (years), and must be accessible in many ways
- No one really knows how many sensors will be accessed at one time (or reporting to a higher level service), or how often they will be accessed
- Security isn’t of concern – YET – except for job data

## Monitoring Systems (2)

- Line between monitoring system and higher level services isn’t always clear
- Archiving
- Summary statistics
- Predictions
- Error detection
- Alarms/notification

## OUTLINE

- Grid Monitoring and Use Cases
- MDS4
- Index Service
- Trigger Service
- Information Providers
- Deployments
- Metascheduling data for TeraGrid
- Service failure warning for ESG
- Performance Numbers

## What is MDS4?

- Grid-level monitoring system used most often for resource selection
- Aid user/agent to identify host(s) on which to run an application
- Uses standard interfaces to provide publishing of data, discovery, and data access, including subscription/notification
- WS-ResourceProperties, WS-BaseNotification, WS-ServiceGroup
- Part of the Globus Toolkit v4
- Functions as an hourglass to provide a common interface to lower-level monitoring tools

## GLUE Schema Attributes

- (cluster info,
- queue info, FS info)

- Information Users :
- Schedulers, Portals, Warning Systems, etc.

- WS standard interfaces for subscription, registration, notification

## Web ServiceResource Framework (WS-RF)

- Defines standard interfaces and behaviors for distributed system integration, especially (for us):
- Standard XML-based service information model
- Standard interfaces for push and pull mode access to service data
- Notification and subscription

## MDS4 UsesWeb Service Standards

- WS-ResourceProperties
- Defines a mechanism by which Web Services can describe and publish resource properties, or sets of information about a resource
- Resource property types defined in service’s WSDL
- Resource properties can be retrieved using WS-ResourceProperties query operations
- WS-BaseNotification
- Defines a subscription/notification interface for accessing resource property information
- WS-ServiceGroup
- Defines a mechanism for grouping related resources and/or services together as service groups

## MDS4 Components

- Higher level services
- Index Service – a way to aggregate data
- Trigger Service – a way to be notified of changes
- Both built on common aggregator framework
- Information providers
- Monitoring is a part of every WSRF service
- Non-WS services can also be used
- Clients
- WebMDS
- All of the tool are schema-agnostic, but interoperability needs a well-understood common language

## MDS4 Index Service

- Index Service is both registry and cache
- Subscribes to information providers
- Publishes (as resource properties)
- Datatype and data provider info, like a registry
- Last value of data, like a cache
- In memory default approach, DB backing store currently being developed to allow for very large indexes
- Soft-state registration
- Can be set up for a site or set of sites, a specific set of project data, or for user-specific data only
- Can be a multi-rooted hierarchy

## Index Service Facts 1

- _No single global Index_ provides information about every resource on the Grid
- No person in the world is part of every VO!
- Hierarchies or special purpose index’s are common
- Each virtual organization will have different policies on who can access its resources
- The presence of a resource in an Index makes _no guarantee about the availability_ of the resource for users of that Index
- Ultimate decision about whether to use the resources is left to direct negotiation between user and rsc
- MDS does not need to keep track of policy information (something that is hard to do concisely)
- Rscs do not need to reveal their policies publicly

## Index Service Facts 2

- MDS has a _soft consistency model_
- Published information is recent, but not guaranteed to be the absolute latest
- Load caused by information updates is reduced at the expense of having slightly older information
- Free disk space on a system 5 minutes ago rather than 2 seconds ago.
- Each registration into an Index Service is subject to _soft-state lifetime_ management
- All registrations has expiry times and must be periodically renewed
- Index is self-cleaning, since outdated entries disappearing automatically

## MDS4 Trigger Service

- Subscribe to a set of resource properties
- Evaluate that data against a set of pre-configured conditions (triggers)
- When a condition matches, email is sent to pre-defined address
- Similar functionality in Hawkeye

## Aggregator Framework

- General framework for building services that collect and aggregate data
- Index and Trigger service both use this
- 1) Common interface implemention
- Java class that implements an interface to collect XML-formatted data from information providers
- Implements WS-RP and WS-N for query and subscription
- 2) Common configuration mechanism
- Maintain information about which information providers to use and their associated parameters
- Specify what data to get, and from where
- 3) Services are self-cleaning
- Each registration has a lifetime
- If a registration expires without being refreshed, it and its associated data are removed from the server

## Aggregator Framework

- General framework for building services that collect and aggregate data
- Index and Trigger service both use this
- 1) Collect information via aggregator sources (information providers)
- Java class that implements an interface to collect XML-formatted data
*Query source* uses WS-ResourceProperty mechanisms to poll a WSRF service
*Subscription source* collects data from a service via WS-Notification subscription/notification
*Execution source* executes an administrator-supplied program to collect information

## Aggregator Framework (cont)

- 2) Common configuration mechanism
- Maintain information about which aggregator sources to use and their associated parameters
- Specify what data to get, and from where
- 3) Aggregator services are self-cleaning
- Each registration has a lifetime
- If a registration expires without being refreshed, it and its associated data are removed from the server.

## Aggregator Framework

## Information Providers

- Data sources for the higher level services (eg. Index, Trigger)
- WSRF-compliant service
- WS-ResourceProperty for Query source
- WS-Notification mechanism for Subscription source
- Other services/data sources
- Executable program that obtains data via some domain-specific mechanism for Execution source.

## Information Providers:Cluster and Queue Data

- Information Providers:Cluster and Queue Data

- Interfaces to Hawkeye, Ganglia, CluMon
- Not WS so these are Execution Sources
- Basic host data (name, ID), processor information, memory size, OS name and version, file system data, processor load data
- Some condor/cluster specific data
- Interfaces to PBS, Torque LSF queue system
- Queue information, number of CPUs available and free, job count information, some memory statistics and host info for head node of cluster

## Information Providers:GT4 Services

- Information Providers:GT4 Services

- Every WS built using GT4 core
- ServiceMetaDataInfo element includes start time, version, and service type name
- Reliable File Transfer Service (RFT)
- Service status data, number of active transfers, transfer status, information about the resource running the service
- Community Authorization Service (CAS)
- Identifies the VO served by the service instance
- Replica Location Service (RLS)
- Note: not a WS
- Location of replicas on physical storage systems (based on user registrations) for later queries

## Slide 28

## WebMDS User Interface

- Web-based interface to WSRF resource property information
- User-friendly front-end to the Index Service
- Uses standard resource property requests to query resource property data
- XSLT transforms to format and display them
- Customized pages are simply done by using HTML form options and creating your own XSLT transforms
- Sample page:
- http://mds.globus.org:8080/webmds/webmds?info=indexinfo&xsl=servicegroupxsl

## WebMDS Service

## Any questions before I walk through two current deployments?

- Grid Monitoring and Use Cases
- MDS4
- Index Service
- Trigger Service
- Information Providers
- Deployments
- Metascheduling Data for TeraGrid
- Service Failure warning for ESG
- Performance Numbers

## Working with TeraGrid

- Large US project across 9 different sites
- Different hardware, queuing systems and lower level monitoring packages
- Starting to explore MetaScheduling approaches
- GRMS (Poznan)
- W. Smith (TACC)
- K. Yashimoto (SDSC)
- User Portal
- Need a common source of data with a standard interface for basic scheduling info

## Cluster Data

- Provide data at the subcluster level
- Sys admin defines a subcluster, we query one node of it to dynamically retrieve relevant data
- Can also list per-host details
- Interfaces to Ganglia, Hawkeye, CluMon, and Nagios available now
- Other cluster monitoring systems can write into a .html file that we then scrape

## Cluster Info

- UniqueID
- Benchmark/Clock speed
- Processor
- MainMemory
- OperatingSystem
- Architecture

- Number of nodes in a cluster/subcluster
- TG specific Node properties
- StorageDevice
- Disk names, mount point, space available

## Data to collect: Queue info

- LRMSType
- LRMSVersion
- DefaultGRAMVersion and port and host
- TotalCPUs
- Status (up/down)
- TotalJobs (in the queue)

- RunningJobs
- WaitingJobs
- FreeCPUs
- MaxWallClockTime
- MaxCPUTime
- MaxTotalJobs
- MaxRunningJobs

- Interface to PBS (Pro, Open, Torque), LSF

## How will the data be accessed?

- Java and command line APIs to a common TG-wide Index server
- Alternatively each site can be queried directly
- One common web page for TG
- http://snipurl.com/j24r
- Query page is next!

## Status

- Currently have a demo system up
- Queuing data from SDSC and NCSA
- Cluster data using CluMon interface at NCSA
- Basic WebMDS interface
- Being deployed more widely for TeraGrid this week
- General patch for 4.0.1 deployments should be available soon – let me know if you’re interested!

## ESG use of MDS4 Trigger Service

- Need a way to notify system administrators and users what the status of their services are
- In particular, interested in
- Replica Locatoin Service (RLS)
- Storage Resource Manager service (SRM)
- OpenDAP
- Web Server (HTTP)
- GridFTP fileservers

## Trigger Service and ESG Cont.

- The Trigger service periodically checks to see if services are up and running
- If a service is gone down or is unavailable for any reason, an action script is executed
- Sends email to administrators
- Update portal status page
- Been in use for over a year (used GT3 version previously)

## Slide 44

## OUTLINE

- Grid Monitoring and Use Cases
- MDS4
- Index Service
- Trigger Service
- Information Providers
- Deployments
- Metascheduling Data for TeraGrid
- Service Failure warning for ESG
- Performance Numbers

## MDS4 Stability

| Vers. | Index
Size | Time up
(Days) | Queries
Processed | Query
Per
Sec. | Round-trip
Time (ms) |
| --- | --- | --- | --- | --- | --- |
| 4.0.1 | 25 | 66+ | 81,701,925 | 14 | 69 |
| 4.0.1 | 50 | 66+ | 49,306,104 | 8 | 115 |
| 4.0.1 | 100 | 33 | 14,686,638 | 5 | 194 |
| 4.0.0 | 1 | 14 | 93,890,248 | 76 | 13 |
| 4.0.0 | 1 | 96 | 623,395,877 | 74 | 13 |

## Scalability Experiments

- MDS index
- Dual 2.4GHz Xeon processors, 3.5 GB RAM
- Sizes: 1, 10, 25, 50, 100
- Clients
- 20 nodes also dual 2.6 GHz Xeon, 3.5 GB RAM
- 1, 2, 3, 4, 5, 6, 7, 8, 16, 32, 64, 128, 256, 384, 512, 640, 768, 800
- Nodes connected via 1Gb/s network
- Each data point is average of 8 minutes
- Ran for 10 mins but first 2 spent getting clients up and running
- Error bars are SD over 8 mins
- Experiments by Ioan Raicu, U of Chicago, using DiPerf

## MDS4 Throughput

## MDS4 Response Time

## Index Maximum Size

| Heap
Size (MB) | Approx. Max.
Index Entries | Index
Size (MB) |
| --- | --- | --- |
| 64 | 600 | 1.0 |
| 128 | 1275 | 2.2 |
| 256 | 2650 | 4.5 |
| 512 | 5400 | 9.1 |
| 1024 | 10800 | 17.7 |
| 1536 | 16200 | 26.18 |

## Performance

- Is this enough?
- We don’t know!
- Currently gathering up usage statistics to find out what people need
- Bottleneck examination
- In the process of doing in depth performance analysis of what happens during a query
- MDS code, implementation of WS-N, WS-RP, etc
- (These numbers are in an HPDC submission)

## Summary

- MDS4 is a WS-based Grid monitoring system that uses current standards for interfaces and mechanisms
- Available as part of the GT4 release
- Currently in use for resource selection and fault notification
- Initial performance results aren’t awful – we need to do more work to determine bottlenecks

## Where do we go next?

- Extend MDS4 information providers
- More data from GT4 WS
- GRAM, RFT, CAS
- More data from GT4 non-WS components
-   RLS, GridFTP
- Interface to other data sources
- Inca, GRASP
- Interface to archivers
- PinGER, NetLogger
- Additional scalability testing and development
- Additional clients

## Other Possible HigherLevel Services

- Other Possible HigherLevel Services

- Archiving service
- The next high level service we’ll build
- Looking at Xindice as a possibility
- Site Validation Service (ala Inca)
- Prediction service (ala NWS)
- What else do you think we need?

## Contributing to MDS4

- Globus is opening up it’s development environment – similar to Apache Jakarta
- MDS4 will be a project in the new scheme
- Contact me for more details
- jms@mcs.anl.gov
- http://dev.globus.org

## Thanks

- MDS4 Team: Mike D’Arcy (ISI), Laura Pearlman (ISI), Neill Miller (UC), Jennifer Schopf (ANL)
- Students: Ioan Raicu, Xuehai Zhang
- This work was supported in part by the Mathematical, Information, and Computational Sciences Division subprogram of the Office of Advanced Scientific Computing Research, U.S. Department of Energy, under contract W-31-109-Eng-38, and NSF NMI Award SCI-0438372. This work also supported by DOESG SciDAC Grant, iVDGL from NSF, and others.

## For More Information

- Jennifer Schopf
- Jms@mcs.anl.gov
- http://www.mcs.anl.gov/~jms
- Globus Toolkit MDS4
- http://www.globus.org/toolkit/mds
- Monitoring and Discovery in a Web Services Framework: Functionality and Performance of the Globus Toolkit's MDS4
- http://www.mcs.anl.gov/~jms/Pubs/ mds4.hpdc06.pdf