# Presentation: 583683

## Distributed Monitoring and Information Services for the Grid

- Jennifer M. Schopf
- National eScience Centre
- &
- Argonne National Lab
- January 28, 2005

## My Definitions

- Grid:
- Shared resources
- Coordinated problem solving
- Multiple sites (multiple institutions)
- Monitoring:
- Discovery
- Registry service
- Contains descriptions of data that is available
- Expression of data
- Access to sensors, archives, etc.

## What do *I* mean by Grid monitoring?

- Different levels of monitoring needed:
- Application specific
- Node level
- Cluster/site Level
- Grid level
- Grid level monitoring concerns data
- Shared between administrative domains
- For use by multiple people
- (think scalability)

## Grid Monitoring Does Not Include...

- All the data about every node of every site
- Years of utilization logs to use for planning next hardware purchase
- Low-level application progress details for a single user
- Application debugging data (except perhaps notification of a failure of a heartbeat)
- Point-to-point sharing of all data over all sites

## Overview of This Talk

- Evaluation of information infrastructures
- Globus Toolkit MDS2, R-GMA, Hawkeye
- Insights into performance issues
- What monitoring and discovery could be
- Next-generation information architecture
- Web Service Resource Framework (WS-RF) mechanisms
- Integrated monitoring & discovery architecture for GT4

## Performance and the Grid

- It’s not enough to use the Grid, it has to perform – otherwise, why bother?
- First prototypes rarely consider performance (tradeoff with dev’t time)
- MDS1–centralized LDAP
- MDS2–decentralized LDAP
- MDS3–decentralized OGSA Grid service
- MDS4–decentralized WS-RF Web service
- Often performance is simply not known

## So We Did Some Performance Analysis

- 3 Monitoring systems
- Globus Toolkit MDS2
- EDG’s R-GMA
- Condor’s Hawkeye
- Tried to compare apples to apples
- Got some numbers as a starting point

## Globus Monitoring andDiscovery Service (MDS2)

- Part of Globus Toolkit, compatible with other elements
- Used most often for resource selection
- aid user/agent to identify host(s) on which to run an application
- Standard mechanism for publishing and discovery
- Decentralized, hierarchical structure
- Soft-state protocols
- Caching
- Grid Security Infrastructure credentials

## MDS2 Architecture

## Relational Grid Monitoring Architecture (R-GMA)

- Implementation of the Grid Monitoring Architecture (GMA) defined within the Global Grid Forum (GGF)
- Three components
- Consumers
- Producers
- Registry
- GMA as defined currently does not specify the protocols or the underlying data model to be used.

## GGF Grid Monitoring Architecture

## R-GMA

- Monitoring used in the EU Datagrid Project
- Steve Fisher, RAL, and James Magowan, IBM-UK
- Based on the relational data model
- Used Java Servlet technologies
- Focus on notification of events
- User can subscribe to a flow of data with specific properties directly from a data source

## R-GMA Architecture

## Hawkeye

- Developed by Condor Group
- Focus – automatic problem detection
- Underlying infrastructure builds on the Condor and ClassAd technologies
- Condor *ClassAd* Language to identify resources in a pool
*ClassAd Matchmaking *to execute jobs based on attribute values of resources to identify problems in a pool
- Passive Caching – updates to Agents done periodically by default

## Hawkeye Architecture

## Generic Model

## Comparing Information Systems

|  | MDS2 | R-GMA | Hawkeye |
| --- | --- | --- | --- |
| Info
Collector | Information Provider | Producer | Module |
| Info 
Server | GRIS | Producer Servlet | Agent |
| Aggregate Info Server | GIIS | Combo Producer-Consumer | Manager |
| Directory Server | GIIS | Registry | Manager |

**Notes:**
We ran a number of experiments. They were all run using the lucky cluster at argonne, and with “users” simulated at the UC nodes (evenly distributed among 20 nodes for up to 600 simulated users or queries) except in the case of R-GMA where due to architectural constraints we could only run up to about 60 users (consumers) per consumer servlet, and to have multiple consumer servlets we had to have them located at the lucky nodes (details in the paper)
All the figures are 10 minute averages, with the queries happening with a one second wait between each query (think synchronous send with a 1 second wait)

Metrics:

Throughput
Response time
Load – percentage of CPU cycles spent in user mode and system mode, recorded by ganglia
Load1 – average number of processes in the ready queue waiting to run, 1 minute average, from ganglia
Load may be high while load1 is low when a machine is running a small number of compute intensive applications. Load may be low, while load1 is high, if the same machine is trying to run a large number of applications that are blocking on I/O.

## Comparing Information Systems

- We also looked at the queries in depth - NetLogger
- 3 phases
- Connect, Process, Response

- Response

- Process

- Connect

## Some Architecture Considerations

- Similar functional components
- Grid-wide for MDS2, R-GMA; Pool for Hawkeye
- Global schema
- Different use cases will lead to different strengths
- GIIS for decentralized registry; no standard protocol to distribute multiple R-GMA registries
- R-GMA meant for streaming data – currently used for NW data; Hawkeye and MDS2 for single queries
- Push vs Pull
- MDS2 is PULL only
- R-GMA allows push and pull
- Hawkeye allows triggers – push model

## Experiments

- How many users can query an information server at a time?
- How many users can query a directory server?
- How does an information server scale with the amount of data in it?
- How does an aggregator scale with the number of information servers registered to it?

## Testbed

- Lucky cluster at Argonne
- 7 nodes, each has two 1133 MHz Intel PIII CPUs (with a 512 KB cache) and 512 MB main memory
- Users simulated at the UC nodes
- 20 P3 Linux nodes, mostly 1.1 GHz
- R-GMA has an issue with the shared file system, so we also simulated users on Lucky nodes
- All figures are 10 minute averages
- Queries happening with a one second wait between each query (think synchronous send with a 1 second wait)

## Metrics

- Throughput
- Number of requests processed per second
- Response time
- Average amount of time (in sec) to handle a request
- Load
- percentage of CPU cycles spent in user mode and system mode, recorded by Ganglia
- High when running small number compute intensive aps
- Load1
- average number of processes in the ready queue waiting to run, 1 minute average, from Ganglia
- High when large number of aps blocking on I/O

## Information Server Throughputvs. Number of Users

- (Larger
- number
- is better)

**Notes:**
The information server is the most heavily accessed component in a monitoring and information service. We evaluated the performance of the information servers for three systems—the MDS GRIS, the R-GMA ProducerServlet, and the Hawkeye Agent—when a large number of users accessed it concurrently.
10 info collectors each

MDS GRIS throughput has a near linear relationship with the number of concurrent users if the data is in cache. Its throughput, however, does not exceed 2 queries per second when the data is not in cache, suggesting that data caching is very important to support a large number of users. We observe throughput thresholds for the information servers that did not cache data.  We believe this is because the network on the server side can no longer handle the traffic from the queries, which limits the number of concurrent queries presented to the information server, and so the throughput does not rise above a set level. 
With respect to response time behavior, the GRIS results show stable performance (approximately 4 seconds per query) for 50 concurrent users or more if the GRIS has data in cache. Otherwise the performance is much worse.  For the R-GMA ProducerServlet, the response time grows almost linearly with the number of users, indicating that for larger number of users the system should be configured to have multiple ProducerServlets for the same information to allow better scaling. The Hawkeye Agent had under a 10 second response time per query until 500 users are tested, quite likely because the Hawkeye Agent does not hold the indexed resident database as the Hawkeye Manager and it has to retrieve new information for each query. 

## Query Times

- 50 users

- 400 users

- (Smaller number is better)

## Experiment 1 Summary

- Caching can significantly improve performance of the information server
- Particularly desirable if one wishes the server to scale well with an increasing number of users
- When setting up an information server, care should be taken to make sure the server is on a well-connected machine
- Network behavior plays a larger role than expected
- If this is not an option, thought should be given to duplicating the server if more than 200 users are expected to query it

## Directory Server Throughput 

- (Larger
- number
- is better)

**Notes:**
The second functionality we tested was the scalability of the directory server with the number of users. In particular, we examined the performance of the MDS GIIS, the R-GMA Registry and the Hawkeye Manager. 
10 info providers, up to 600 users

## Directory Server CPU Load

- (Smaller
- number
- is better)

## Query Times

- 50 users

- 400 users

- (Smaller number is better)

## Experiment 2 Summary

- Because of the network contention issues, the placement of a directory server on a highly connected machine will play a large role in the scalability as the number of users grows
- Significant loads are seen even with only a few users, it will be important that this service be run on a dedicated machine, or that it be duplicated as the number of users grows.

## Overall Results

- Performance can be a matter of deployment
- Effect of background load
- Effect of network bandwidth
- Performance can be affected by underlying infrastructure
- LDAP/Java strengths and weaknesses
- Performance can be improved using standard techniques
- Caching; multi-threading; etc.

## So what could monitoring be?

- Basic functionality
- Push and pull (subscription and notification)
- Aggregation and Caching
- More information available
- More higher-level services
- Triggers like Hawkeye
- Viz of archive data like Ganglia
- Plug and Play
- Well defined protocols, interfaces and schemas
- Performance considerations
- Easy searching
- Keep load off of clients

## Topics

- Evaluation of information infrastructures
- Globus Toolkit MDS2, RGMA, Hawkeye
- Throughput, response time, load
- Insights into performance issues
- What monitoring and discovery could be
_**Next-generation information architecture**_
_**Web Service Resource Framework (WS-RF) mechanisms**_
_**Integrated monitoring & discovery architecture for GT4**_

## Web Service Resource Framework (WS-RF)

- Defines standard interfaces and behaviors for distributed system integration, especially (for us):
- Standard XML-based service information model
- Standard interfaces for push and pull mode access to service data
- Notification and subscription

## MDS4 Monitoring and Discovery System 

- Components
- Information sources
- Native
- Add-on
- Higher level services
- Index
- Trigger
- Archiver
- Clients
- All of the tool are schema-agnostic, but interoperability needs a well-understood common language

## Key WS-RF concept:Resource Properties

- Every service advertises Resource Properties
- Monitoring data is “baked right in”
- WS-RF has common mechanism to expose a state data to requestors for query, update and change notification
- Service-level concept, not host-level concept
- “Native” information sources

## MDS4 Information Sources

- XML Based – not LDAP
- “Native” information sources
- All GT4 services
- “Add-on” information sources
- Code that generates resource property information
- Were called service data providers in GT3
- Soft-state registration
- Push and pull data models

## Current Service Information

- Some service data from GT4 services
- Start, timeout, etc
- GRAM
- Cluster data using interfaces to clusters (Ganglia, Hawkeye)
- Queue data using interfaces to queuing systems (PBS,LSF)
- Uses GLUE schema
- RFT
- Bits/Files transferred, load on server, etc

## “Add-on” Information Sources

- Several GT4 “services” are not WSRF services
- GridFTP
- RLS
- Will build a service to talk to it to advertise needed resource properties
- Other data can also be gathered this way
- Interfaces to other probes, archives

## MDS4 Index Service

- Index Service is both registry and cache
- Subscribes to information providers
- Data, datatype, data provider information
- Caches last value of all data
- In memory default approach

## MDS4 Trigger Service

- Compound consumer-producer service
- Subscribe to a set of resource properties
- Set of tests on incoming data streams to evaluate trigger conditions
- When a condition matches, email is sent to pre-defined address
- GT3 tech-preview version in use by ESG
- GT4 version alpha is in GT4 alpha release currently available

## MDS4 Archive Service

- Compound consumer-producer service
- Subscribe to a set of resource properties
- Data put into database (Xindice)
- Other consumers can contact database archive interface
- Will be Tech Preview in GT4 Beta release (Spring ’05)

## MDS4 Clients

- Command line, Java and C APIs
- MDSWeb Viz service
- Tech preview in current alpha release

## Slide 43

## Slide 44

## Comparing Information Systems

|  | MDS2 | R-GMA | Hawkeye | MDS4 |
| --- | --- | --- | --- | --- |
| Info
Collector | Info. Provider | Producer | Module | Service (RPs) |
| Info 
Server | GRIS | Producer Servlet | Agent | Index Service |
| Agg. Info Server | GIIS | (Archiver) | Manager | Index Service |
| Directory Server | GIIS | Registry | Manager | Index Service |

## Current Release Schedule

- Available now: Alpha
- www-unix.globus.org/toolkit/downloads/development/
- March 2005- beta
- May (ish) 2005 – final

## Many places where additionalwork with MDS4 is needed...

- Extend MDS4 information providers
- More data from GT4 services (GRAM, RFT, RLS)
- Interface to other tests (Inca, GRASP)
- Interface to archiver (PinGER, Ganglia, others)
- Scalability testing and development
- Additional clients
- If tracking job stats is of interest this is something we can talk about

## Other Possible HigherLevel Services

- Site Validation Service
- Prediction service (ala NWS)
- Interfacing to Netlogger?
- What else do you think we need?

## We Need Security

- Need I say more?

## Summary

- Current monitoring systems
- Insights into performance issues
- What we really want for monitoring and discovery is a combination of all the current systems
- Next-generation information architecture
- WS-RF
- MDS4 plans
- Additional work needed!

## Thanks

- Students: Xuehai Zhang (UC), Jeffrey Freschel (UW)
- Testbed/Experiment support and comments
- John Mcgee, ISI; James Magowan, IBM-UK; Alain Roy and Nick LeRoy at University of Wisconsin, Madison;Scott Gose and Charles Bacon, ANL; Steve Fisher, RAL; Brian Tierney and Dan Gunter, LBNL.
- This work was supported in part by the Mathematical, Information, and Computational Sciences Division subprogram of the Office of Advanced Scientific Computing Research, U.S. Department of Energy, under contract W-31-109-Eng-38. This work also supported by DOESG SciDAC Grant, iVDGL from NSF, and others.

## For More Information

- Jennifer Schopf
- _[jms@mcs.anl.](mailto:jms@mcs.anl.gov)__[gov](mailto:jms@mcs.anl.gov)_
- _[http://www.mcs.anl.gov/~jms](http://www.mcs.anl.gov/~jms)_
- Globus Toolkit MDS4
- _[http://www.globus.org/mds](http://www.globus.org/mds)_
- Scalability comparison of MDS2, Hawkeye, R-GMA
- _[www.mcs.anl.gov/~jms/Pubs/xuehaijeff-hpdc2003.](http://www.mcs.anl.gov/~jms/Pubs/xuehaijeff-hpdc2003.pdf)__[pdf](http://www.mcs.anl.gov/~jms/Pubs/xuehaijeff-hpdc2003.pdf)_
- Journal paper in the works – email if you want a draft
- Monitoring Clusters, Monitoring the Grid – ClusterWorld
- _[http://www.grids-center.org/news/clusterworld/](http://www.grids-center.org/news/clusterworld/)_

## Extra Slides