# Presentation: 700168

## Globus Monitoring and Discovery

- Ben Clifford
- USC/ISI & Globus
- For MDS info: contact Dr. Jennifer Schopf, jms@mcs.anl.gov
- Copyright (c) 2002-5 University of Chicago and The University of Southern California. All Rights Reserved.  This presentation is licensed for use under the terms of the Globus Toolkit Public License.See http://www.globus.org/toolkit/download/license.html for the full text of this license.

## Talk Outline

- What is MDS?
- Concepts
- Architecture
- Components
- Deployment
- Programming
- Past and future

## MDS is

- The Monitoring and Discovery System

## MDS addresses two problems

- Two problems
- Monitoring
- Discovery
- Look different but we can use similar techniques for both

**Notes:**
MDS addresses two problems: monitoring and discovery. These are two distinct problems but similar techniques can be used to address both.

## Monitoring grid resources

- Status of resource on grid
- Up/down?

## Discovery

- Start with a task to perform on the grid
- For example, want to perform run a simulation

**Notes:**

Resource Discovery
“Which resources are relevant to my task?”
Resource Status Inquiry
“Which resource is best to use (now)?”
Resource Control
Use the resource
submitting a job
Not part of MDS!

Sequence of questions we must ask in choosing a resource.
(give examples using compute nodes)
DISCOVERY: “What IRIX boxes with more than 8 processors exist in VO?”
STATUS INQ: “What is the CPU and disk utilisation at the moment?”

## Discovering a grid resource

- Compute system

- storage

- other

- user

**Notes:**
Here is the grid – a lot of resources of various kinds: compute resources, storage resources and other kinds that we do not necessarily need to know about.
In the middle of it all, we have a user.
The user doesn’t necessarily know which resources are available.
MDS makes this information available, so that we can perform the first step in discovery...

## 1. Which resources are relevant?

**Notes:**
The first step is to select the resources which are relevant. In our example, the user selects the compute resources, discarding the other types. So now the user has a list of (4) compute systems.
But the user doesn’t know which to choose...

## 2. Which resources are best for task?

- 16 fast CPUs
- 500Gb disk
- High load

- 2 CPUs
- 100Gb disk
- Medium load

- 128 slow
- CPUs
- Low load

**Notes:**
The next stage is for MDS to provide information about each of the relevant resources, so that we can pick which one is best.
Note that there is not an overall ‘best’ that fits every situation. For example, in the case of computation, we might have code that is very parallelised, in which case we can make use of a large number of CPUs; or it might be more single threaded, in which case a small number of CPUs is appropriate.

## 3. Choose a resource

- 128 slow
- CPUs
- Low load

**Notes:**
So having got information about each relevant resource, the user can pick the best resource for the task at hand...

## 4. Attempt to use resource

- Job submission with GRAM
**NOT** part of MDS

**Notes:**
And having chosen the resource, can try to use it (for example, by submitting a job through the GRAM component of GT)

## What makes this difficult on the grid?

- Distributed users and resources
- Sometimes unreliable network
- Variable resource status
- Resources come up and go down without any centralised co-ordination
- Variable grouping
- Different people belong to different groups (*Virtual Organisations*)
- The grid is not cleanly partitioned

## Resource Discovery/Monitoring

- Distributed users and resources
- Variable resource status
- Variable grouping
- Green VO has become partitioned because of network failure!

**R**

**R**

**R**

**R**

**R**

**R**

**?**

**?**

**R**

**R**

**R**

**R**

**R**

**R**

**R**

**R**

**R**

**?**

**?**

**R**

**R**

**R**

**R**

**R**

*dispersed users*

**Pink VO**

**Green VO**

* **network*

**Notes:**
This is an example of a grid environment
Two groups of people and resources: “purple” and “green” – these are Virtual Organisations.
Distributed between locations
Some things are in both groups.
There are failures – individual resources fail; connectivity between resources fail
Eg. The green VO is partitioned into two pieces.
When stuff fails, we want what is left to still work

## What does this look like to users?

**Notes:**
To give something a bit more concrete, here is one of our user interfaces

## Examples of Useful Information

- Characteristics of a compute resource
- Software available, networks connected to, load, type of CPU, disk space
- Characteristics of the Globus infrastructure
- Hosts, resource managers, service availability
- Policy
- Difficult...
- More examples later on

**Notes:**

Policy is ‘?’ because no one seems to have worked out how to express it well even though people do want to express it

## Key Concepts

- Virtual Organizations (VOs)
- Group together resources and users in related communities
- Support community-specific discovery
- Specialized views
- Scalability

**Notes:**
Probably seen VOs talked about in various other talks.

## Virtual Organizations

- Collaborating individuals and institutions
- Enable sharing of resources
- Non-locality of participants
- Dynamic in nature
- VOs come and go
- Resources join and leave VOs
- Resources change status and fail
- Community-wide goals
- Must not interfere with each other
- In support of this, provide VO-wide resources for MDS
- Other VO-wide resources, such as CAS for security

**Notes:**
Shared goals – for example working on the same research project
Sharing of resources – towards those same shared goals
Non-locality of participants – affiliated to different real-world organisations (for example universities) on different networks

For example, we might form a VO of people with the community goal of analysing a specific set of this week’s results from an experiment. This is quite dynamic – the VO might not exist for more than a week or so.

Some participants will provide resources (compute systems, storage, datasets) to the VO.

## Pink VO

**R**

**R**

**R**

**R**

**R**

**R**

**?**

**?**

**R**

**R**

**R**

**R**

**R**

**Pink VO**

**Notes:**
Here is a VO, in a pink bubble. Call it the pink VO.

Users at two locations
Resources at four locations – two failed resources
A network connects the resources and users together.

## Green VO

**R**

**R**

**R**

**R**

**R**

**R**

**R**

**R**

**?**

**?**

**R**

**Green VO**

* **network*

**Notes:**
Here is another VO, this one in a green bubble... call it the Green VO
Similar to the Pink VO.
But network has a failure! so the VO has been partitioned into two sections that cannot communicate with each other.
Users in each section to be able to use the resources in their section.

## The Grid

- Some resources are in both VOs
- Some resources are in neither VO
- But are in other VOs

**R**

**R**

**R**

**R**

**R**

**R**

**?**

**?**

**R**

**R**

**R**

**R**

**R**

**R**

**R**

**R**

**R**

**?**

**?**

**R**

**R**

**R**

**R**

**R**

*dispersed users*

**Pink VO**

**Green VO**

* **network*

**Notes:**
On the Grid there are lots of VOs.
Some of them overlap – here some resources are in both the Pink and Green VO.
Things happening in the green VO should not affect the pink VO, and vice-versa, even though there are shared resources.

## Scalability

- Large numbers
- Many resources
- Many users
- Independence
- Resources shouldn’t affect one another
- VOs shouldn’t affect one another
- Graceful degradation of service
- “As much function as possible”
- Tolerate partitions, prune failures

**Notes:**
Failures in one resource or VO shouldn’t have affect in another VO or on a resource.

Partitions and prunes: where parts of the network are cut off from each other

## Grid Information: Facts of Life

- Information is always old
- Time of flight; changing system state
- Distributed snapshot of state hard to obtain
- Components will fail
- Scalability and overhead
- Many different usage scenarios

**Notes:**
Time  of flight – takes some time for information to be delivered from its source to its destinations
Changing system state means that by the time you get information, it is ‘wrong’. For example, say we take several seconds to propagate memory usage information from a resource, and memory usage changes on the resource host, then the information is wrong.
The above means its difficult to say ‘this is what the grid looks like *NOW*’. Instead we can mark data with aging information to let you determine when it was valid. To illustrate, it is useful to know that a disk had a gigabyte free on it 60 seconds ago, even though it now has only 0.9 gigabytes free.

Components will fail. Of course they will.

Discussed scalability on a previous slide
Het.policy: different organisations (virtual and non-virtual) have different policies on what their resources can be used for.

Eg of rapidly changing info: cpu load, free disk space.
But even if logged into console and type ‘uptime’, that data is out of date within 5 seconds.
We believe this is enough.
Qualitydata==timestamping of validity – later on in schema examples, we’ll see what this looks like

applications need to be aware that the data maybe slightly out of date.
We’re trying to provide general purpose information as part of toolkit. Other more specialised stuff can be added.

## MDS in a VO

*VO-specific indexes*

**R**

**R**

**R**

**R**

**A**

**A**

- information

**Notes:**
Here is a sketch outline of MDS in a VO
Will go through the various components on following slides

## MDS in a VOComponents - Resources

- Resources
- Things you can use on the grid
- For example, GRAM installations
- Resources have MDS information
- Might publish themselves (eg. WSRF based services)
- Might be collected by a separate probe (eg. GridFTP)
- Resources do other things – for example, GRAM is primarily intended to run compute jobs

**R**

## MDS in a VOComponents

- Indexes
- Collect information from Resources and publishes in one place
- A is for ‘Aggregator’

## MDS in a VOFlow of information

- Users can get information in two ways
- Query from resources directly
- Information more recent
- Higher load on resources and on requestor
- Need to know which resources exist in order to query them
- Query from index
- Do not need to know existence of each resource
- Information is older

*VO-specific indexes*

**R**

**R**

**R**

**R**

**A**

**A**

- information

## MDS in a VOConstructing a hierarchy

- Resources registered into Indexes
- Soft-state registration
- Keeps index clean
- Old entries disappear automatically
- Registrations configured at index or at resource

*VO-specific indexes*

**R**

**R**

**R**

**R**

**A**

**A**

- registrations

**Notes:**
The indexes and resources need to know to communicate with each other – there is a registration process for this.

## MDS in a VOConstructing a hierarchy

- Indexes can be registered to other indexes
- Index at top of slide contains information about all 6 resources

**R**

**R**

**A**

**A**

- registrations

**Notes:**
Can do more complicated registration patterns– indexes can be registered into other indexes to produce larger and more scalable indexes

## MDS in a VOAge of information

- As information flows upwards
- There is information about more resources
- The information is generally older

**R**

**R**

**A**

**A**

- Older
- more

- Fresher
- less

**FRESH**

**MORE**

- ~30s

- ~5m

- ~10m

**Notes:**
Can query high up to get lots of information (about many resources); this information will be older.
Can query lower down to get fresher information about fewer resources.
At the extreme, can talk to the resource itself – we get the latest information (often only seconds old) – but:
	i) the information is just about that resource
	ii) we need to know the resource exists and how to communicate with it – we can get this information from a higher index

## Distributed Services

- Service scales with Grid growth
- Loose consistency model tolerates failures
- Interoperability through common protocols

**A**

**A**

**A**

**A**

**Notes:**
Here is how we can deploy this into the VOs that we saw earlier. 

## MDS components

- MDS services
- Index
- Trigger
- Component-specific information
- Queue status information for GRAM
- WebMDS
- Web based UI
- GT4 WSRF core provides basic protocol handling, command line tools

## Grid Resources

- MDS collects information about each Grid Resource
- WSRF resources can provide their own information
- Other types of Grid Resource have an MDS information source to collect the information.
- Much information is dynamic
- Load, process information, storage information, etc.
- “White pages” lookup of resource information
- Ex: How much memory does this compute resource have?
- “Yellow pages” lookup of resource options
- Ex: Which queues on this resource allows large jobs?

## Collective services

- Collective services aggregate information from multiple resource services, and process in specific ways
- Index service
- Trigger service

## Index Service

- Maintain set of registered Grid Resources
- Track incoming live registrations
- Indexes can be registered hierarchically
- Cache of monitoring data for each Grid Resource
- Akin to web search engines
- “Which compute resources have 32 or more processors?”

## Trigger Service

- List of rules
- Rules applied to monitoring data
- When rules match, action performed
- Send e-mail to users/administrators

## MDS Security

- GT WS core security allows operations to be restricted to certain listed users.
- Restrict who can see service data
- Restrict who can register into an index service

**Notes:**
Fairly basic security at the moment, using GT3 core security mechanisms
Future plans include more interesting policy mechanisms

## Gathering monitoring data

- Data can be gathered about Grid Resources in a number of ways.
- WSRF
- Most GT4 services
- Protocol specific sources
- GridFTP, RLS, other

## WSRF

- Web Services Resource Framework
- Grid Resources <-> *WS-Resources*
- XML-based *WS-Resource Properties*
- Monitoring and discovery information
- Standard access mechanisms
*Polling *
*Subscription (using *WS-Notification*)*

## Information models

- Each information sources publishes information in XML according to some schema.
- Some times the author of the information source or the grid resource defines that schema.
- Some collaborative efforts to define common schemas– for example *GLUE* for compute information
- Schema typically written in XSD, but not required

## GLUE schema

- Grid Laboratory Uniform Environment
- Schema developed by DataTAG for EU/USA interoperability.
- Modelled in UML
- Implementations
- XML version for MDS
- Information collected from various cluster monitoring systems
- Also: LDAP and SQL versions (used by older versions of MDS and other monitoring systems).

## GLUE Schema v1.0

**Notes:**
This is here just for the sake of completeness – Don’t expect anyone to be able to read it...

## GLUE schema example

- <ce:Cluster ce:Name="stomoxys.isi.edu" ce:UniqueID="stomoxys.isi.edu“ogsi:availableUntil="2003-12-01T23:00:25.690Z“ ogsi:goodFrom="2003-12-01T22:59:54.690Z“ ogsi:goodUntil="2003-12-01T23:00:24.690Z">
- <ce:SubCluster ce:Name="stomoxys.isi.edu“ ce:UniqueID="stomoxys.isi.edu">
- <ce:Host ce:Name="stomoxys.isi.edu“ ce:UniqueID="stomoxys.isi.edu">
- <ce:OperatingSystem ce:Name="Linux“ ce:Release="2.4.18-14" />
-  <ce:ProcessorLoad ce:Last15Min="074"  ce:Last1Min="095" ce:Last5Min="078"  />
- ...
- </ce:Cluster>

**Notes:**
This is a rather cut down fragment of a piece of GLUE schema service data. We’ll go through the interesting components one by one.

## MDS user interfaces

- General purpose UIs
- Web browser based interface - WebMDS
- Command line tools
- Specialized clients
- Brokers

## WebMDS

- Web-based interface to display monitoring information
- Easily extensible for new data using XSLT

## MDS in the GT4 container

- Layout of MDS services in container
- Configuration

## Containerwide index

**R**

**R**

**A**

- Each GT4 container has a local index
- Collects information about services in that container
- Each service registers to container index when correctly configured

- GRAM

- RFT

- CAS

- Container wide index

- Container

**Notes:**
More complicated registration patterns can be formed – indexes can be registered into other indexes to produce larger and more scalable indexes

## VO-wide indexes

- Local indexes can be registered to VO wide indexes
- Config file at resource container or at VO index – contains URL for resource or VO index

- VO-wide index

**Notes:**
More complicated registration patterns can be formed – indexes can be registered into other indexes to produce larger and more scalable indexes
Resources are registered into an index. This can be configured with at the indexes or at the resources – both ways are approximately equivalent – up to administrators of indexes and resources to decide which way to do it.
Configuration takes the form of specifying a URL 

## Registering a container index into a VO index

- Config file:
- $GL/etc/etc/globus_wsrf_mds_index/hierarchy.xml
- Two ways:
- At the VO index
- Configure the URLs of container indexes
- Add lines like:
- <upstream>http://myresource.isi.edu:8080/wsrf/services/DefaultIndexService</upstream>
- At the resource containers
- Configure the URLs of the VO indexes
- Add lines like:
- <downstream>http://myvo.org:8080/wsrf/services/DefaultIndexService</downstream>

## Configuring GRAM to use a cluster monitoring system

- GRAM extracts and publishes cluster information from either Ganglia or Hawkeye
- $GLOBUS_LOCATION/etc/globus_wsrf_mds_usefulrp/gluerp.xml
- <defaultProvider> tag specifies whether to use Ganglia or Hawkeye or none.
- Uncomment appropriate example supplied in the config file

## Programming with MDS4

- More information
- Directly from resources
- Gatewaying via other monitoring systems
- Write new collective functionality

## WSRF APIs

- WSRF services can be implemented in a number of languages.
- Java – Globus
- C – Globus
- Python – LBL
- .NET – Virginia
- Write your service using GT4 WSRF core and you will get MDS compatibility almost magically.

**Notes:**
 A lot (but not all) of MDS communication happens through WSRF protocols. By making a WSRF compliant service, MDS will be able to very easily publish information from your service.

## Publish new information

- Decide what information to publish into MDS and where
- Define XML schema
- Write code to publish info
- As part of a new service
- As part of existing implementation

## Write new collective layer services

- Aggregator framework provides:
- Common VO-level service functionality
- Registration management
- Collection of information from Grid Resources
- Developer can plug in specialised functionality
- Index, Trigger, as well as a prototype of an archiver all use this framework.

## New ways to collect informtion

- Aggregator has pluggable sources
- Collect via WSRF (poll or subscription)
- Collect using custom executable
- Extensible – new ways to collect (new information sources) can be written in Java or as executables

## Historical MDS

- MDS1 (Metacomputing Directory Service)
- Centralized database
- Globus 1.1.2 and earlier
- Did not scale
- Single point of failure
- LDAP based
- MDS2 (Monitoring and Discovery Service)
- Distributed services
- In Globus Toolkit 1.1.3 and GT2.x
- Two classes of server: GIIS and GRIS
- LDAP based
- Lazy caching presented scalabilty problem
- MDS3 (Monitoring and Discovery System)
- Even more distributed services
- Based around OGSI standard
- In Globus Toolkit 3.x
- MDS4
- Based around WSRF standards
- In Globus Toolkit 4.x
- More native components - (web UI, trigger service, ganglia, hawkeye)

**Notes:**
MDS1 –one giant central ldap server. Everyone fed information into that.
Catastrophic failure when it became overloaded.

In mds 2, when a GRIS or GIIS fails, then only single service (eg. Resource info or the index) is unavailable
Softstate reconstruction when an index recovers

## Ways forward

- Archiving
- Record historical monitoring data
- Partial notification
- Send only small updates rather than a complete result set
- Helps scalability
- Gateway into other monitoring/discovery systems
- Eg. monalisa
- More interesting monitoring data
- Security
- Who can see what
- Referrals

## What now?

- Download the toolkit
- http://www.globus.org/toolkit
- Globus support lists:
- discuss@globus.org
- developer-discuss@globus.org
- MDS web pages:
- http://www.globus.org/mds