# Presentation: 355075

## Introduction to Grid Monitoring

*Stratos Efstathiadis*
*BNL*
*ITD – STAR PPDG*

## Introduction to Grid Monitoring

- Grid Monitoring
- Monitoring in distributed systems
- Grid Monitoring Architecture (GMA)
- Grid Monitoring Systems
-  Monitoring and Discovery System (MDS)
- Components
- Hierarchical structure
- Clients
- A Short Introduction to JINI
- MonALISA

## Monitoring

- Several kind of monitoring:
- Monitoring of the Resource (facility)
- Network Monitoring
- Job Monitoring ( status of jobs)
- ...
- I’ll talk mostly about Monitoring of the Resource.

## Monitoring

- The process of dynamic collection, interpretation and presentation of information about hardware and software systems
- Why do we need monitoring?
- Debugging purposes
- Resource Utilization
- Performance Evaluation
- Security
- Management Decisions
- Accounting

## Monitoring Distributed Systems

- The challenges of monitoring Distributed Systems:
-    (from monitoring tools to monitoring systems)
- No single point of observation
- No central point of monitoring information
- Diverse Hardware and Software Systems
- Different policies and decision making mechanisms
- Network monitoring is very important
- Larger monitoring data sets
- Security

## Grid Monitoring

- Characteristics for Grid Monitoring:
- Scalable
- Dynamic
- Robust
- Flexible
- Should be integrated with other Grid Technologies and middleware (security infrastructure, resource brokers, schedulers, ...)
- Must Perform

## Grid Monitoring Architecture 

## Grid Monitoring Systems

-         Relational GMA  _[http://www.r-gma.org](http://www.r-gma.org/)_
- Monitoring and Discovery System (MDS)
-                             _[http://www.globus.org/mds](http://www.globus.org/mds)_
- MonALISA
-    Monitoring Agents in a Large Integrated Services Architecture
-                        _[http://monalisa.cacr.caltech.edu](http://monalisa.cacr.caltech.edu/)_

## Monitoring Tools

- NetLogger
-         Networked Application Logger
-         _[http://www-](http://www-didc.lbl.gov/NetLogger)__[didc.lbl.gov/NetLogger](http://www-didc.lbl.gov/NetLogger)_
- Network Weather Service
-                              _[http://](http://nws.cs.ucsb.edu/)__[nws.cs.ucsb.edu](http://nws.cs.ucsb.edu/)__[/](http://nws.cs.ucsb.edu/)_
- Ganglia
-                   _[http://](http://ganglia.sourceforge.net/)__[ganglia.sourceforge.net](http://ganglia.sourceforge.net/)_
- Nagios
-                      _[http://](http://www.nagios.org/)__[www.nagios.org](http://www.nagios.org/)_

## R-GMA

- R-GMA is used in the European Data Grid Project
- Based on a Relational Data Model (uses Individual RDBMS and SQL statements to provide the functionality outlined in GMA)
- Uses Java Servlets (tomcat). Moving to Web Services
- Can be used as a replacement to MDS (tools are provided to invoke MDS Info Providers)
- Nagios is used for graphs and notification
- Clients: R-GMA Browser (Java Graphical display Tool), command line tool (Python) and an API for programmatic access.

## R-GMA

## MDS - Overview

- _[http://www.globus.org/mds](http://www.globus.org/mds)_
- MDS provides directory services for Grids using the Globus Toolkit.
- Provides a mechanism for publishing and discovering resource stats and configuration info
- Based on OpenLDAP
- Decentralized and Scalable
- Security provided by combining GSI (Grid Security Infrastructure) with OpenLDAP ACLs

## MDS - Components

-  Information Providers
-  GRIS (Grid Resource Information Service)
-  GIIS (Grid Index Information Service)
- Clients

## MDS – Information Providers

- Provide resource info to GRIS.
-  Three Types of Information Providers
-      Core Information Providers
-      GRAM Reporters
-      Custom Information Providers
- The provided info must be in a format that GRIS understands.

## MDS – GRIS

- GRIS runs on each resource and provides resource specific info.
- GRIS invokes Info Providers to collect Resource data.
- Each GRIS supports multiple Info Providers
- Data gets cached for a period of time (Cachettl parameter)
- GRIS registers with one or more GIIS to form a hierarchy.

## MDS – GIIS

## MDS – Hierarchical GIIS

- _Registrar configuration (GIIS)_
- Every Registrar (GIIS) determines whether to accept incoming registration requests (grid-info-site-policy.conf )
- _Registrant (GRIS/GIIS)_
- Every Registrant determines which GIIS’s will register to (grid-info-resource-register.conf) and which providers will be available to send data to the GIIS’s this GRIS is registered (grid-info-resource-ldif.conf).
**regperiod**:
-         How often this GRIS will send a message to GIIS announcing its existence
**ttl**:
-         How long the registration info will be good for, before assuming that this GRIS is no longer available (typically ttl=2xregperiod)
**cachettl**:
-        How long info from this GRIS will be kept in cache.
**bindmethod**:
-        What method will be used for mutual authentication

## MDS – Clients

- MDS data can be accessed with a wide range of utilities:
_**Command Line Tools**_:
- grid-info-search: a grid enabled ldapsearch
_**Programmatically**_:
-    LDAP Client API for Java, Python and Perl
- Java uses the JNDI package for accessing LDAP directories.
- Java CoG uses it.
_**Various LDAP/Web Browsers**_
- The **Grid technology Repository** is a good place to look for MDS info providers and clients. _[http://gtr.globus.org](http://gtr.globus.org/)_

## GLUE Schema

- GLUE: Grid Laboratory for a Uniform Environment
- Goal: To provide interoperability between US and European Physics Grid Projects
- GLUE Schema: A common Schema used in describing and monitoring Grid Resources. The major components are: Computing Element (CE) a Storage Element (SE) and a Network Element (NE).
- Glue Schema can be implemented in LDAP, XML,SQL
- An MDS implementation of the GLUE Schema includes modified core Info Provider, modified GRAM Reporter and a Ganglia Interface for Cluster Info.

## MDS3

- GT3 is an OGSI implementation.
- A Grid Service is a Web Service + extra concepts and mechanisms defined by OGSI.
- A key concept, as far as monitoring is concerned, is the **serviceData**.
- serviceData is a structured collection of information that is associated with an instance of a Grid Service. Basically, is an in XML representation of its internal state.
- Each serviceData is composed of **servicaDataElement**s
- The status of a host is exposed as an SDE. This is similar to GRIS functionality in MDS2

## MDS3

- MDS3 supports both push and pull mechanisms to retrieve serviceData
- _Pull mechanism_: **findServiceData** operation (required by OGSI; send one query and gets one response)
- OGSI supports a _query type_: **queryByServiceNames**
- GT3 supports a query type that uses XPath (and working in supporting other query languages such as XQuery and XSLT)
- _Push mechanism_: subscribe to receive notification about serviceData (optional)

## MDS3

- The OGSI query type queryByServiceDataNames (and subscribeByServiceDataNames) both take SDEs as arguments.
- They are pretty inefficient though, in that they return entire Service Data Elements (which could by a large chunk of data).
- Globus defines a query type based on XPath. The input query takes a list of SDEs and an XPath Query. The output is the result of evaluating the XPath query against a set os SDEs.

## MDS3 -- XPath

- The primary purpose of XPath is to address parts of an XML document.
- XPath views an XML document as a tree made up of nodes. XPath is a language for picking nodes and sets of nodes out of this tree.
- XPath uses a compact, non-XML syntax to facilitate use of XPath within URIs and XML attribute values. The Syntax is similar to filesystem addressing.

## XPath Query Example

- //Host[@Name=“dc-user.isi.edu”]/ProcessorLoad
- First selects only the Host Elements
- From that subset, selects only those elements that Name=“dc-user.isi.edu”
- And finally from that subset, select only the ProcessorLoad Elements
- Output:
- <ProcessorLoad  Last1Min="00"  Last5Min="00"  Last15Min="00“ />

## <Cluster Name="pygar.isi.edu" UniqueID="pygar.isi.edu">

-   <SubCluster Name="pygar.isi.edu" UniqueID="pygar.isi.edu">
-     <Host Name="pygar.isi.edu" UniqueID="pygar.isi.edu">
- <Processor
-   Vendor=" GenuineIntel"  Model=" Intel(R) XEON(TM) CPU 2"  Version="15.2.4"
-   ClockSpeed="2193"  CacheL2="512"/>
- <MainMemory VirtualSize="2047" RAMSize="1004" RAMAvailable="119" VirtualAvailable="1716" />
- <OperatingSystem Name="Linux" Release="2.4.7-10"  Version="#1 Thu Sep 6 17:27:27 EDT 2001" />
- <FileSystem  Name="/“   Size="23510"  AvailableSpace="650"  Root="/"  Type="unavailable"  ReadOnly="false“ />
- <NetworkAdapter  Name="eth0"  IPAddress="128.9.72.46"  InboundIP="True"  OutboundIP="True"  MTU="1500"/>
- <ProcessorLoad  Last1Min="00"  Last5Min="00"  Last15Min="00“ />
-     </Host>
-   </SubCluster>
- </Cluster>

## MDS3

- In addition to findServiceData, OGSI provides additional support for handling service data:
- Support for **Xindice**: an XML database for persistant XML service data; You may reboot your services and service data can still be there.
**Aggregator mechanism**: acts as a notification sink. Takes service data from notifications and republishes under one service data. So, you can perform queries under this service data instead of having to query each individual service data.
**Provider mechanism**: plug in your Java or Unix scripts service data providers. Similar to MDS2 Info Providers.

## MDS3

- Putting together the functionality described in the previous slide you get something similar to MDS2 GIIS:
- Gather serviceData from other grid services
- Publish collection as one service
- Clients can query the index in the same way as they can query any other service data.

## Jini network technology is an open software architecture that enables the creation of network-centric solutions which are highly adaptive to change. 

- A Short Introduction to JINI

## JINI: A set of APIs and Network protocols that can help build and deploy distributed applications that are organized as federations of services.
Federation: a set of equal peers. There is no central controlling authority. 
Instead of a central authority, JINI provides a mechanism for clients and services to find each other: Lookup Service 

- Federation: a set of equal peers. There is no central controlling authority.
- Instead of a central authority, JINI provides a mechanism for clients and services to find each other: **Lookup Service**

- A Short Introduction to JINI

##  The Discovery protocol: Clients and Service Providers use the discovery protocol to find a Lookup Service
Once a Lookup Service has been located a ServiceRegistrar object is the first object that is sent over to the Service that registers or to the Client that searches for a service.
Two major methods in the ServiceRegistar register() and lookup()

- Once a Lookup Service has been located a **ServiceRegistrar** object is the first object that is sent over to the Service that registers or to the Client that searches for a service.
- Two major methods in the ServiceRegistar **register()** and **lookup()**

- A Short Introduction to JINI

##  Once the service provider has located a Lookup Service will create a ServiceItem object that will pass it as an argument to the register() method of ServiceRegistar.

package net.jini.core.lookup; 

public Class ServiceItem { 

   public ServiceID serviceID; 
   public java.lang.Object service; 
   public Entry[] attributeSets; 

   public ServiceItem(ServiceID serviceID, java.lang.Object service, Entry[] attrSets); 
} 

- package net.jini.core.lookup;
- public Class ServiceItem {
-    public ServiceID serviceID;
-    public java.lang.Object service;
-    public Entry[] attributeSets;
-    public ServiceItem(ServiceID serviceID, java.lang.Object service, Entry[] attrSets);
- }

- A Short Introduction to JINI

##  Once the client has located a Lookup Service will create a ServiceTemplate object that will pass it as an argument to a lookup() method of ServiceRegistar.

package net.jini.core.lookup;

public Class ServiceTemplate {
     public ServiceID serviceID;
     public java.lang.Class[] serviceTypes;
     public Entry[] attributeSetTemplates;

     ServiceTemplate(ServiceID serviceID, 
                     java.lang.Class[] serviceTypes, 
                     Entry[] attrSetTemplates);
}

- package net.jini.core.lookup;
- public Class ServiceTemplate {
-      public ServiceID serviceID;
-      public java.lang.Class[] serviceTypes;
-      public Entry[] attributeSetTemplates;
-      ServiceTemplate(ServiceID serviceID,
-                      java.lang.Class[] serviceTypes,
-                      Entry[] attrSetTemplates);
- }

- A Short Introduction to JINI

##  The Client looks for an object of a Class that implements a known Interface. 
What gets in return is either a Service object that will enable the client to implement the service locally or a service proxy that will invoke the service remotely (over RMI).
JINI raises the level of abstraction of distributed systems programming from the Network protocol level to the object interface level.

- What gets in return is either a Service object that will enable the client to implement the service locally or a service proxy that will invoke the service remotely (over RMI).
- JINI raises the level of abstraction of distributed systems programming from the Network protocol level to the object interface level.

- A Short Introduction to JINI

## http://monalisa.cacr.caltech.edu/

- _[http://monalisa.cacr.caltech.edu/](http://monalisa.cacr.caltech.edu/)_

## MonALISA

- A distributed monitoring service based on JINI/JAVA and WSDL/SOAP technologies that provides monitoring information from large and distributed systems to “higher level services” that require such information.
- It is truly dynamic:
- Discover all the “Farm Units” that make up a Group/Community
- Provide a notification mechanism to propagate configuration changes
-  Provide a Lease mechanism.
- It can integrate existing monitoring tools to collect parameters describing computational nodes, applications and network performance.

## MonALISA

- It Provides:
- Single farm values and details for each node that makes up the farm.
- Network parameters, connectivity values and traffic information.
- Real time data for subscribed listeners
- Historical data
- SNMP support and interfaces with other tools: Ganglia, MRTG, LSF, PBS, user defined scripts
- Active filters to process the data and provide customized information to other services.
- Dynamic proxies (WSDL) so that clients can access the data in a flexible way.
- Authentication and a secure GUI to configure and administer the monitoring service.
- Global monitoring repositories for a group/community.
- Access to the monitoring information from mobile phones using WAP.

## MonALISA

- Data Collection

## MonALISA

- The Service System

## Repositories

## Web Services

## Global Client

## Regional Center GUI Client

##  Global Views – Filter Agents

## MonALISA

- Deployed two MonALISA Services: one in ITD (for testing and learning purposes) and one for STAR (about a month ago). Another one will soon be deployed in PDSF.
- Documentation: _[http://www.star.bnl.gov/STAR/comp/Grid/Monitoring/](http://www.star.bnl.gov/STAR/comp/Grid/Monitoring/)_
- Developed custom Monitoring Modules (LSFjobs).
- Comparison of monitored values between MDS and MonALISA
- Setup a STAR Group in the Lookup Services.
- Started looking into setting up a private Lookup service to be used exclusively for STAR (firewall issues).
- Setting up a Web Repository for STAR is under way. It exists but it needs to be reconfigured.
- Looking into possible secure access to monitored values.

## Slide 45