# Presentation: 383293

## GridMonitor: Integration of Large Scale Facility Monitoring With MDS

** ****Richard Baker, Antonio Chan**
**Jason Smith, Dantong Yu**
**USATLAS/RHIC Computing Facility**
**Brookhaven National Lab**

## Outline

- Requirements
- System Framework, Structure  and Characteristics
- I: Ganglia and Its Information Provider
- II: Relational Database Based Archiving and Its Information Provider
- Gridview and GStat, Front End System:
- http://heppc1.uta.edu/atlas/grid-status/mds.gremlin.usatlas.bnl.gov.html
- Current Status and Future Works

- R. Baker                        GGF6 Large Site AA

- October 16, 2002

## Requirements

- Requirements :
- Modularity and Extensibility: Make Use of Existing Monitoring Pieces
- Flexibility: Adjustable to the Dynamics of the Monitored Systems
- Overhead: Non-intrusive
- Scalability
- Security, Consistency, Inter-operability, Etc-bility

- R. Baker                        GGF6 Large Site AA

- October 16, 2002

## What Need to Be Monitored

- Linux Farm Monitoring
- Description
- About 1100 Dual CPU LINUX Nodes
- Performance Data Must Be Summarized for Advertising to Grid
- Performance Events Required:
- Configuration Information
- Status Information: CPU Load, (1, 5, 10, 15), Memory Load, Disk Load, and Network Load
- Example Usage: A Resource Broker Might Ask the Availability of Linux Farm System Resources in Order to Plan the Efficient Execution of Tasks

- R. Baker                        GGF6 Large Site AA

- October 16, 2002

## More...

- Network Monitoring:
- Description:
- 8 USATLAS Testbeds
- Publish the Connectivity of These Test-beds, Monitor the Healthiness of the USATLAS Network
- Archived Performance Data Can Be Used to Predict the Network Behavior a User Can Choose the Source and Destination for File Replication
- Performance Events Required:
- Bandwidth, Delay ( Round Trip Time), Trace Route

- R. Baker                        GGF6 Large Site AA

- October 16, 2002

## Monitoring Framework

- R. Baker                        GGF6 Large Site AA

- October 16, 2002

## Monitoring System Components

- Four Tier Structure
- Sensors
- Host: Ganglia, Top, /Proc and lsf Host Load
- Archive System (Database System)
- Round Robin Database (RRD)
- Relational Database: UNIXodbc+myodbc+mysql Database
- Information Providers
- Monitoring and Discovery Service (Mds2.2), GLUE Schema, Customized Ganglia Client Tool Reporting the Lastest Monitoring Data, and Database Client Tools Reporting the Summary Information
- Front-end Browsing System
- Gridview, GStat (Grid Visualization Tool Developed at
-    Univ. of Texas at Arlington)

- R. Baker                        GGF6 Large Site AA

- October 16, 2002

## Advantages

- Information Provider Provides Cache for the Newest Value From the Mysql Database
- Non-intrusiveness: Information Provider Can Eliminate the User Random Accesses to the Database Server
- Scalability Can Be Significantly Increased
- 1000 Linux Nodes Are Being Monitored
- Network Connectivity of Eight Usatlas Testbeds: Each Site Monitoring the Paths From Itself to the Other Seven. Network Topology and Traffic Can Be Easily Constructed
- Flexibility:
- Independent on Sensors. Many Sensors Can Be Easily Plugged As Long It Has Well Defined Protocol and API: We Could Switch Among Ganglia, top, /proc
- Archive System Is Independent to Underlying Database
-  Can Be rdbms, Oracle, Mysql, Sybase, Informix, Flat Files, Objectivity As Long the Odbc Drivers Is Available

- R. Baker                        GGF6 Large Site AA

- October 16, 2002

## I: Ganglia Monitoring with MDS

- Ganglia Information Provider
- Front-end: Glue-schema Http://www.cnaf.Infn.It/~sergio/datatag/glue/
- Back-end: XML

**Cluster A Multicast Channel**

**Cluster A Multicast Channel**

**Gmond**

**Gmond**

**Gmond**

**Gmond**

**Gmond**

**Gmond**

**Gmond**

**Gmond**

**Gmond**

**Gmond**

**Gmond**

**Gmond**

**XML**

**XML**

**Gmetad **
**(filtered)**

**Gmetad **
**(filtered)**

**...**

- ?

- MDS

- Ganglia IP

- XML

- GLUE

**Layered**
**Gmetad **

- R. Baker                        GGF6 Large Site AA

- October 16, 2002

## I: Ganglia Monitoring with MDS

**gremlin % grid-info-search -x -h spider.usatlas.bnl.gov -s one**
**# ATLAS Linux Cluster, local, grid**
**dn: cl=ATLAS Linux Cluster, mds-vo-name=local, o=grid**
**objectClass: GlueClusterTop**
**objectClass: GlueCluster**
**GlueClusterName: ATLAS Linux Cluster**
**GlueClusterUniqueID: ATLAS_Linux_Cluster-RCF_and_ACF_Linux_Farm_Group**
**GlueClusterService: compute**
** ****# PHOBOS CAS Linux Cluster, local, grid**
**dn: cl=PHOBOS CAS Linux Cluster, mds-vo-name=local, o=grid**
**objectClass: GlueClusterTop**
**objectClass: GlueCluster**
**GlueClusterName: PHOBOS CAS Linux Cluster**
**GlueClusterUniqueID: PHOBOS_CAS_Linux_Cluster-RCF_and_ACF_Linux_Farm_Group**
**GlueClusterService: compute**
** ****# STAR CAS Linux Cluster, local, grid**
**dn: cl=STAR CAS Linux Cluster, mds-vo-name=local, o=grid**
**objectClass: GlueClusterTop**
**objectClass: GlueCluster**
**GlueClusterName: STAR CAS Linux Cluster**
**GlueClusterUniqueID: STAR_CAS_Linux_Cluster-RCF_and_ACF_Linux_Farm_Group**
**GlueClusterService: compute**

- R. Baker                        GGF6 Large Site AA

- October 16, 2002

## II: Farm Monitoring

- Linux Farm Is Divided Into Different Sub-clusters Based on Site Policy, Different Experiments, OS and Version, CPU Speed. A Sub-cluster Contains the Host With the Same Configuration
- Bnl Atlas Farm Is Partitioned Into Four Subclusters: Cpu400mhz, Cpu700hz, Cpu1ghz, Cpu1.4ghz and CPU 2.4GHZ
- The Status Information of a Sub-cluster Is Summarized From All Nodes in This Sub-cluster
- Grid Resource Broker Schedules in the Level of Farm Sub-clusters

- R. Baker                        GGF6 Large Site AA

- October 16, 2002

## Information Schema (Linux Farm Monitoring)

- Queue-Info:
- objectclass ( 1.3.6.1.4.1.3536.2.6.0.0.0.0    NAME 'Queue-Info'    SUP  'Mds'    STRUCTURAL    MUST ( MdsQueueNumberOfCpu  $           MdsQueueSpeed        $           MdsQueueAverageLoad  $           MdsQueueAverageUserPercent   $           MdsQueueAverageSysPercent ))
- Need to be replaced by GLUB-schema

- R. Baker                        GGF6 Large Site AA

- October 16, 2002

## Backend Data Structure

- Node Status Information
- mysql> describe node_load;
- +-------------+-------------------------+------+-----  +---------+---------------------+
- | Field           |Type                    | Null | Key   |Default| Extra                  |
- +-------------+------------------------+------+--------+--------+----------------------+
- | load_index | int(10) unsigned |          | PRI   | NULL| auto_increment |
- | sampletime| timestamp(14)    | YES  | MUL | NULL|                           |
- | machine_id| varchar(31)         |          |          |          |                           |
- | owner        | varchar(8)            |          |          |          |                           |
- | load_5       | float(10,2)            |          |          | 0.00  |                           |
- | user_cpu   | float(10,2)            |          |          | 0.00  |                           |
- | sys_cpu     | float(10,2)            |          |          | 0.00  |                           |
- +---------------+-----------------------+-------+--------+-------+---------------------+

- R. Baker                        GGF6 Large Site AA

- October 16, 2002

## Information Provider (Linux Farm Monitoring)

- # generate Farm information every 10 minutesdn: MdsFarmQueueName=1000,  MdsHostNodeDomainName=usatlas.bnl.gov, Mds-Host-hn=gremlin.usatlas.bnl.gov, Mds-Vo-name=local, o=gridobjectclass: GlobusTopobjectclass: GlobusActiveObjectobjectclass: GlobusActiveSearchtype: execpath: /usr/local/globus-new/customizebase: mds-farm-batch-info.plargs: -dn MdsFarmQueueName=1000,MdsHostNodeDomainName=usatlas.bnl.gov,Mds-Host-hn=gremlin.usatlas.bnl.gov,Mds-Vo-name=local,o=grid -ttl 900cachetime: 600timelimit: 20sizelimit: 400

- R. Baker                        GGF6 Large Site AA

- October 16, 2002

## Observation from Grid-View

- R. Baker                        GGF6 Large Site AA

- October 16, 2002

## Current Status and Future Work

- Current Status:
- Sensors & Local Monitoring Tools Put Less Than 1 Percent CPU Load: Non-intrusive
- Improved the Ganglia Information Provider, It Can Obtain Information From Both Gmond and Gmetad
- Multiple & Hierarchical Clusters Are Supported
- Future Works
- Merge the Ganglia RRD Information Provider and the Archive DB Information Provider
- Work With the Ganglia Team and Glue-schema, Help to Define Requirements for What Information Be Monitoring for Job Scheduling
- Automate the Mapping From Xml to ldif (via Glue Schema?), Provide Flexibility
- Continue to Optimize The Information Provider to Deliver Data Faster
- Scalability Test

- R. Baker                        GGF6 Large Site AA

- October 16, 2002