# Presentation: 503037

## Grid Monitoring and Accounting

- Dave Kant
- CCLRC e-Science Centre, UK
- HEPiX at Brookhaven
- 18th – 22nd Oct 2004

## Monitoring the Grid is a Challenge      

**Monitoring the Grid is a Challenge      **

- Number of participating sites is growing every day:
- August  2003 => 12 sites ;
- October 2004 => 83 sites ;  8000 CPUs; 96TB Disk

**Grid Operations Centre**
- Monitor the operational status of sites;
- Fault detection
- Problem Management
- Identify problems; escalate; track;

## Distributed GOC

- LCG sites are distributed all over the globe
- There has been a coordinated effort to develop and integrate a variety of monitoring tools from CERN, CCLRC (UK), GridPP, INFN (Italy) and Taiwan

## We have only fragmentary information about the services that sites are running.
We don’t know what RBs/SEs/Sites the VOs are using for data challenges.
We don’t know what the core services are and who is running them.
We don’t have a toolkit to test specific core services.
We have to concentrate on functional behaviour of services e.g If an RB sends your job to a CE, then we must assume the RB is working fine. Is this the only test of a RB?
Not all the tests that we perform are effective at finding problems so we must take tests written by the experts and integrate them into GOC monitoring.
We must develop tests which simulate the life cycle of real applications in a Grid environment.
There are lots of monitoring tools available, so we need to bring them together. 
Do we spend time investigating new tools, or make the ones which we already have better?
...and probably lots more!

- We don’t know what RBs/SEs/Sites the VOs are using for data challenges.
- We don’t know what the core services are and who is running them.
- We don’t have a toolkit to test specific core services.
- We have to concentrate on functional behaviour of services e.g If an RB sends your job to a CE, then we must assume the RB is working fine. Is this the only test of a RB?
- Not all the tests that we perform are effective at finding problems so we must take tests written by the experts and integrate them into GOC monitoring.
- We must develop tests which simulate the life cycle of real applications in a Grid environment.
- There are lots of monitoring tools available, so we need to bring them together.
- Do we spend time investigating new tools, or make the ones which we already have better?
- ...and probably lots more!

**Monitoring Challenges**

**Notes:**
Useful operations information.

##       There are many tools which can be used to monitor sites in a distributed environment. Many developed from other projects e.g. EDG, DataTAG, GridPP including the open source community. 

MAPCENTER   http://mapcenter.in2p3.fr/
GPPMON           http://goc.grid-support.ac.uk/
GRIDICE           http://grid-ice.esc.rl.ac.uk
NAGIOS            http://www.nagios.org/
GIIS Monitor     http://goc.grid.sinica.edu.tw/gstat/
Ganglia	  http://ganglia.sourceforge.net/

- MAPCENTER   _[http://mapcenter.in2p3.fr/](http://mapcenter.in2p3.fr/)_
- GPPMON           _[http://](http://goc.grid-support.ac.uk/)__[goc](http://goc.grid-support.ac.uk/)__[.grid-support.ac.](http://goc.grid-support.ac.uk/)__[uk](http://goc.grid-support.ac.uk/)__[/](http://goc.grid-support.ac.uk/)_
- GRIDICE           _[http://grid-ice.esc.](http://grid-ice.esc.rl.ac.uk/)__[rl](http://grid-ice.esc.rl.ac.uk/)__[.ac.](http://grid-ice.esc.rl.ac.uk/)__[uk](http://grid-ice.esc.rl.ac.uk/)_
- NAGIOS            http://www.nagios.org/
- GIIS Monitor     http://goc.grid.sinica.edu.tw/gstat/
- Ganglia	  http://ganglia.sourceforge.net/

**Monitoring Services**

- By no means a complete list!

**Notes:**
I shall talk about each of these.

## GOC Configuration Database

**GOC **
**GridSite **
**MySQL**

**Resource Centre**
**Resources & Site Information**
**EDG, LCG-1, LCG-2,** ...

- ce

- se

- bdii

- rb

-  **Monitoring Services**
-  Operations Maps
-  Configure other Tools
-    Mapcenter:30 sites = 500 lines in config file
-    Nagios:12 configuration scripts  with dependencies
-  Organisation Structures
-  Secure services
- - Site News
- - Self Certification

- Secure Database Management via HTTPS / X.509
- People, Contact Information, Resources
- Scheduled Maintenance

- RC

**SQL**

**https**

**SERVER**

## Operations Map – Job Submission Tests

**Operations Map – Job Submission Tests**

-       **GPPMON**
- Displays the results of tests against sites.
- Test: Job Submission
- Job is a simple test of the grid middleware components e.g. Gatekeeper service, RB service,  and the Information System  via JDL requirements.
- Site resources can be taken from different sources:

-  BDII          List of sites belonging to a grid e.g. Production, Development, LCG, GILDA ...
-  GOC DB   Tailor the information for different customers e.g. regional monitoring for EGEE

## Operations Map – Certificate Lifetime

**Operations Map – Certificate Lifetime**

-        **GPPMON**
- Displays the results of tests against sites.
- Test:Certificate Lifetime
- Many grid services require a valid certificate for security.
- Can be used to provide advanced warning to sites.

## GRIDICE – Architecture

- Developed by the INFN-GRID Team http://infnforge.cnaf.infn.it/gridice

**Notes:**

Measurement service – Machines such as Worker Nodes run Monitoring agents.
These agents probe “core processes” and gather related metrics e.g memory, cpu, # of process instances

Publisher service        - At each site, the monitoring information is collected and stored in a database by the fmonServer

Query MDS to a list of get resources; then query resources to collect fmonServer information

[MDS works on a pull and can be quite slow]

## GRIDICE – Global View

- Display shows the processes belonging to the Broker service. Problems are flagged

- List of Sites

- Resource Usage CPU#, Load, Storage, Job Info

**Notes:**
Web Interface shows what you might see if you want an overall Global view of the grid.
Participating sites, resource usage, total CPU and disk.
The information can be viewed in different ways: for example Geographically or for each VO  

## GRIDICE – Expert View

- Display shows the processes belonging to the Broker service. Problems are flagged

- Node

- Processes

**Notes:**
Expert view shows what you might see if you drill down to individual machines at a site.
For example a machine running a broker service: You see all the processes that must run in order for the service to function properly – Process Names, number of process instances.

## Ganglia Monitoring 

- http://gridpp.ac.uk/ganglia
- Can use Ganglia to monitor a cluster

**RAL Tier-1 Centre**
**LCG PBS Server displays Job status for each VO**

**Notes:**
Ganglia is a scalable distributed monitoring system for clusters and grids which uses RRDTool for data storage and visualisation. Its relatively easy to install and you get a lot for little effort.
One of its strengths is that it can federate clusters together.

## Federating Cluster Information

- Can also use Ganglia to monitor clusters of clusters

**Notes:**
Separate and distinct clusters federated together.
Ganglia provides a wealth of information, much of it low level, that can be useful for operations.

## GIIS Monitor

- Developed by MinTsai (GOC Taipei)
- Tool to display and check information published by the site GIIS
- http://goc.grid.sinica.edu.tw/gstat/

**Notes:**
GIIS Monitor which has been developed by the GOC based in Taipei
It’s a tool to monitor the grid information system.
The primary goal of the application is to detect faults, perform sanity checks and display useful data.
Provides an overview of the current grid status and you can drill down to get more information. 

## Regional Monitoring

- EGEE is made up of regions.
- Each region contains many computing centres.
- Regional Operational Centres is a focus for operations.

**Notes:**
One of the ways in which we can deal with operations and the complexity of managing a grid is to divide the task into smaller pieces. For example in EGEE: GOC can provide monitoring services that are tailored to each region.

## Regional Monitoring Maps

- Provide ROCs with a package to monitor the resources in the region
- Tailored Monitoring
- GUI to automate site locations on the map
-  Hierarchical view of Resources
- Example UK Particle Physics GridPP

**Notes:**
Organisational Structures such as the Regional Operational Centres belonging to EGGE, can be described in the GOC database using Materialized Path encoding.   

## Site Certification Service

- In terms of middleware, the installation and configuration of a site is quite a complicated procedure.
- When there is a new release, sites don’t upgrade at the same time
- Some upgrades don’t always go smoothly
- Unexpected things happen (who turned of the power?)
- Day-to-day problems; robustness of service under load?
- Its necessary to actively hunt for problems
- Site certification testing is by CERN deployment team on a daily basis. First step toward providing this service involves running a series of replica manager tests which register files onto the grid, move them around, delete them; and 3rd party copies from remote SE.
- Unlike the simple job submission tests described earlier, these are more heavy weight and attempt simulate the life cycle of real applications.

## Certification Test Results

- Individual Test Results

**Notes:**
There is a quite a bit of information here. 
Each row is a site identified by the GIIS siteName with its test results.
Large Matrix of data
One of the difficulties of having too much information, is that it can be difficult to find the information you need. Its also quite detailed ... again for the expert!

## Syndication of Monitoring Information

**Aggregator RSSReader (Windows Client)**

- GOC generates RSS feeds which clients can pull using an RSS aggregator.
- Aggregators available for Linux, Windows and MacOS
- The aggregator shown displays test results for the RAL CE. These results are archived and popup on the desktop when the feed is updated.

**Notes:**
Clearly, there is a lot of monitoring information.
Need a way to syndicate this information to the right people 

Really Simple Syndication (RSS) is an XML schema that is used by many organisations to syndicate content
Slashdot, Nature
Client Pull model: GOC creates RSS formatted documents, clients pull these feeds which render them in html.

Can tailor feeds to communities:
per site, per ROC, whole Grid, specific test

The screenshot shows the layout of the results that a site administrator would see subscribing to such a feed.
There are plenty of open source RSS clients out there to render the feeds into html.  

## Real Time Grid Monitor

- http://www.hep.ph.ic.ac.uk/e-science/projects/demo/index.html

- A Visualisation tool to track jobs currently running on the grid.
- Applet queries the logging and bookkeeping service to get information about grid jobs.

- Why are jobs failing?
- Why are jobs queued at sites while others are empty?

**Notes:**
Visualisation tool developed by GridPP at Imperial College.
The monitor works by querying the RB Logging and Bookkeeping database for job information.
Because the L&B service is continually being updated, the tool shows jobs flowing from the RB to a site for processing or returning back to the source once completed.

Tool is useful to get a global picture of trends and quickly identify potential problems such as “job pile-up”, and it help to publicise the grid at conferences to non experts.

[ Applet queries files not older than 6 hours. Long running jobs don’t show up]

## Monitoring Paradigm

- GOC Services collect information and publish into an archiver.
- ROC/CIC Services provide a means for the community to interact with this information on-demand. GOC provides services tailored to the requirements of the community.
- Uses infrastructure (R-GMA, database and web portal developed for accounting)

**Notes:**
Thus far I’ve described lots of tools.

They have a few things in common:
   - they have all been developed independently
   - all the information which they generate is hidden away or difficult to access
   - limited interfaces: the data can only be accessed in specific ways e.g. web page

Therefore, its difficult to build “on-demand” services to allow communities “Players” to interact with the data.

What sort of services are we thinking about?

Examples include 
Job Accounting service : to allow an Organisation to compare resources usage for each VO
Certification Testing service: Secure service to allow a site administrator to run the certification test suite against their site through a RB of their choice?

The idea is for the services to collect information and put it into a common repository such as an RGMA Archiver. In this way, the information can be shared and accessible to all.

Services (EGEE parlance: ROC and CIC services)  munch the data and present it to the community.
Example: The current limitation with GIIS is that its hard to drill down to the information you want e.g How much CPU in GridPP today? How much disk in the UKI ROC? The new paradigm solves this problem by allowing the data to be aggregated in different ways.

Each community has its own requirements so we need to tailor services.

## GOC UseCase Accounting

- An accounting package for LCG has been developed by the GOC at RAL
- There are two main parts
- the accounting data-gathering infrastructure based on R-GMA which brings the data to a central point
- a web portal to allow on-demand reports for a variety of players.

## Accounting Flow Diagram

**Batch Log**

**GK Log**

**messages **

- CE

**Site GIIS**

- LCG SITE

- LCG SITE

- Data Sources

- MON

- RGMA

**Notes:**
Picture shows the flow of accounting data from collection at source to the presentation of this information to the community via the GOC. This system uses R-GMA – a relational implementation of the GGF Grid Monitoring Architecture and serves as the transport layer for sending information from once place to another.

At each site, filters process data from log files and write to database tables on the site MON via JDBC/MySQL.
	GK filter to get DN (a requirement for grid jobs)
	Event filter to get job information
	Messages to map Grid users to Local jobs 
	GIIS filter to get specINTs and specFLOATs 

Data is joined to build accounting records. 

Accounting records sent to GOC via RGMA.

GOC Collect records from the sites (1 records per grid job)

GOC processes records to create summary tables from which the on-demand services run (ROC views of data/ VO views etc)

## GOC Accounting Services

- http://goc.grid-support.ac.uk/gridsite/accounting/index.html

- BaseCpuSeconds Aggregated across EGEE

- Simple interface to customise views of data: VO, time frame and Region (default = EGEE)

- Each Region, per VO, per Month

- On Demand Services to EGEE Community

- Other Distributions
- Normalised CPU
- # Jobs

**Notes:**
GOC provides “services” to the  EGEE community.
Use case 1: Accounting service
Accounting data fed into RGMA database
MARKUS/PIOTR/LAURENCE/MIN WORRY ABOUT GETTING MONITORING INFORMATION INTO COMMON DATASTORE
GOC SHOULD WORRY ABOUT PRESENTING THIS INFORMATION
                NO DUPLICATION OF EFFORT !
Use case 2: Monitoring/Testing service
NB: Monitoring/Testing information from Piotr/Markus via R-GMA database.
Philosophy: Piotr worries about what kind of tests to run; GOC worries about how to present the information to the community.
Use case 3: gstat
Show disk/cpu/ # of jobs running/waiting per site/ region etc.
Use case 4: Job monitoring service
Hourly job submission tests of sites by GOC

## Future Plans

- Extend the ideas developed in the accounting useCase  to all the tools that have been described.
- Want to move toward a Service Orientated Architecture model and provide the community with a direct interface into the monitoring.

## Summary

- Accounting Information gathering infrastructure has been developed
- It has been through the C&T cycle and should be deployed in the next release.
- A web portal for display of this information has been developed (work in progress)
- This is an EGEE deliverable (DSA1.3)
- The display infrastructure can be deployed for other information (e.g monitoring)

## Summary

- Since August 2003, the LCG GOC has been working to understand the problems of running a large scale distributed grid.
- Setup a distributed GOC and deployed tools to help understand the issues.
- Development towards on-demand services to provide the community with up-to-date information, aggregated at different levels.
- Development of Visualisation tools to enhance our understanding of the grid.