# Presentation: 517407

## Lecture 7Building, Monitoring and Maintaining a Grid

- Pradeep Padala
- University of Florida
- ppadala@cise.ufl.edu
-                  Grid Summer Workshop
-                        June 21-25, 2004

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Credit Where Credit Is Due

- Slides from Jorge Rodriguez
- One slide from Richard Cavanaugh
- Thanks to the input from Rob Gardner

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Outline

- Why do you want to build a grid?
- What are the issues involved in building in a grid?
- Monitoring the health of a grid
- Maintaining a robust and reliable grid
- Expanding a grid
- A Sample Grid (Grid3) and Details of its operations
- SC’03 demo – showing the complexity involved in building, using, maintaining and monitoring the grid

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Why do you want to build a grid?

- Different perspectives
- User: I want to run my scientific application on the grid so that I can get results in 10 hours instead of 10 days
- Organization: Our next big experiment will generate tera-bytes of data and we want to distribute, share and analyze the data
- Organization: We want to tap into the existing grids and share resources

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Why grid? User perspective

- So, you need
- More CPU cycles
- More disk space
- More bandwidth
- All of the above
- Do you really need a grid for the above?
- A CPU cycle stealer, A simple Database or SRM (Storage Resource Management) system might do the trick for you

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Why grid? User perspective

- Your application is complex. Requires
- A lot of resources
- Reservation of resources at a particular time
- Monitoring of status of the submitted jobs to multiple sites
- Storage that is not easily available at a single place

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Why grid? Organizational perspective

- Federation of scientists – distributing, sharing and analyzing data
- Tapping into existing grids
- Cost-effective: A grid can be built from commodity software and hardware without spending millions on the next super duper computer.
- Reliability: If a site fails, we can simply move our jobs to another site (this can be seen as a user perspective as well)

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Broad Division of Grids

- Before, we plunge into building a grid, let’s classify them in an easy-to-understand manner
- Many confusing names and categorizations
- A good way to characterize grids
- Data Grids
- 	Managing and manipulating large amounts of data. Main objective is to share large amounts of data that is otherwise impossible with out the grid
- Compute Grids
-  	For compute-intensive tasks. Emphasis is on the federation of CPU cycles and distribution of compute intensive tasks
- There is no consensus on these categorization and it only aids in understanding the requirements

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Building a Grid - Issues

- Infrastructure
- Network
- CPU
- Disk Space
- Deciding on the kind of hardware
- Usually, Grids are built with existing infrastructure
- Software
- Globus, Condor, VDT ...
- Packaging
- Deciding on the operating system, Package versions. Linux is the most popular OS for building grids
- Standards !!!

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Building a Grid - Issues

- Policies
- Security
- Certificates
- Authorization mechanisms
- Accounting
- Configuration
- One of the most difficult things
- Configuring various pieces of software
- Customization
- Monitoring
- Monitoring your jobs
- Monitoring the health of a grid
- Some metrics: Load average, Number of jobs, Network delay ...
- Maintaining

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## So, you still want a grid, 

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Building blocks

- Animation showing different pillars of a grid. Blocks with names information mgmt, resoruce mgmt ... and then software blocks like MDS, GRAM, GridFTP ...

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Hardware

- You don’t need specific hardware to build a grid, fortunately
- You can build a grid out of existing commodity hardware. A cluster of Dell PCs might (will) work
- But (that’s a big but), you should consider a few questions
- Can your machine handle the load of a CPU intensive job for days?
- Can the gatekeeper machine handle the load?
- Failovers
- We will see some details of the hardware used in Grid3 later

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Choosing the software

- Interoperability
- Ease of use
- Ease of configuration
- Development groups
- Maintenance

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Starting from Scratch 

- Buy a cluster of PCs
- Download and Install Linux
- Download Globus packages
- Packages are available for each component
- Install and configure them
- Get and install certificates for hosts and users
- Assign a gatekeeper and start submitting jobs
- Easy, isn’t it?
- Unfortunately, it’s pretty difficult to configure and maintain such a grid
- Multitude of configuration files
- Technology overload

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Using existing grid packages

- VDT (Virtual Data Toolkit)
- Ensemble of grid middleware
- It’s as easy as typing the following command on your command line
- pacman -get VDT:VDT
- source setup.sh
- Grid3 Package
- Built on top of VDT
- Provides a particular configuration of the VDT to work in the Grid3 environment
- Provides additional packages needed only by the Grid3 environments

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Enter pacman (package manager) !

- One of the most useful grid packages
- A tool for fetching, installing and managing software pacakges
- You can use it to install, configure and manage your applications as well
- We will see an example in the exercise

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## An example pacman file

**  **

- Pacman helps you in fetching, installing and configuring software packages effortlessly
- .pacman file is similar to a Makefile.

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Configuration

- Most difficult part of building a grid
- VDT is great but some of the software packages require extensive configuration (I had experience with RLS configuration for the SC’03 demo)
- Need to understand the technology involved
- Many complex software packages. Each with its own quirks
- Use an existing configuration package (Grid3, any more? ...)

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## A Sample Configuration procedure after you install Globus packages

- Animation or flowchart showing the steps. Some thing like. Get certs, update gridmap file, start services ...

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Monitoring a Grid

- Why do you need to monitor the grid?
- To find the current status so that you can submit your jobs to the most reliable site
- To find the most suitable site for your jobs
- To predict the usage patterns for a site
- Grid Monitoring Software
- Monalisa
- Ganglia
- Many others GridCat (Grid3), GridIce (LCG), Inca (TeraGrid)

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Maintaining a Grid

- Keeping up with the latest technologies
- New software packages
- Web and Grid Services
- New paradigm
- Security updates
- User management
- Certificates
- User addition
- Accounting (currently, no easy way of doing this)
- Site maintenance

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## A Sample Existing Grid: Grid3

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## What is Grid2003/Grid3?

- International Data Grid with dozens of sites
- Serving applications across various disciplines HEP experiments (LHC, BTeV) Bio-chemical, CS demonstrators...
- Currently over 2000 CPUS available for use by over 100 users
- A peak throughput of 1100 concurrent jobs with a completion efficiency of approximately 75%

- Note: Grid2003 refers to the initial project from 8/2003 – 12/2003
-           Grid3 refers to the persistent grid infrastructure

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Grid3 Organization

- Stakeholders:
- US LHC Software and Computing Projects
- US ATLAS, US CMS
- Grid projects (iVDGL, PPDG, GriPhyN)
- CS groups, VDT team, iGOC
- GriPhyN experiments
- LIGO, SDSS as well as ATLAS and CMS
- New collaborators
- Vanderbilt BTeV (Fermilab) Group
- Argonne computational biology group
- U Buffalo chemical structure

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Contributors

- Boston University
- Caltech
- Hampton University
- Harvard University
- Indiana University
- Johns Hopkins University
- Vanderbilt University
- University of Oklahoma
- University of Chicago
- University of Florida
- University of Michigan
- University at Buffalo

- Argonne National Laboratory
- Brookhaven National Laboratory
- Fermi National Accelerator Laboratory
- Kyungpook National University
- Lawrence Berkeley National Laboratory
- University of California San Diego
- University of New Mexico
- University of Southern California-ISI
- University of Texas, Arlington
- University of Wisconsin-Madison
- University of Wisconsin-Milwaukee

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Contributors 

- _Argonne National Laboratory_: Jerry Gieraltowski, Scott Gose, Natalia Maltsev, Ed May, Alex Rodriguez, Dinanath Sulakhe, _Boston University__:_ Jim Shank, Saul Youssef, _Brookhaven National Laboratory__:_ David Adams, Rich Baker, Wensheng Deng, Jason Smith, Dantong Yu, _Caltech__:_ Iosif Legrand, Suresh Singh, Conrad Steenberg, Yang Xia, _Fermi National Accelerator Laboratory__:_ Anzar Afaq, Eileen Berman, James Annis, Lothar Bauerdick, Michael Ernst, Ian Fisk, Lisa Giacchetti, Greg Graham, Anne Heavey, Joe Kaiser, Nickolai Kuropatkin, **Ruth Pordes***, Vijay Sekhri, John Weigand,  Yujun Wu, _Hampton University__:_ Keith Baker, Lawrence Sorrillo, _Harvard University:_ John Huth, _Indiana University:_ Matt Allen, Leigh Grundhoefer, John Hicks, Fred Luehring, Steve Peck, Rob Quick, Stephen Simms, _Johns Hopkins University:_ George Fekete, Jan vandenBerg, _Kyungpook National University/KISTI:_ Kihyeon Cho, Kihwan Kwon, Dongchul Son, Hyoungwoo Park, _Lawrence Berkeley National Laboratory:_ Shane Canon, Jason Lee, Doug Olson, Iowa Sakrejda, Brian Tierney, _University at Buffalo:_ Mark Green, Russ Miller, _University of California San Diego:_ James Letts, Terrence Martin, _University of Chicago:_ David Bury, Catalin Dumitrescu, Daniel Engh, Ian Foster, **Robert Gardner***, Marco Mambelli, Yuri Smirnov, Jens Voeckler, Mike Wilde, Yong Zhao, Xin Zhao, _University of Florida:_ Paul Avery, Richard Cavanaugh, Bockjoo Kim, Craig Prescott, Jorge L. Rodriguez, Andrew Zahn, _University of Michigan:_ Shawn McKee, _University of New Mexico:_ Christopher T. Jordan, James E. Prewett, Timothy L. Thomas, _University of Oklahoma:_ Horst Severini, _University of Southern California:_ Ben Clifford, Ewa Deelman, Larry Flon, Carl Kesselman, Gaurang Mehta, Nosa Olomu, Karan Vahi, _University of Texas, Arlington:_ Kaushik De, Patrick McGuigan, Mark Sosebee, _University of Wisconsin-Madison:_ Dan Bradley, Peter Couvares, Alan De Smet, Carey Kireyev,  Erik Paulson, Alain Roy, _University of Wisconsin-Milwaukee:_ Scott Koranda, Brian Moe, _Vanderbilt University:_ Bobby Brown, Paul Sheldon

- * Team Leads

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Grid3 Services 

- Software packaging Service (pacman)
- Virtual Data Toolkit (VDT)
- Additional middleware configuration packages
- Monitoring Services
- GridCat
- MonALISA
- ganglia
- Metrics Data Viewer
- ACDC Job Monitor
- User Authentication Service
- Virtual Organization Management Service (VOMS)
- Grid3 Operations
- The international Grid Operations Center (iGOC)

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Grid3 Packaging

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Grid Packaging Service

- Packaging is the key to success!
- Automation in software installation greatly improves reliability of software deployments
- Pacman package manager is used in Grid3
- Complete installation and site configuration is simplified to a single command:
- In reality it takes a little more work. However...

- % pacman –get iVDGL:Grid3

- ref. pacman --- _[http://physics.](http://physics.bu.edu/~youssef/pacman)__[bu](http://physics.bu.edu/~youssef/pacman)__[.](http://physics.bu.edu/~youssef/pacman)__[edu](http://physics.bu.edu/~youssef/pacman)__[/~](http://physics.bu.edu/~youssef/pacman)__[youssef](http://physics.bu.edu/~youssef/pacman)__[/](http://physics.bu.edu/~youssef/pacman)__[pacman](http://physics.bu.edu/~youssef/pacman)__[/](http://physics.bu.edu/~youssef/pacman)_

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## The VDT packages vers 1.1.14

-  Globus Alliance
- Grid Security Infrastructure (GSI)
- Job submission (GRAM)
- Information service (MDS)
- Data transfer (GridFTP)
- Replica Location (RLS)
-  Condor Group
- Condor/Condor-G
- DAGMan
- Fault Tolerant Shell
- ClassAds
-  EDG & LCG
- Make Gridmap
- Cert. Revocation List Updater
- Glue Schema/Info provider

-  ISI & UC
- Chimera & related tools
- Pegasus
-  NCSA
- MyProxy
- GSI OpenSSH
-  LBL
- PyGlobus
- Netlogger
-  Caltech
- MonALISA
-  VDT
- VDT System Profiler
- Configuration software
-  Others
- KX509 (U. Mich.)

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Grid3 Monitoring

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Monitoring Services

- GridCat - _[http://www.ivdgl.org/grid3/catalog/](http://www.ivdgl.org/grid3/catalog/)_
- Site catalog and summary information and site status display
- Ganglia - _[http://gocmon.uits.iupui.edu/ganglia-webfrontend](http://gocmon.uits.iupui.edu/ganglia-webfrontend)_
- Open source tool to collect cluster monitoring information such as CPU and network load, memory and disk usage
- MonALISA - _[http://gocmon.uits.iupui.edu:8080/index.html](http://gocmon.uits.iupui.edu:8080/index.html)_
- Monitoring tool to support resource discovery, access to information and gateway to other information gathering systems
- ACDC Job Monitoring System - _[http://](http://acdc.ccr.buffalo.edu/statistics/acdc/fullsizeindexqueue.php)__[acdc](http://acdc.ccr.buffalo.edu/statistics/acdc/fullsizeindexqueue.php)__[.](http://acdc.ccr.buffalo.edu/statistics/acdc/fullsizeindexqueue.php)__[ccr](http://acdc.ccr.buffalo.edu/statistics/acdc/fullsizeindexqueue.php)__[.buffalo.](http://acdc.ccr.buffalo.edu/statistics/acdc/fullsizeindexqueue.php)__[edu](http://acdc.ccr.buffalo.edu/statistics/acdc/fullsizeindexqueue.php)__[/statistics/](http://acdc.ccr.buffalo.edu/statistics/acdc/fullsizeindexqueue.php)__[acdc](http://acdc.ccr.buffalo.edu/statistics/acdc/fullsizeindexqueue.php)__[/](http://acdc.ccr.buffalo.edu/statistics/acdc/fullsizeindexqueue.php)__[fullsizeindexqueue](http://acdc.ccr.buffalo.edu/statistics/acdc/fullsizeindexqueue.php)__[.](http://acdc.ccr.buffalo.edu/statistics/acdc/fullsizeindexqueue.php)__[php](http://acdc.ccr.buffalo.edu/statistics/acdc/fullsizeindexqueue.php)_
- Application uses globus GRAM to query job managers and collect information about jobs. This information is stored in a DB and available for aggregated queries and browsing.
- Metrics Data Viewer (MDViewer) - _[http://grid.](http://griddev.uchicago.edu/swhome/mdviewer/mdviewer.jnlp)__[uchicago](http://griddev.uchicago.edu/swhome/mdviewer/mdviewer.jnlp)__[.](http://griddev.uchicago.edu/swhome/mdviewer/mdviewer.jnlp)__[edu](http://griddev.uchicago.edu/swhome/mdviewer/mdviewer.jnlp)__[/metrics/](http://griddev.uchicago.edu/swhome/mdviewer/mdviewer.jnlp)_
- Application to display and analyze information collected by the different monitoring tools, queries Metrics DBs at iGOC.
- Globus MDS
- Information and Index Service for resource discovery, selection and optimization. GLUE schema with Grid3 extension

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Monitoring Infrastructure

- Monitoring Infrastructure

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Grid3 Authentication

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Grid3 Authentication

**iVDGL VOMS server**

**edg-mkgridmap**

** ****FNAL VOMS server**

**  ****BNL VOMS server**

**user DNs**

**user DNs**

**user DNs**

**site a client**

**site b client**

**site n client**

- mapping of user’s grid credentials (DN) to local site group account

**USCMS, SDSS**

**USATLAS**

**BTeV, LSC, iVDGL**

**DN mappings**

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Grid3 Operations

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Grid3 Operations: (iGOC)

- Grid3 Operations: (iGOC)

- _[http://www.ivdgl.org/grid2003/catalog](http://www.ivdgl.org/grid2003/catalog)_

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Grid3 OperationsSupport and Policy

- Investigation and resolution of grid middleware problems at the level of 16-20 contacts per week
- With other iGOC personnel develop Service Level Agreements for iVDGL Grid service systems and iGOC support service.
- Membership Charter completed which defines the process to add new VO’s, sites and applications to the Grid Laboratory
- Support Matrix defining Grid3 and VO services providers and contact information

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Grid2003 Applications

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Project Application Overview

- 7 Scientific applications and 3 CS demonstrators
- All iVDGL experiments participated in the Grid2003 project
- A third HEP and two Bio-Chemical experiments also participated
- Over 100 users authorized to run on Grid3
- Application execution performed by dedicated individuals
- Typically 1, 2 or 3 users ran the applications from a particular experiment
- Participation from all Grid3 sites
- Sites categorized according to policies and resource
- Applications ran concurrently on most of the sites
- Large sites with generous local use policies where more popular

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Running on Grid3

- With information provided by the Grid3 information system
- Composes list of target sites
- Resource available
- Local site policies
- Finds where to install application and where to write data
- Use of Grid3 Information Index Service (MDS)
- Provides pathname for $APP, $DATA, $TMP and $WNTMP
- User sends and remotely installs application from a local site
- Entire application environment is shipped with the executable!
- User submit job(s) through globus GRAM
- User never needs to interact with local site administrators other than through the Grid3 services!

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Grid3 Metrics

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Grid3 Metrics Collection

- Grid3 monitoring applications (information consumers)
- MonALISA
- MetricsData Viewer
- Queries to  persistent storage DB (on the gocmon server)
- MonALISA plots
- MDViewer plots

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Grid3 Metrics Collection

**MDViewer**

**MonALISA**

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Grid3 Status Summary

- Current hardware resources
- Total of 2693 CPUs
- Maximum CPU count
- Off project contribution > 60%
- Total of 25 sites
- 25 administrative domains with local policies in effect
- All across US and Korea
- Running jobs
- Peak number of jobs 1100
- During SC2003 various Scientific applications were running simultaneously across various Grid3 sites

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Conclusions

- Grid computing has a long way to go to reach the goal: “plug in and you get the power”
- Many complex issues are involved in building and maintaining a grid
- Various software packages are developed to ease the burden
- Happy Grid hacking 

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Extra Slides

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Scientific Applications

- High Energy Physics Simulation and Analysis
- USCMS: MOP, GEANT based full MC simulation and reconstruction
- Work flow and batch job scripts generated by McRunJob
- Jobs generated at MOP master (outside of Grid3) submit jobs to Grid3 sites via condor-G
- Data products are archived at FermiLab: SRM/dCache
- USATLAS: GCE, GEANT based full MC simulation and reconstruction
- Workflow is generated by Chimera VDS, Pegasus grid scheduler and globus MDS for resource discovery
- Data products archived at BNL : Magada and globus RLS are employed
- USATLAS: DIAL, Distributed analysis application
- Dataset catalogs built, n-tuple analysis and histogramming (data generated on Grid3)
- BTeV : Full MC simulation
- Also utilizes the Chimera workflow generator and condor G (VDT)

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Scientific Applications

- Astrophysics and Astronomical
- LIGO/LSC: blind search for continuous gravitational waves
- SDSS: maxBcg, cluster finding package
- Bio-Chemical
- SnB: Bio-molecular program, analyses on X-ray diffraction to find molecular structures
- GADU/Gnare: Genome analysis, compares protein sequences
- Computer Science
- Evaluation of Adaptive data placement and scheduling algorithms

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## CS Demonstrator     Applications

- Exerciser
- Periodically runs low priority jobs at each site to test operational status
- NetLogger-grid2003
- Monitored data transfers between Grid3 sites via NetLogger instrumented pyglobus-url-copy
- GridFTP Demo
- Data mover application using GridFTP designed to meet the 2TB/day metric

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004

## Metrics Summary Table

| Metric | Target | Grid2003 “SC2003” |
| --- | --- | --- |
| Number of CPUs | 400 | 2762 (27 sites) |
| Number of users | > 10 | 102 (16) |
| Number of Applications | > 4 | 10 |
| Number of site running concurrent applications | > 10 | 17 |
| Peak number of concurrent jobs | 1000 | 1100  |
| Data Transfer per day | > 2-3 TB | 4.4 TB (11.12.03)  |

- Lecture 7: Building, Monitoring and Maintaining a Grid

- June 21-25, 2004