# Presentation: 009873

## SAMGrid Monitoring and Information Service

- Siniša Veseli
- Fermilab

## Outline

- Credits
- Introduction
- MIS Design Requirements and Considerations
- MIS Arhitecture
- Monitoring Service
- Information Service
- Available Manpower
- Concluding Remarks

## Credits

- Excellent work by Mariano, Petr, and Matt this summer!
- Matt Leslie (monitoring server infrastructure, db work)
- Mariano Zimmler (station code, python client support, SAM-HDTV)
- Petr Vokac (station code, MonaLisa support)
- Adam Lyon (integration testing, SAMTV)
- SV (early prototype, design)

## Introduction: What do we need?

- Data Handling System (SAM) monitoring, diagnostics and troubleshooting
- Server performance information and statistics
- User job monitoring
- Hardware (networks, machines, etc.) monitoring and performance information
- Server runtime support
- Resource availability and usage information
- Archiving of historical data

## Introduction: What do we have?

- Monitoring
- SAMTV – great tool for monitoring stations (Adam)
- CAB monitoring scripts (Robert)
- SAAG pages
- Sam admin diagnostics tools
- DB Server statistics plots
- JIM monitoring, uses XML DB
- Information Service
- Centralized SAM DB
- JIM XML DB

## Introduction: What are the problems?

- Centralized SAM DB/DB Server:
- Single point of failure: entire SAM is down when the DB is down
- Non-scalable
- Monitoring tools based on log file parsing:
- Maintenance nightmare
- Non-scalable
- No real-time monitoring

## Introduction: Proposed Solution

- We started working towards new distributed Monitoring and Information Service
- Introduce event-based Monitoring Service
- Distribute existing Information Service by moving the DH server runtime support out of the central DB, replicate central DB catalog services

## MIS Design Requirements and Considerations

- Scalability
- Reliability and performance
- Non-intrusiveness
- Flexibility and extendibility
- Interoperability with other monitoring systems
- Security

## MIS Architecture

## Monitoring Service

- Provides support for:
- Monitoring, troubleshooting, and diagnostics for DH servers
- Monitoring of user jobs
- Monitoring hardware, networks, collective grid services, etc.
- Lots of progress this summer (thanks to Matt, Mariano, and Petr)

## Monitoring Service: Design and Implementation Status

- Distributed network of CORBA-based monitoring servers
- Events based on dictionaries (reflected in event DB schema)
- Support for push event mode
- Events handled by event processors/handlers
- Multithreading: thread pool for event handlers, designated threads for configured event processors

## Monitoring Service: Design and Implementation Status

- Abstract DB interface: allows any DB backend (only mysql used at the moment)
- Client libraries: allow easy instrumentation of SAM code with monitoring events (implemented for python and C++)
- Monitoring tools: SAM-HDTV almost ready: http://www-d0.fnal.gov/~zimmler/some_history.html
- Interoperability with other Grid monitoring systems: Mona Lisa event processor (initial implementation)

## Monitoring Service: Missing Features

- Subscription-based real-time monitoring (e.g., monitor your projects in real-time)
- Support for pull mode
- Other DB backends (postgres, oracle, xindice)

## Monitoring Service: Design Links

- Server architecture: http://cdfsamth.fnal.gov/mis/misdesign.pdf
- Client event design: http://kmlinux.fjfi.cvut.cz/~vokac/sam-mis/#d0e280

## Monitoring Service: Server Performance

- Matt's preliminary numbers (dual PIII 800MHz, 1GB RAM):
- Can sustain 45 events/second event rate, with occasional load spikes (CPU limited)
- DB disk space requirement: for 850 bytes average event size, 40 events/second rate => ~20GB/week
- Need test harness testing with real station/projects to confirm these numbers, get estimate of the expected real load

## Monitoring Service: Server Performance

- Server configuration can be further tuned (e.g., additional designated db processor threads, increased event queue sizes)
- Overloaded server does not crash, but throws events away

## Monitoring Service: Problems

- Very flexible DB schema, but certain queries seem to be fairly non-performant (may need to involve DBAs)
- Complex db installlation/configuration

## Monitoring Service: Interoperability with Grid Monitoring Systems

- Implementation of MonaLisa event processor is on its way
- Collaboration with MonaLisa developers possible => they would implement displays that we need, we could use MonaLisa as framework for visualizing and aggregating our monitoring data

## Monitoring Service: Milestones

- Milestone 1 (estimate: ~October '04)
- Build/release procedure cleanup
- Documentation cleanup
- Streamline installation/configuration (especially db)
- Integration testing: investigate performance, db space requirements
- Automatic db cleanup procedure
- DB query performance optimization
- Complete Mona Lisa integration
- Complete SAM-HDTV

## Monitoring Service: Milestones

- Milestone 1 (estimate: ~October '04)
- Finalize station events (e.g., station status event)
- Deploy production service at Fermilab
- CHEP '04 papers/presentations
- Merge station code

## Monitoring Service: Milestones  

- Milestone 2 (estimate: ~March '05)
- Real-time monitoring support
- Support for pull event mode
- Interoperability with other monitoring services, new db backends

## Information Service

- Centralized SAM DB:
- Provides runtime support for DH servers (e.g., station configuration, resource availability, etc.)
- Project bookkeeping services
-  JIM XML DB:
- Provides runtime support for JIM services
- Used for product and site configuration

## Information Service: Goals

- Primary goals:
- Station autonomy
- Removing centralized DB as the single point of failure
- Work on new components of the distributed information service hasn't started yet
- Roadmap not as clear as for monitoring
- Not starting from scratch: existing SAM DB/DB Servers, JIM XML DB/DB Servers

## Information Service: Goals

- Will share lots of components with monitoring: CORBA infrastructure, DB/DB interfaces, etc.
- Will try to incorporate existing services and code infrastructure

## Information Service: Milestones  

- Milestone 1 (wild estimate: ~March '05)
- Investigate oracle replication technology
- Investigate existing grid software and tools
- Design/prototype
- Station configuration: local DB design, aim to move it out of the central DB
- Complete information service interfaces: ensure smooth transition for station code

## Information Service: Milestones  

- Milestone 2 (wild guess: ~September '05)
- Replicate central DB catalog services
- Server caching/db server proxy: if information isn't cached/stored locally, find it elsewhere (similar to calibration db servers)
- Full station/fss autonomy: SAM fully operational while central db is down (running jobs/storing files)

## Available Manpower

- Exact effort still needs to be determined
- In the short term (next 3-4 months), contributions will likely come from:
- Mariano (for another week)
- Matt
- Adam
- Siniša

## Concluding Remarks

- Monitoring service:
- Lots of progress in the last couple of months (thanks to Petr, Matt, Mariano)
- Expected first production release in 6-8 weeks
- Information service:
- Starting work on formulating requirements, needs, goals, design, etc.
- Need to look into existing (grid) software and tools, how to incorporate our own existing services