# Presentation: 158503

## Information Systems in SAMGrid

**Adam Lyon**
***10 May 2005 GDM***

_**Outline**_
** ****Current SAMGrid Monitoring**
** ****Future Plans**
** ****Connections to outside**

## Types of Current SAMGrid Monitoring

- Trouble detection and investigation
*Is SAMGrid OK? What’s wrong?*
- _[Sam-at-a-glance](http://cdfsam-prd.fnal.gov/sam_local/SamAtAGlance/)_
- _[DBS Monitor Plots](http://dbsmon.fnal.gov/)_
- Text station _[dumps](http://d0db-prd.fnal.gov/sam_local/SamAtAGlance/fnal-cabsrv1.dump.html)_/logs
- _[SamTV](http://www-clued0.fnal.gov/~sam/samTV/current/)_
- Internal Efficiency
*Has SAMGrid been running well?*
- SamTV
- Rate plots
- SAMGrid Efficiency Plots

- Performance & Public Relations
*What has SAMGrid done for the experiment lately?*
- Rate & Efficiency plots
- Customized by hand plots
- Users’ Monitoring
*What is my project doing (why isn’t it done)?*
- SamTV
- SAMGrid Job Monitoring

**Notes:**
Trouble detection -- For SAMGrid Shifters and Experts

## Plots made by hand

## Observations

*Monitoring is not “one tool fits all”; rather it’s “use the best tool for the job”*
*Hard to imagine one tool that can do all of the monitoring we’ve created*
*... Because there are different levels of monitoring*
*30,000 ft level [Users, stakeholders, funding agencies]*
*Have covered some of this, but nothing unified or easy*
*User specific level *
*Only SAMGrid job monitoring is geared for this*
*Low level [shifters and experts]*
*This is where we’ve had the most need, and so it is well covered, but room for improvement*
*While some monitoring is a no-brainer, knowing what to monitor a priori is not easy*

## Issues with Current Monitoring

- Gathering the monitoring data
- Parsing log files [creaky, slow]
- Trolling the DB [incomplete, difficult]
- XML DB (for SAMGrid Job Info - easier since they have specific monitoring needs)
- Presenting the data to the user, shifter, expert, stakeholder
- Often the harder part - again, different people have different needs
- GNUPlot for the rate plots
- Java plotting for DBS Monitoring
- Customized SVG and *R* for SamTV
- Customized Web pages

## Current activities

- We are unifying the “Gather the data” part now
- “Instrument” SAM DH services & clients so they send events (via CORBA - UDP) to a Monitoring Server

## Gathering the data...

- Using technology we already use & know (started ~2 yrs ago)
- Highly extensible and unified (any “source” can be instrumented, all sources talk the same way) - But SAM specific
- Status:
- Monitoring server infrastructure is written and in use for testing
- Client is already instrumented
- Station related services are 95% instrumented
- What to do with the data?
- Act on it immediately [Push] (send mail, NGOP)
- Store for future retrieval [Pull]
- Use a MySQL DB - want it to be simple and fast. Schema is geared for maximal speed in writing (mirrors CORBA dictionary structure).

## Presenting the Data

- Sam**HD**TV is in the works
- Works off of DB instead of log files
- Summer student put all current functionality of SamTV into SamHDTV (plus some extra stuff)
- Needs testing and deployment
- Connection to MonALISA has been explored
- Have a prototype connector
- More time needs to be spent on determining MonALISA limitations and what exactly we want to display
- Future goals - Initial deployment (we’re about a month away)! Displays geared for users, 30,000 feet plots, connection to NGOP, incorporating JIM information

## Fitting in with a broader project

- Because SAMGrid has its own services, data gathering instrumentation is SAMGrid specific
- Aside from use of CORBA, monitoring server is **not** SAMGrid specific
- Could gather info via other mechanisms. Server is built for robustness (minimizes lost monitoring data with zero impact on monitored service)
- We need to monitor SAMGrid specific things
- So a generic monitoring service is difficult to utilize
- So a generic monitoring schema is difficult to utilize (GLUE)
- But there’s no reason why we can’t export (MonALISA)
- But these exports will be a subset of our monitoring data

## “Information Service”

- Motivation
- When FNAL DB goes down, SAM DH stops worldwide [service outages]
- SAM DH Stations must talk to FNAL DB for configuration information [can be slow for far away stations]
- Solution
- A replacement DB Server (SAM DH specific) that can provide some level of Station autonomy
- Get the best out of a (excellent) CS graduate student before he graduates in June

## Summary

- Monitoring is crucial for success
- With each improvement in monitoring, SAMGrid has benefited enormously
- But we’ve learned there’s no “one solution for all”
- Monitoring must meet many disparate needs
- Monitoring must be present at many disparate levels
- SAMGrid project would be happy to examine monitoring solutions, so long as our stakeholder needs are met

## References

- Monitoring:
- CHEP 2004: 451-SAMGrid Monitoring Service and its Integration with MonALisa
- Information Service:
- See talk athttps://plone3.fnal.gov/SAMGrid/Wiki/SAM_InformationService_SAM_Planning_Meeting-03_31_05.ppt