# Presentation: 106490

## MDS Update for CEDPS Troubleshooting

## Standard Schemas

- (GLUE schema, eg)

- Information Users :
- Schedulers, Portals, Warning Systems, etc.

- WS standard interfaces for subscription, registration, notification

## MDS4 Trigger Service

- Subscribe to data from information providers
- Evaluate that data against a set of pre-configured conditions (Trigger rules)
- When a condition matches, action occurs
- Email is sent to pre-defined address
- Website updated
- Write a structured log file that will later be read by some other program

## Information Providers

- Data sources for the higher-level services
- Some are built into services
- Any WSRF-compliant service publishes some data automatically
- WS-RF gives us standard Query/Subscribe/Notify interfaces
- GT4 services: ServiceMetaDataInfo element includes start time, version, and service type name
- Most of them also publish additional useful information as resource properties

## Trigger Service

- Site-wide Trigger service collects data and sends email upon errors
- Information providers are polled at pre-defined services
- Value must be matched for set number of intervals for Trigger to occur to avoid false positives
- Trigger has a delay associated for vacillating values
- Used for offline debugging as well

## ESG Deployment Example

| Service Being Monitored | ESG Location |
| --- | --- |
| GridFTP server | NCAR |
| OPeNDAP server | NCAR |
| Web Portal | NCAR |
| HTTP Dataserver | LANL, NCAR |
| Replica Location Service (RLS) servers | LANL , LBNL, LLNL, NCAR, ORNL |
| Storage Resource Managers | LANL, LBNL,  NCAR, ORNL |
| Hierarchical Mass Storage Systems | LBNL, NCAR, ORNL |

## 1 Month of Error Messages

| Total error messages for May 2006 | 47 |
| --- | --- |
| Messages related to certificate and configuration problems at LANL | 38 |
| Failure messages due to brief interruption in network service at ORNL on 5/13 | 2 |
| HTTP data server failure at NCAR 5/17 | 1 |
| RLS failure at LLNL 5/22 | 1 |
| Simultaneous error messages for SRM services at NCAR, ORNL, LBNL on 5/23 | 3 |
| RLS failure at ORNL 5/24 | 1 |
| RLS failure at LBNL 5/31 | 1 |

## Trigger Service Next Steps

- Work currently by Robert Miller, partially in response  to ESG requests
- Editing of Trigger state data via a Web interface
-  Allow changes to parameters such as which scripts get executed or how often the Trigger checks condition changes
- Ability to disable and re-enable individual Triggers via a Web interface without destroying them altogether or restarting the container
- Full integration with a Trigger Web interface for state management

## Trigger configuration files can be created by entering form data instead of manual entry of XML data

## Trigger Web Interface Prototype (currently in development)

- Create Trigger configuration files
- Form to enter Trigger data
- Submission of completed form generates a Trigger configuration file in the appropriate location
- Trigger configuration files
- List all configuration files
- View each configuration file
- Delete configuration file
- Link to the action script associated to the Trigger
- Active Triggers
- List all active Triggers
- Link to Trigger details
- View Trigger details
- Customizable-  XHTML-compliant, CSS-driven
- Colors, borders, etc. easily modified

- Previously the only way to list active Triggers was to query the Service, which returned an XML list
- This visual interface shows the information more simply, and clicking on the Trigger ID (in green) links to a table of the Trigger details, in an easy to read format

## Trigger Discussion

## MDS4 Archiver Services

- A WSRF service that allows storage and access of "Snapshot" data, useful for capacity planning
- Built on the MDS4 Aggregator Framework so that data is retrieved in a standard way just like an Index Service
- Uses a Xindice database backend
- Data is stored as a hierarchical XML database on disk organized by archival date and time

## Ways to access the Archiver Service (WSDL operations)

- Add operation
- Stores XML documents and returns URIs to those documents, allowing them to be retrieved directly
- Get operation
- Takes a URI and returns the document matching it
- Query operation
- Searches via XPath over a specified date/time range and a set of namespace mappings
- Command line tools are available now to perform these operations

## MDS4 Archiver Pros

- Data is stored in XML and searchable with XPath
- Queries are optimized for many small and medium sized collections of XML documents
- Queries are optimized for timestamp searches (e.g. find all documents from one day/time to another)
- Modular code: allowing another backend (e.g. a relational DB) is possible, although data access methods will also need work or translation
- Xindice is Apache Licensed and already included in GT4's WSRF core package
- No additional DB installation/configuration is required

## MDS4 Archiver Cons

- Queries are NOT optimized for arbitrary string searches over large documents (potentially containing multiple copies of the same data or data that's changed little over time)
- Large queries may potentially result in "Out of Memory" errors due to the way we use Xalan processing for queries
- all results are collected in memory before returning them
- Data and query conversion may be necessary to interface with applications/services that don't natively "speak" XML

## Potential Next Steps

- Investigate the performance of storing, retrieving and querying archive data
- Look at other storage backends and retrieval methods that may be better suited for rapid event or real-time logging data
- Design a graphical Web-based interface for rapid data retrieval and general user visualization

## Archiver Discussion

## OSG work

- Agreement to Troubleshooting Best Practices document
- Alignment with CEDPS recommendation of syslog-ng as the underlying initial infrastructure for recording, filtering, reporting, central storage, of  troubleshooting logging/auditing information
- Readiness of Troubleshooting "V0" for testing on the ITB in August 2007. The CEDPS effort would develop a readiness plan and follow through in collaboration for testing and use of the infrastructure, storage and analysis of the information stored as an  OSG 0.8.0 deliverable.
- Work with Condor and Globus developers to have the information sent from these services - on a time frame possible for the development groups. An initial deployment of Troubleshooting in OSG would use information from currently available logs.
- Q for Alain: From Alains Comments:
- This covers a LOT of software and we cannot have the use of Troubleshooting tools be contingent on this happening. In the OSG model of “adaptors” is CEDPS prepared to provide such adaptors for existing software?   Clearly V0 of Troubleshooting will not be able to depend on the common format being available. What will be in V0 to help us debug problems or analyze security incidents and manage the mess of logs we have today?