# Presentation: 345097

## RTES Approach to Software Fault Tolerance

- Jim KowalkowskiFermilab-CD

- JBK

## Purpose

- To give you a brief, high-level overview of some of the concepts, techniques, and tools that the RTES collaboration developed to address software reliability concerns in the BTeV trigger system.

- (Although looking at these slides, one may think it is not so brief.)

- JBK

**Notes:**
More information is in this talk than I can cover.
I will skip much of the details.
Would like to go through the material fairly quickly and then go back to discuss any part in more detail.

## Very brief history

- BTeV was developing a complex trigger system, which included a large collection of FPGAs and commodity hardware in two large clusters with
- High availability requirement
- High throughput requirement
- Real time processing constraints
- The RTES collaboration was formed to help assure that the availability requirement was met
- Four universities with expertise in software and hardware fault tolerance, reliability engineering, and real time processing
-     *(I was the liaison between BTeV and RTES)*

- JBK

## Problem and RTES Goal

- Problem: Software reliability depends on
- Physics detector-machine performance
- Program *testing procedures, implementation, and design quality*
- Behavior of the electronics (front-end and within the trigger)
- Goal: Create fault handling infrastructure capable of
- Accurately identifying problems (where, what, and why)
- Compensating for problems (shift the load, changing thresholds)
- Performing automated recovery procedures (restart / reconfiguration)
- Accurate accounting
- Being extended (capturing new detection/recovery procedures)
- Policy driven monitoring and control
-   ( also wanted to simplify operations)

- JBK

**Notes:**
Policy statement – how the machine is being used e.g. shared with sim, under heavy load, partitioned
Programs ... - most of which we had little control over

## What aspects are interesting?

- Hierarchical decomposition of problem, which addresses
- Real time constraints (react quickly when necessary)
- Scalability
- Resource usage constraints
- Protocols between levels
- Separation of concerns
- the various contributors write code specific to their need
- Very low coupling
- linkage through message subscriptions
- Separation of monitoring, problem detection, and actions
- The system can change dynamically (as it is running)
- Interprocess messaging infrastructure
- Based on Elvin
- A Publish/subscribe system
- Supplied gateways and routers at various levels
- High-level abstractions
- System behavior and configuration and can be expressed using domain specific concepts and terms
- Tools actually executing the configuration can evolve independently

- JBK

**Notes:**
Separation of concerns – software infrastructure takes care of the rest
Gateways and router – into specific software elements not directly on the network

## Development approach

- Generated a series of use cases that describe typical system behavior.
- Generated a set of prototypical problems that may occur on each of the systems.
- Generated a system architecture that looked similar to the vision of the real system
- Created demonstration systems that match this architecture and emulate operation of various parts of the system using RTES developed products
- Purpose is to detect and react to a given set of problems
- Made Level 1 trigger project using DSP event processors, Linux regional managers, and a high-level control system
- Made Level 2/3 trigger project using a Linux farm

- JBK

## Hierarchical Detection/Mitigation

- JBK

**Notes:**
Each product satisfies some research goal.
Most everything can be explained from with page.  Products/technologies are on the left.A couple of frameworks, each defining an API, designed to pull info out of the applications andhardware and ripple it up through a hierarchy until something can deal with it.A modeling tool sits on top to stitch together the various frameworks and applicationsaccording to a user specification.
Diagrams are made of various views of the system, then instantiate everything necessary
to configure and execute the desired behavior.
Classes or types of trigger control and event processing entities and with multiplicities
Lowers levels are simple and see high rates at with a narrow view
Higher levels are more complex and see lower rates of summarized data with a broad view

## Configuration through Modeling

- Multi-aspect tool, separate views of
- Hardware – components and physical connectivity
- Executables – configuration and logical connectivity
- Fault handling behavior using hierarchical state machines
- Model interpreters can generate the system image
- At the code fragment level (for fault handling)
- Download scripts and configuration
- Validation and testing
- Modeling “languages” are application specific
- Shapes, properties, associations, constraints
- Appropriate to application/context
- System architecture/configuration
- Component states
- Message structures
- Fault mitigations

- JBK

**Notes:**
Dataflow between executables or modules aspect
Code generation depends on the complexity of the model interpreter
Tool set from VU
Systems modeling

## Modeling Environment

- JBK

## ARMORs

- Are multithreaded processes composed of replaceable or ‘pluggable’ building blocks called Elements
- Elements provide error detection and recovery services to the trigger and other applications
- Restarts, reconfiguration
- Removal from service
- ARMOR framework routes “messages” and schedules Elements based on their message subscriptions
- A Hierarchy of ARMOR processes form a reconfigurable runtime environment:
- System management, error detection, and error recovery services are distributed across ARMOR processes
- ARMOR runtime environment *can handle self failure*
- ARMOR support for the application:
- Completely transparent and external support
- Instrumentation with ARMOR API

- JBK

**Notes:**
Moving up towards PC class machines running Linux
Existing technology out of UIUC
Reduce or increase the verbosity of monitoring information

## Very Lightweight Agents (VLA)

- Message scheduling and priority assignments
- Fast, simple reactive decisions
- Reads, summarizes, and reports sensors data
- Are “*pluggable*” components
- Lives alongside application
- Some predictive capabilities

- JBK

**Notes:**
Light-weight Message bus with “pluggable” components
Watching hardware, interactions with communications channels, physics algo performance issues

(early view into a level-1 processor)

## Shortcomings

- VLAs
- Standarized API and management framework never established
- Definition much to vague to know precisely if your code is one of these or not
- ARMORs
- Difficult to write ‘pluggable’ components (complex execution model)
- Only support for C++
- Management and configuration not adequate

- (Please note: Some of the developments were cut short due to the cancellation of BTeV)

- JBK

## Current Activities

- LQCD cluster, need to automate
- Routine administration tasks
- Recovery procedures (for jobs and nodes)
- Collecting of performance information
- CMS online
- Automate system validation and testing

- (These are the ones I am aware of.)

- JBK