# Presentation: 345096

## Understanding and Coping with Hardware and Software Failures in a Very Large Trigger Farm 

- The RTES group
- Presented by
- Jim Kowalkowski
- Fermilab Computing Division

- J.Kowakowski - CHEP 2003 La Jolla, CA

## The Problem

- The BTeV trigger has very large number of detector electronics and computing resources
-  ~1000s of FPGAs
- ~2500 embedded processors for level 1
- ~2500 PCs for level 2/3
- ~25,000,000 detector channels
- Millions of lines of code
- Must operate in real-time with no fixed time latency, averaging
- 300us for level-1 decision
- 13ms for level-2 for decision
- 130ms level-3 decision
- Failures can happen a few times a week for commodity parts
- Software reliability depends on
- Detector-machine performance
- Program test procedures, implementation, and design quality
- Behavior of the electronics (front-end and within trigger)

- J.Kowakowski - CHEP 2003 La Jolla, CA

**Notes:**
Complexity of the event and noise, missing channels, aliasing, etc.

## Goals Summary

- Implement a large, aggressive trigger, that
- Applies computation to every interaction
- Has high sustained computational performance
- Maintains functional integrity for long periods of time
- Is highly available
- Is dynamically reconfigureable, maintainable, and evolvable
- Create fault handling infrastructure capable of
- Accurately identifying problems (where, what, and why)
- Compensating for problems (shift the load*, changing thresholds*)
- Automated recovery procedures (restart / reconfiguration)
- Accurate accounting
- Being extended (capturing new detection/recovery procedures)
- Policy driven monitoring and control
- Simplify operations

- J.Kowakowski - CHEP 2003 La Jolla, CA

**Notes:**
Available despite failures
Have a “pluggable” nature
dynamic nature
Easy configuration and interfacing to control components like an alarm tool

## What is RTES?

- A collaboration of five institutions, funded by NSF ITR grant ACI-0121658
- University of Illinois** **(M. Haney, R.K. Iyer, Z. Kalbarczyk, Q. Liu, A. Mahajan, M. Selen, K. Whisnant, Z. Yang)
- University of Pittsburgh (D. Mosse, O. Shigiltchoff)
- University of Syracuse (R. Chopade, J. Oh, L. Hovey, S. Stone, M. Tamhankar)
- Vanderbilt University (T. Bapty, S. Neema, S. Norsdstrom, P. Sheldon, S. Shetty, E. Vaandering, D. Vashishtha)
- Fermilab (J. Appel, J. Butler, E. Gottschalk, J. Kowalkowski, L. Piccoli, M. Votava)
- Physicists and Computer Scientists/Electrical Engineers at BTeV institutions with expertise in
- High performance, real-time, embedded system software and hardware,
- Reliability and fault tolerance,
- System specification, generation, and modeling tools.
- A group working on fault management in large computing clusters

- J.Kowakowski - CHEP 2003 La Jolla, CA

**Notes:**
Including both studies of fault management and producing software

## Hierarchical Approach

- J.Kowakowski - CHEP 2003 La Jolla, CA

**Notes:**
Classes or types of trigger control and event processing entities and with multiplicities
Lowers levels are simple and see high rates at with a narrow view
Higher levels are more complex and see lower rates of summarized data with a broad view

## FPGA Directions

- High-speed recognition of software problems
- Increase cycles available for event processing
- Add reliability

- J.Kowakowski - CHEP 2003 La Jolla, CA

**Notes:**
Claim some of the FPGA resources for fault management and handling
Add protection and debugging abilities (diagnosis) to simple embedded processors

## Very Lightweight Agents (VLA)

- Message scheduling and priority assignments
- Fast, simple reactive decisions
- Reads, summarizes, and reports sensors data
- Are “*pluggable*” components
- Lives alongside application
- Some predictive capabilities

- J.Kowakowski - CHEP 2003 La Jolla, CA

**Notes:**
Light-weight Message bus with “pluggable” components
Watching hardware, interactions with communications channels, physics algo performance issues

## ARMORs

- Are multithreaded processes composed of replaceable building blocks called Elements
- Provide error detection and recovery services to the trigger and other applications
- Restarts, reconfiguration
- Removal from service
- A Hierarchy of ARMOR processes form a reconfigurable runtime environment:
- System management, error detection, and error recovery services are distributed across ARMOR processes
- ARMOR runtime environment *can handle self failure*
- ARMOR support for the application:
- Completely transparent and external support
- Instrumentation with ARMOR API

- J.Kowakowski - CHEP 2003 La Jolla, CA

**Notes:**
Moving up towards PC class machines running Linux
Existing technology out of UIUC
Reduce or increase the verbosity of monitoring information

## ARMOR View

**Daemon**

**TCP Connection**
**Mgmt.**

**Named Pipe****Mgmt.**

- ProcessMgmt.

**Detection****Policy**

**ARMOR**** ****Microkernel**

**Process****Mgmt.**

**Network**

**Daemon**

**Daemon**

***Remote daemons***

**Node 1**

**Node 2**

**Node 3**

**ARMOR Microkernel**

**Recovery**
**Policy**

**Local Manager ARMOR**

**Trigger**
**Application**

**Execution**
**Controller**

- J.Kowakowski - CHEP 2003 La Jolla, CA

**Notes:**
Mention element collection dictates what recoveries are possible (reconfigurable by changing elements)
Attached ARMOR performs a specialized function
Message routing bus
Simple hierarchy shown, lower level has
  reporting/sensing -> detection/analysis -> recovery action sequence elements

## Configuration through Modeling

- Multi-aspect tool, separates views of
- Hardware – components and physical connectivity
- Executables – configuration and logical connectivity
- Fault handling behavior using hierarchical state machines
- Model interpreters can generate the system image
- At the code fragment level (for fault handling)
- Download scripts and configuration (event filter)
- Modeling “language” is BTeV specific
- Shapes, properties, associations, constraints
- Starting to think about how to capture physics run types, configuration information, and fault handling

- J.Kowakowski - CHEP 2003 La Jolla, CA

**Notes:**
Dataflow between executables or modules aspect
Code generation depends on the complexity of the model interpreter
Tool set from VU
Systems modeling

## Generic Modeling Environment

- Fault handling
- Process dataflow
- Hardware Configuration

- J.Kowakowski - CHEP 2003 La Jolla, CA

## ITR Experience

- Overall application development vs. individual research interests
- Momentum – the way business has been done
- Learning curves
- The end application (BTeV trigger)
- Embedded system hardware (currently DSPs)
- Absorbing new thoughts and practices
- Spatial distance between the groups
- All groups involved are enjoying the sharing of knowledge and are good at listening and interacting with each other
- A good model for what is takes to make progress in computing within HEP?

- J.Kowakowski - CHEP 2003 La Jolla, CA

**Notes:**
Balancing can be challenging at times
Everyone comes with an investment in a methodology and tools
Students comes and go
Cannot just walk over to talk with people
Use of wikis and H.323 cameras
Forces each group to think through decisions they made and explain purpose clearly

## Why is all of this interesting?

- It is an *integrated approach* – from hardware to physics algorithms
- Standardization of resource monitoring, management, error reporting, and integration of recovery procedures can make operating the system more efficient and make it possible to comprehend and extend.
- There are real-time constraints
- Scheduling and deadlines
- Numerous detection and recovery actions
- The product of this research will
- Automatically handle simple problems that occur frequently
*Be as smart as the detection/recovery modules plugged into it*
- The product can lead to better or increased
- Trigger uptime by compensating for problems or predicting them instead of pausing or stopping a run
- Resource utilization - the trigger will use resources that it needs
- Understanding of the operating characteristics of the software
- Ability to debug and diagnose difficult problems

- J.Kowakowski - CHEP 2003 La Jolla, CA

**Notes:**
Error analysis and reliability analysis opportunities across many components
Want to quickly arrive at an accurately diagnosis and treatment that you have confident in
Discover software bugs that only occur once every few hours or on

## Summary

- This is an integrated approach, involving the
- DAQ
- Messages/Protocols/APIs
- Controls and monitoring system
- The application infrastructure
- Data handling system
- This is a highly scalable approach
- Actions taken as close to the problem as reasonably possible
- New detection/action elements can be dynamically added to the system
- New and valuable experiences and software are available for use in the BTeV trigger
- Research and development meant to be widely applicable

- J.Kowakowski - CHEP 2003 La Jolla, CA

## Further information

- General information about RTES
- _[http://www-btev.fnal.gov/public/hep/detector/rtes/](http://www-btev.fnal.gov/public/hep/detector/rtes/)_
- General information about BTeV
- _[http://www-btev.fnal.gov/public/gen/index.shtml/](http://www-btev.fnal.gov/public/gen/index.shtml/)_
- Information about the ISIS group
- _[http://www.isis.vanderbilt.edu/](http://www.isis.vanderbilt.edu/)_
- Information about ARMOR technology
- _[http://www.crhc.uiuc.edu/DEPEND/rtes.htm/](http://www.crhc.uiuc.edu/DEPEND/rtes.htm/)_
- Talks from our last workshop
- _[http://false2002.vanderbilt.edu/program.php/](http://false2002.vanderbilt.edu/program.php/)_

- J.Kowakowski - CHEP 2003 La Jolla, CA

## End...

- J.Kowakowski - CHEP 2003 La Jolla, CA

## Analysis

- Fault conditions list will be used as test cases
- Measure time to detect and react
- Simulation
- Functional (e.g. Matlab)
- Performance (Timing, Discrete Event)
- Diagnosability
- Failure Modes + Sensors
- Predict ability to Detect/Isolate Failures
- Reliability Analysis
- Predict MTBF, Maximum Failures
- Robustness
- Stability Analysis
- Reconfiguration Strategies/Control System

- J.Kowakowski - CHEP 2003 La Jolla, CA