# Presentation: 073528

## DemoSystem2004

- RTES Collaboration

- (NSF ITR grant ACI-0121658)

**Real-Time and Embedded Technology **
**       ****& Applications Symposium (IEEE)**
- FALSE II Workshop
*San Francisco,     *March 7-10, 2005

## Introduction

- Our context: BTeV
- A 20 TeraHz Real-Time System
- Our Solution
- MIC, ARMORs, VLAs
- The demonstration system
- Demonstration
- Comments

- FALSE II Workshop

- 7 March 2005

## BTeV High Energy Physics Experiment: A 20 TeraHz Real-Time System

**BTeV** High Energy Physics Experiment: A 20 TeraHz Real-Time System

- Input: 800 GB/s (2.5 MHz)
- Level 1
- Lvl1 processing: 190s
- rate of 396 ns
- 528 “8 GHz” G5 CPUs
- (factor of 50 event reduction)
- high performance interconnects
- Level 2/3:
- Lvl 2 processing: 5 ms
- (factor of 10 event reduction)
- Lvl 3 processing: 135 ms
- (factor of 2 event reduction)
- 1536 “12 GHz” CPUs commodity networking
- Output: 200 MB/s (4 kHz) = 1-2 Petabytes/year

- FALSE II Workshop

- 7 March 2005

## The Problem

- Monitoring, Fault Tolerance and Fault Mitigation are crucial
- In a cluster of this size, processes and daemons are constantly hanging/failing without warning or notice
- Software reliability depends on
- Physics detector-machine performance
- Program testing procedures, implementation, and design quality
- Behavior of the electronics (front-end and within the trigger)
- Hardware failures will occur!
- one to a few per week

- FALSE II Workshop

- 7 March 2005

**Notes:**
Complexity of the event and noise, missing channels, aliasing, etc.

## The Problem (continued)

- Given the very complex nature of this system  where thousands of events are simultaneously and asynchronously cooking, issues of data integrity, robustness, and monitoring are critically important and have the capacity to cripple a design if not dealt with at the outset...    BTeV [needs to] supply the necessary  level  of “self-awareness” in the trigger system.
- [June 2000 Project Review]

- FALSE II Workshop

- 7 March 2005

**Notes:**
Complexity of the event and noise, missing channels, aliasing, etc.

## BTeV’s Response:  RTES

- The Real Time Embedded System Group
- A collaboration of five institutions,
- University of Illinois
- University of Pittsburgh
- University of Syracuse
- Vanderbilt University (PI)
- Fermilab
- NSF ITR grant ACI-0121658
- Physicists and Computer Scientists/Electrical Engineers at BTeV institutions with expertise in
- High performance, real-time system software and hardware,
- Reliability and fault tolerance,
- System specification, generation, and modeling tools.
- Working on fault management in large computing clusters

- FALSE II Workshop

- 7 March 2005

**Notes:**
Including both studies of fault management and producing software

## RTES Goals for BTeV

- High availability
- Fault handling infrastructure capable of
- Accurately identifying problems (where, what, and why)
- Compensating for problems (shift the load, changing thresholds)
- Automated recovery procedures (restart / reconfiguration)
- Accurate accounting
- Extensibility (capturing new detection/recovery procedures)
- Policy driven monitoring and control
- Dynamic reconfiguration
- adjust to potentially changing resources

- FALSE II Workshop

- 7 March 2005

**Notes:**
Available despite failures
Have a “pluggable” nature
dynamic nature
Easy configuration and interfacing to control components like an alarm tool

## RTES Goals for BTeV (continued)

- Faults must be detected/corrected ASAP
- semi-autonomously
- with as little  human intervention as possible
- distributed and hierarchical monitoring and control
- Life-cycle maintainability and evolvability
-  to deal with new algorithms, new hardware and new versions of the OS

- FALSE II Workshop

- 7 March 2005

**Notes:**
Available despite failures
Have a “pluggable” nature
dynamic nature
Easy configuration and interfacing to control components like an alarm tool

## The RTES Solution

**Analysis**

**Runtime **

**Design and**
**Analysis**

- Algorithms

**Fault ****Behavior**

**Resource**

**Synthesis**

**Performance**
**Diagnosability**
**Reliability**

**Experiment****Control**
**Interface**

**Synthesis**

**Feedback**

**Modeling**

**Reconfigure **

**Region ****Operations Mgr**

**L2,3/CISC/RISC**

**L1/DSP**

** ****Soft                    Real Time                         Hard**

- FALSE II Workshop

- 7 March 2005

## RTES Deliverables

- A hierarchical fault management system and toolkit:
- Model Integrated Computing
- GME (Generic Modeling Environment) system modeling tools
- and application specific “graphic languages” for modeling system configuration, messaging, fault behaviors, user interface, etc.
- ARMORs (Adaptive, Reconfigurable, and Mobile Objects for Reliability)
- Robust framework for detection and reaction to faults in processes
- VLAs (Very Lightweight Agents for limited resource environments)
- To monitor/mitigate at every level
- DSP, Supervisory nodes, Linux farm, etc.

- FALSE II Workshop

- 7 March 2005

## Configuration through Modeling

- Multi-aspect tool, separate views of
- Hardware – components and physical connectivity
- Executables – configuration and logical connectivity
- Fault handling behavior using hierarchical state machines
- Model interpreters can generate the system image
- At the code fragment level (for fault handling)
- Download scripts and configuration
- Modeling “languages” are application specific
- Shapes, properties, associations, constraints
- Appropriate to application/context
- System model
- Messaging
- Fault mitigation
- GUI, etc.

- FALSE II Workshop

- 7 March 2005

**Notes:**
Dataflow between executables or modules aspect
Code generation depends on the complexity of the model interpreter
Tool set from VU
Systems modeling

## Modeling Environment

- FALSE II Workshop

- 7 March 2005

## ARMOR

**A**daptive **R**econfigurable **M**obile **O**bjects of **R**eliability:
- Multithreaded processes composed of replaceable building blocks
- Provide error detection and recovery services to user applications
- Hierarchy of ARMOR processes form runtime environment:
- System management, error detection, and error recovery services distributed across ARMOR processes.
- ARMOR Runtime environment is itself self checking.
- 3-tiered ARMOR support of user application
- Completely transparent and external support
- Enhancement of standard libraries
- Instrumentation with ARMOR API

- FALSE II Workshop

- 7 March 2005

## ARMOR Scaleable Design

- ARMOR processes designed to be *reconfigurable*:
- Internal architecture structured around event-driven modules called *elements*.
- Elements provide functionality of the runtime environment, error-detection capabilities, and recovery policies.
- Deployed ARMOR processes contain only elements necessary for required error detection and recovery services.
- ARMOR processes *resilient* to errors by leveraging multiple detection and recovery mechanisms:
- Internal self-checking mechanisms to prevent failures from occurring and to limit error propagation.
- State protected through checkpointing.
- Detection and recovery of errors.
- ARMOR runtime environment *fault-tolerant* and *scalable*:
- 1-node, 2-node, and *N*-node configurations.

- FALSE II Workshop

- 7 March 2005

## ARMOR System: Basic Configuration

- FALSE II Workshop

- 7 March 2005

## ARMOR: Internal Structure

- FALSE II Workshop

- 7 March 2005

**Notes:**
Mention element collection dictates what recoveries are possible (reconfigurable by changing elements)
Attached ARMOR performs a specialized function
Message routing bus
Simple hierarchy shown, lower level has
  reporting/sensing -> detection/analysis -> recovery action sequence elements

## Very Lightweight Agents

- Minimal footprint
- Platform independence
- Employable everywhere in the system!
- Monitors hardware and software
- Handles fault detection& communications with higher level entities

- FALSE II Workshop

- 7 March 2005

## BTeV Prototype Farms at Fermilab

- FALSE II Workshop

- 7 March 2005

## L2/3 Prototype Farm Setup

- FALSE II Workshop

- 7 March 2005

## L2/3 Prototype Farm Components

- Using PC’s from old PC Farms at Fermilab
- 3 dual-CPU Manager PCs
- boulder (1 GHz P3) - meant for data server
- iron (2.8 GHz P4) - httpd, BB and Ganglia gmetad
- slate (500 MHz P3) - httpd, BB
- Managers have a private network through 1 Gbps link
- bouldert, iront, slatet
- 15 dual-CPU (500 MHz P3) workers (btrigw2xx)
- 84 dual-CPU (1GHz P3) PC workers (btrigw1xx)
- No plans to add more, but may replace with faster ones
- Ideal for RTES
- 11 workers already have problems!

- FALSE II Workshop

- 7 March 2005

## The Demonstration System Components

- Matlab as GUI engine
- GUI defined by GME models
- Elvin publish/subscribe networking (everywhere)
- Messages defined by GME models
- RunControl (RC) state machines
- Defined by GME models
- ARMORs
- Custom elements defined by GME models
- FilterApp, DataSource
- Actual physics trigger code
- File-reader supplies physics/simulation data to the FilterApp
- Demo “faults” encoded onto Source-Worker data messages for execution on the Worker

- FALSE II Workshop

- 7 March 2005

## The Demonstration System Architecture

- public

- private

- FALSE II Workshop

- 7 March 2005

## BB and Ganglia

- For traditional monitoring

- FALSE II Workshop

- 7 March 2005

## Matlab

- For RTES/Demo-specific monitoring

- FALSE II Workshop

- 7 March 2005

## Comments

- This is an integrated approach – from hardware to physics applications
- Standardization of resource monitoring, management, error reporting, and integration of recovery procedures can make operating the system more efficient and make it possible to comprehend and extend.
- There are real-time constraints that must be met
- Scheduling and deadlines
- Numerous detection and recovery actions
- The product of this research will
- Automatically handle simple problems that occur frequently
- Be as smart as the detection/recovery modules plugged into it

- FALSE II Workshop

- 7 March 2005

**Notes:**
Error analysis and reliability analysis opportunities across many components
Want to quickly arrive at an accurately diagnosis and treatment that you have confident in
Discover software bugs that only occur once every few hours or on

## Comments (continued)

- The product can lead to better or increased
- System uptime by compensating for problems or predicting them
- instead of pausing or stopping the experiment
- Resource utilization
- the system will use resources that it needs
- Understanding of the operating characteristics of the software
- Ability to debug and diagnose difficult problems

- FALSE II Workshop

- 7 March 2005

**Notes:**
Error analysis and reliability analysis opportunities across many components
Want to quickly arrive at an accurately diagnosis and treatment that you have confident in
Discover software bugs that only occur once every few hours or on

## Further Information

- General information about RTES
- http://www-btev.fnal.gov/public/hep/detector/rtes/
- General information about BTeV
- http://www-btev.fnal.gov/
- Information about GME and the Vanderbilt ISIS group
- http://www.isis.vanderbilt.edu/

- FALSE II Workshop

- 7 March 2005

## Further Information (continued)

- Information about ARMOR technology
- http://www.crhc.uiuc.edu/DEPEND/projects-ARMORs.htm
- Talks from our last workshop
- http://false2002.vanderbilt.edu/program.php
- Wiki (internal, today)
- whcdf03.fnal.gov/BTeV-wiki/DemoSystem2004
- Elvin publish/subscribe networking
- www.mantara.com

- FALSE II Workshop

- 7 March 2005

## Demonstration Outline

- GME
- SIML, DTML, Custom Elements, Matlab GUI
- Meta language
- Building and Deployment
- Runtime
- ARMORs, VLAs
- Fault detection and mitigation
- Reconfigure and Redeploy
- 2x, 4x

- FALSE II Workshop

- 7 March 2005

## Demonstration Slides

- FALSE II Workshop

- 7 March 2005

## MIC Based System Modeling 

- Design a domain-specific modeling language
- Provision concepts for all the different aspects of the system
- Express their interactions
- A “super” system-wide modeling language
- Implement a suite of translators
- Generate fault-mitigation behaviors
- Generate system build configurations
- Link with user code/libraries

- FALSE II Workshop

- 7 March 2005

## SIML –System Integration Modeling Language

- Model Component Hierarchy and Interactions
- Loosely specified model of computation
- Model information relevant for system configuration

- FALSE II Workshop

- 7 March 2005

## System Architecture

- RunControl Manager
- Router Information
- How many regions ?
- How many worker nodes inside the region?
- Node Identification information

- FALSE II Workshop

- 7 March 2005

## Data Type Modeling Language -DTML

**Modeling of Data Types and Structures**
**Configure marshalling-demarshalling interfaces for communication**

- FALSE II Workshop

- 7 March 2005

## Configuration of ARMOR Infrastructure (A)
Modeling of Fault Mitigation Strategies (B)
Specification of Communication Flow (C)

- Modeling of Fault Mitigation Strategies (B)
- Specification of Communication Flow (C)

**A**

**B**

**C**

- Fault Mitigation Modeling Language (1)

- FALSE II Workshop

- 7 March 2005

## Fault Mitigation Modeling Language (2)

- Model translator generates fault-tolerant strategies and communication flow strategy from FMML models
- Strategies are plugged into ARMOR infrastructure as ARMOR elements
- ARMOR infrastructure uses these custom elements to provide customized fault-tolerant protection to the application

- FALSE II Workshop

- 7 March 2005

## User Interface Modeling Language

- Enables reconfiguration of user interfaces
- Structural and data flow codes generated from models
- User Interface produced by running the generated code

- FALSE II Workshop

- 7 March 2005

## User Interface Generation

- FALSE II Workshop

- 7 March 2005

## GME meta language(s) –Meta Model

- FALSE II Workshop

- 7 March 2005

## Versioning/Build System

- An equivalent XML representation of GME models is stored in the CVS tree
- UDM tools provide forward and backward translation from MGA to XML
- Model transformers developed with UDM
- Can be built for Windows and Linux platforms
- Model transformation code is also stored in CVS

- FALSE II Workshop

- 7 March 2005

## Versioning/Build System

- Multi-stage build
- Compiles model transformers
- Makefiles invoke model transformers upon the stored models and generates code artifacts (behavior/config code)
- Generated code place in the existing source tree
- Source tree is compiled and linked to produce binaries
- Additional tools take over to traverse the run tree and distribute the components to all the nodes

- FALSE II Workshop

- 7 March 2005

## ARMOR Configuration in RTES

- FALSE II Workshop

- 7 March 2005

## Execution ARMOR in Worker

- FALSE II Workshop

- 7 March 2005

## Slide 44

- FALSE II Workshop

- 7 March 2005