# Presentation: 235625

## The RTES Project – BTeV, and Beyond

- Michael J. Haney1Shikha Ahuja2, Ted Bapty2, Harry Cheung3, Zbigniew Kalbarczyk4, Akhilesh Khanna4, Jim Kowalkowski3, Derek Messie5, Daniel Mossé6, Sandeep Neema2, Steve Nordstrom2, Jae Oh5, Paul Sheldon7, Shweta Shetty2, Dmitri Volper5, Long Wang4, Di Yao21High Energy Physics, University of Illinois, 1110 W. Green Street, Urbana, IL 61801 USA2Institute for Software Integrated Systems, Vanderbilt University, Nashville, TN 37235 USA3Fermi National Accelerator Laboratory, Batavia, IL 60510 USA4Electrical and Computer Science, University of Illinois, Urbana, IL 61801 USA5Electrical Engineering and Computer Science, Syracuse University, Syracuse, NY 13244 USA6Computer Science, University of Pittsburgh, Pittsburgh, PA 15250 USA7Physics and Astronomy Department, Vanderbilt University, Nashville, TN 37235 USA

## Outline

- Real Time Embedded System Project
- BTeV => RTES
- Prototypes
- SuperComputing 2003
- Demo System 2004
- Beyond BTeV

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## BTeV - High Energy Physics

- Input: 500 GB/s (2.5 MHz)
- Level 1 processing: 190s
- rate of 396 ns
- 528 “8 GHz” G5 CPUs
- (factor of 50 event reduction)
- high performance interconnects
- Level 2/3 processing: 5+135 ms
- (factor of 10+2 event reduction)
- 1536 “12 GHz” CPUs commodity networking
- Output: 200 MB/s (4 kHz) = 1-2 Petabytes/year

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## BTeV’s Need

- “Given the very complex nature of this system  where thousands of events are simultaneously and asynchronously cooking, issues of data integrity, robustness, and monitoring are critically important and have the capacity to cripple a design if not dealt with at the outset...    BTeV [needs to] supply the necessary  level  of “self-awareness” in the trigger system.”
- [June 2000 Project Review]

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

**Notes:**
Complexity of the event and noise, missing channels, aliasing, etc.

## thus, RTES

- The Real Time Embedded System Group
- University of Illinois
- University of Pittsburgh
- University of Syracuse
- Vanderbilt University (PI)
- Fermilab
- Physicists and Computer Scientists/Electrical Engineers with expertise in
- High performance, real-time system software and hardware,
- Reliability and fault tolerance,
- System specification, generation, and modeling tools.
- NSF ITR grant ACI-0121658

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

**Notes:**
Including both studies of fault management and producing software

## The RTES Solution

- Model Integrated Computing
- Graphical representation of complex system,with modeling (simulation) resources
- ARMORs
- To protect Linux processes
- And sub processors
- VLAs
- To monitor/mitigate at every level
- embedded, supervisory Linux,Linux trigger farm, etc.

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## Modeling Environment: GME*

- Fault handling
- Process dataflow
- HW Configuration

- * GME is an Open-Source, Meta-configurable, multi-aspect graphical modeling tool

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## ARMOR: Adaptive Reconfigurable Mobile Objects of Reliability

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## Very Lightweight Agents

- Minimal footprint
- Platform independence
- Employable everywhere in the system!
- Monitors hardware and software
- Handles fault detection& communications with higher level entities

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## RTES view of the BTeV L1 Trigger

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## SC2003 Prototype 

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## EPICS GUI

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## Independent Review

- Following SuperComputing 2003, a software review was conducted
- GME needs to coherently address multiple, differing domains
- System modeling, messaging, fault mitigation, Run Control function, GUI, other
- ARMORs need to be easily customized
- Via GME
- Overall packaging and version control - vital

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## Domain-specific languages

- GME models, metamodels, and interpreters for
- system description, messaging, state machine (run control, ARMOR), GUI
- Each language generates appropriate artifacts
- C++, Python, Matlab M-files, Elvin config files, etc.

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## Versioning/Build System

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## Demo System 2004 - L2/3 Trigger

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## Demo System 2004

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## Matlab GUI

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## Beyond BTeV - CMS

- GME modeling for XDAQ
- System descriptions, state machines, messaging...
- Work in progress
- Fault tolerance for HLT
- ARMORs and VLAs
- Being discussed
- Balancing CMS needs and RTES goals
- Adding value, without requiring changes

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## Beyond BTeV - LQCD

- Lattice Gauge Theory Computation
- farm at Fermilab
- Single-point sensitivities
- Single process fault can compromise entire farm computation
- Checkpointed; can be restarted, but...
- ARMORs and VLAs
- Batch/autonomous protection
- No operator
- Dynamic mix of protection requirements
- Not a (quasi)static L2/3 Trigger

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## Beyond BTeV - Grid, Other

- Grid Projects
- Load balancing and networks studies
- Nodes-in-farm => farms-in-grid
- Resource driven, deadline driven, other
- Extension of studies done for BTeV/partitioning
- Other - Dark Energy Survey (astro camera)
- “Simple” system (few nodes)
- Not real-time hard (can reacquire image)
- But it will be a good case-study for the “cost” of incorporating RTES (GME, ARMORs, VLAs)

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## Conclusions

- The RTES project developed two prototypes (L1, and L2/3) for BTeV
- Demonstrated at conferences
- RTES is now applying its design-time modeling and runtime middleware to severalhigh performance heterogeneousembedded application environments

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005

## Slide 23

- The RTES Project - BTeV, and Beyond

- M. Haney; RT 2005