# Presentation: 185686

## Reliability, Availability, and Serviceability (RAS) for High-Performance Computing

- Stephen L. ScottChristian Engelmann
**Computer Science Research Group****Computer Science and Mathematics Division**

## Research and development goals

**Provide high-level RAS capabilities for current terascale and next-generation petascale high-performance computing (HPC) systems**
**Eliminate many of the numerous single points of failure and control in today’s HPC systems**
**Develop techniques to enable HPC systems to run computational ****jobs 24/7**
**Develop proof-of-concept prototypes and production-type ****RAS solutions**

## MOLAR: Adaptive runtime support for high-end computing operating and runtime systems

**Addresses the challenges for operating and runtime systems to run large applications efficiently on future ultrascale high-end computers**
**Part of the *****Forum to Address Scalable Technology for Runtime and Operating Systems (FAST-OS)***
**MOLAR is a collaborative research effort (www.fastos.org/molar)**

## Many active head nodes
Workload distribution
Symmetric replication between head nodes
Continuous service
Always up to date
No fail-over necessary
No restore-over necessary
Virtual synchrony model
Complex algorithms
Prototypes for Torque and Parallel Virtual File System metadata server

**Many active head nodes**
**Workload distribution**
**Symmetric replication ****between head nodes**
**Continuous service**
**Always up to date**
**No fail-over necessary**
**No restore-over necessary**
**Virtual synchrony model**
**Complex algorithms**
- Prototypes for Torque and Parallel Virtual File System metadata server

**Active/active head nodes**

- Symmetric active/active redundancy

**Compute nodes**

## Symmetric active/active Parallel Virtual File System metadata server

**Reading throughput**

| Nodes |  Availability |  Est. annual downtime |
| --- | --- | --- |
| 1 |  98.58% |  5d, 4h, 21m |
| 2 |  99.97% |       1h, 45m |
| 3 |  99.9997% |                1m, 30s |

**0**

**20**

**40**

**60**

**80**

**100**

**120**

**1**

**2**

**4**

**8**

**16**

**32**

**Number of clients**

**Throughput (Requests/sec) **

**PVFS**
**A/A 1**

**A/A 2**
**A/A 4**

## Operational nodes: Pause
BLCR reuses existing processes
LAM/MPI reuses existing connections
Restore partial process state from checkpoint
Failed nodes: Migrate
Restart process on new node from checkpoint
Reconnect with paused processes
Scalable MPI membership management for low overhead
Efficient, transparent, and automatic failure recovery

**Operational nodes: Pause**
**BLCR reuses existing processes**
**LAM/MPI reuses existing connections**
**Restore partial process state ****from checkpoint**
**Failed nodes: Migrate**
**Restart process on new node from checkpoint**
**Reconnect with paused processes**
**Scalable MPI membership management for low overhead**
- Efficient, transparent, and automatic failure recovery

- Reactive fault tolerance for HPC with LAM/MPI+BLCR job-pause mechanism

**Paused MPI **
**process**

**Live node**

**Migrated MPI **
**process**

**Spare node**

**Paused MPI **
**process**

**Live node**

**Failed MPI **
**process**

**Failed node**

**Shared storage**

**Failed**

**Process**
**migration**

**New connection**

**New connection**

**Failed**

**Existing**
**connection**

## 3.4% overhead over job restart, but
No LAM reboot overhead
Transparent continuation of execution

**3.4% overhead over job restart, but**
**No LAM reboot overhead**
**Transparent continuation of execution**

**0**

**1**

**2**

**3**

**4**

**5**

**6**

**7**

**8**

**9**

**10**

**BT**

**CG**

**EP**

**FT**

**LU**

**MG**

**SP**

**Seconds**

**Job pause and migrate**

**LAM reboot**

**Job restart**

**No requeue penalty**
**Less staging overhead**

- LAM/MPI+BLCR job pause performance

## Proactive fault tolerance for HPC using Xen virtualization

**Standby Xen host (spare node without guest VM)**
**Deteriorating health**
**Migrate guest VM to spare node**
**New host generates unsolicited ARP reply**
**Indicates that guest VM has moved**
**ARP tells peers to resend to new host**
- Novel fault-tolerance scheme that acts before a failure impacts a system

## Single node failure

**0**

**50**

**100**

**150**

**200**

**250**

**300**

**350**

**400**

**450**

**500**

**BT**

**CG**

**EP**

**LU**

**SP**

**Seconds**

** **

** **

**Single node failure**

**Single node failure: 0.5–5% additional cost over total wall clock time**
**Double node failure: 2–8% additional cost over total wall clock time**

- VM migration performance impact

**without migration**
**one migration**

**Double node failure**

**BT**

**CG**

**EP**

**LU**

**SP**

**0**

**50**

**100**

**150**

**200**

**250**

**300**

**without migration**
**one migration**
**two migrations**

**Seconds**

## HPC reliability analysis and modeling

**Programming paradigm and system scale impact reliability**
**Reliability analysis**
**Estimate mean time to failure (MTTF)**
**Obtain failure distribution: exponential, Weibull, gamma, etc.**
**Feedback into fault-tolerance schemes for adaptation**

| Exponencial | 2653.3 |
| --- | --- |
| Weibull | 3532.8 |
| Lognormal | 2604.3 |
| Gamma | 2627.4 |

**Negative likelihood value**

**Cumulative probability**

**1**

**0**

**0.2**

**0.4**

**0.6**

**0.8**

**0**

**50**

**100**

**150**

**200**

**250**

**300**

**Time between failure (TBF)**

## Contacts regarding RAS research

-    Scott_RAS_SC07

- Stephen L. Scott
- Computer Science Research GroupComputer Science and Mathematics Division
- (865) 574-3144
- scottsl@ornl.ornl
- Christian Engelmann
- Computer Science Research GroupComputer Science and Mathematics Division
- (865) 574-3132
- engelmannc@ornl.ornl