# Presentation: 434801

## Using a Large-Scale Survivability Architecture to ControlGrids: A Status Report

**Zach Hill*, Jonathan Rowanhill*, Jim Basney**, Glenn Wasson*, John Knight*, Anh Nguyen-Tuong*, Andrew Grimshaw* and Marty Humphrey*,**
- *Department of Computer Science, University of Virginia, Charlottesville, VA
- **NCSA/University of Illinois, Urbana-Champaign

**Supported in part by the National Science Foundation under grant SCI-0426972**

## Definitions

**Reliability**
- An attribute of any system that consistently produces the same results, preferably meeting or exceeding its specifications
- In engineering in general, reliability is the capacity of a component or a system of such components to perform as designed
_**Robustness**_
- ... an ability to recover gracefully from the whole range of exceptional inputs and situations in a given environment
- In terms of computer software, refers to the resilience of the system when under stress - for example, when running a large number of processes, or when starved of memory or storage space, or when confronted with an application that has bugs or is behaving in an illegal fashion

## Problem Statement

**To date: Grids used to aggregate resources for scientific applications**
_***Problem:***_** only tolerant users can use “The Grid”**
_**Survivability**_**: alternate services (different, less dependable, degraded) if primary services are not available due to attack or failure**
- Each alternative can deal with a different class of faults
- Note: _*cannot*_ re-engineer from scratch
**Goal: Grids as *****critical infrastructure***

## Slide 4

## Challenges

**How do you implement a single service to meet a particular dependability/survivability requirement?**
**How do you implement service + alternative services to meet a particular requirement?**
**How do you **_**dynamically**_** compose services to meet overall dependability/survivability requirements?**
**How do you monitor/reconfigure the Grid infrastructure for dependability/survivability?**
***How do you do all of this ***_***across admin domains?***_

## Approach

**Support/Influence OGSA (and Web services)**
**Leverage experience**
- UVa: Legion, OGSI.NET / WSRF.NET
- UVa: Willow
- NCSA: TeraGrid/NCSA operations, MyProxy
**Two major efforts**
- Per-service Dependability characteristics (XML)
- Application of Survivability architecture to Grids

## Dependable Applications in Service-Oriented Architectures (SOAs)

## Willow: Survivability As Control 

- Control Function

- “Sensor” Signals

- “Actuator” Commands

- From Sensors

- To Actuators

##  Willow Architectural Issues

*** ******Willow Architectural Issues***

**Hierarchic faults**
**Control loop interactions:**
- Asynchronous
- Priority & resources
- Conflicting goals
**Network scale:**
- Volume of software
- State model
- Wide area change
**Exceptions and results:**
- Dynamic app’l membership
- Absolute vs. statistical
- Result “harvesting”
**Target system actuation:**
- Lightweight
- Standard interface & protocol

- Network State &Analysis Model

- External Input

## Focus: Monitoring/Reconfiguring

**As grids grow in size we must find a way to monitor and manage them**
- Must maintain site-autonomy
**Fast-reaction systems are required**
- Viruses and Worms can propagate across the entire internet in minutes or hours.
- In grids with bursts of high load (i.e. LHC output) failure can cause loss of unrecoverable data.
- Real-Time services in grids (i.e. visualization and collaboration) must minimize service interruptions.  Particularly if grids are to become woven into critical infrastructure.

## Our Solution: ANDREA and Spartan

**Use a real-time sensing and actuation system with potentially complex rules to monitor and manage the grid both locally and globally**
**ANDREA and Spartan are components of the Willow system **
- ANDREA is a _distributed actuation system _that handles workflows and decentralized control while allowing node-level autonomy.
- Spartan is a _distributed sensing system _composed of finite-state machines which implement the sensing rules based on sets of events.

## Architecture

- Global Node

- Compute
- Node

- Compute
- Node

- ANDREA
- Controller

- Spartan
- Node

- ANDREA
- Controller

- Spartan
- Node

- ANDREA
- Controller

- Spartan
- Node

- ANDREA
- Controller

- ANDREA
- Controller

- Site Node

- Latency Events

- Workflows

- Site Node

- “High Latency” events

- Workflows

- Compute
- Node

- ANDREA
- Controller

- Compute
- Node

- ANDREA
- Controller

- Compute
- Node

- ANDREA
- Controller

- Compute
- Node

- ANDREA
- Controller

## Experiment

**A VO is not getting its work done in a timely fashion**
**Basic premise is that one site cannot handle its computational load **
- Priorities or resources must be reconfigured to help the grid meet overall latency requirements
**The site tries to decrease latency locally **
**If that doesn’t work then the global controller is activated to reconfigure/reprioritize the entire grid to reduce latency**

## “High Latency” Event count crosses threshold.  Local (NCSA site) workflow enacted.

**“****High Latency” Event count crosses threshold.  Local (NCSA site) workflow enacted.**

**Workflow lifetime expires (artificially)**

**Global workflow enacted **
**(delay at NCSA but not at UVA)**

## Summary

**Grids must become **_**robust**_**, *****critical infrastructure***
**Application and extension of Willow survivability architecture shows good promise**
**Next Steps**
- More comprehensive use-cases (and subsequent implementation)
- Looking for concrete deployment
- More site autonomy
- What if we can’t get all monitoring info from a site?
- What if a site says “no” to an actuation?