# Presentation: 940400

## The Formal Verification of SPIDER

- Lee Pike
- Department of Computer Science
- Indiana University, Bloomington
- lepike@indiana.edu

## Thanks to

- Steven Johnson,                                                                       Indiana University, Bloomington
- The National Institute of Aerospace
- The NASA LaRC Formal Methods Team,                                     especially Paul Miner

## Overview

- SPIDER Overview
- Reasoning about Faults
- The Old vs. New Interactive Consistency (IC) Protocol
- SPIDER Formal Verification Goals & Future Work
- References

## SPIDER OverviewWhy?

- Develop a fault-tolerant architecture based on an ultra-reliable bus
- Scalable
- Handle a large number of possibly-simultaneous faults, specifically transient faults from electromagnetic effects.
- Provide reintegration services
- Case study for the FAA
- Developed in accordance with *RTCADO-*254*: Design Assurance Guidance for Airborne Electronic Hardware*.
- Provide a test-bed for techniques in the specification and verification of safety-critical electronic systems.

**These sort of architectures are the foundation of tomorrow's X-by wire safety-critical systems.**

## SPIDER OverviewWhat?

- Scalable Processor-Independent Design for Electromagnetic Resilience

## SPIDER OverviewWhat?

- Scalable Processor-Independent Design for Electromagnetic Resilience
- Processor Elements (PEs)

- PE

- PE

- PE

## SPIDER OverviewWhat?

- Scalable Processor-Independent Design for Electromagnetic Resilience
- Processor Elements (PEs)
- Reliable Optical BUS (ROBUS)
- Time Division Multiple Access (TDMA) bus
- Maintains Synchrony between PEs.
- Prevents Babbling Idiots & PE-to-PE interference
**The services of the ROBUS are the focus of the verification effort.**

- ROBUS

- PE

- PE

- PE

## ROBUS OverviewTopology

- BIU1

- to PE

- to PE

- to PE

- BIU2

- BIU3

- RMU1

- RMU2

- ROBUS

- RMU3

*n* Bus Interface Units (BIUs)
*m* Redundancy Management Units (RMUs)
- The BIUs and RMUs are called *nodes*.
- Every BIU and RMU is directly connected.
- No two BIUs are directly connected.  Similarly for the RMUs.

## ROBUS OverviewServices (Protocols)

- Interactive Consistency                                                              Purpose: Reliably broadcast messages between PEs.
- Clock Synchronization                                                              Purpose: Maintain synchrony between all nodes and PEs.
- Distributed Diagnosis                                                                Purpose: Convict faulty nodes in the ROBUS.

- The focus of this talk is Interactive Consistency.

## Global Fault Classifications

- Good Not faulty

- node

- d

- d

- d

## Global Fault Classifications

- Good Not faulty
- Benign Broadcasts only detectably faulty messages

- node

- garbage

- garbage

- garbage

## Global Fault Classifications

- Good Not faulty
- Benign Broadcasts only detectably faulty messages
- Symmetric Broadcasts the same arbitrary message to all

- node

- d'

- d'

- d'

## Global Fault Classifications

- Good Not faulty
- Benign Broadcasts only detectably bad messages
- Symmetric Broadcasts the same arbitrary message to all
- Asymmetric (Byzantine) Arbitrarily sends arbitrary messages

- node

- d'

- d''

- d

## Local Fault InformationEach Node Maintains

- Accusations A node accuses other nodes based on the messages it receives as well as indirect information.

## Local Fault InformationEach Node Maintains

- Accusations A node accuses other nodes based on the messages it receives as well as indirect information.
- Convictions Periodically, the distributed diagnosis protocol is executed; nodes exchange accusations to produce convictions.
- NOTE: While a good node knows that all good nodes have the same convictions, it does not know that all good nodes have the same accusations.

## Local Fault InformationEach Node Maintains

- Accusations A node accuses other nodes based on the messages it receives as well as indirect information.
- Convictions Periodically, the distributed diagnosis protocol is executed; nodes exchange accusations to produce convictions.
- NOTE: While a good node knows that all good nodes have the same convictions, it does not know that all good nodes have the same accusations.
- Eligible Voters For each BIU, the set of RMUs that it neither accuses nor convicts.  Similarly for each RMU.

## Interactive Consistency ProtocolExternal View

- Purpose: Reliably communicate data between processing elements (PEs) over the ROBUS.

- ROBUS

- PE

- PE

- PE

## Interactive Consistency ProtocolExternal View

- A PE sends its data to the ROBUS.

- ROBUS

- PE

- PE

- PE

- sender

- data in

## Interactive Consistency ProtocolExternal View

- The IC Protocol is executed in the ROBUS.

- ROBUS

- PE

- PE

- PE

- ...IC Protocol...

## Interactive Consistency ProtocolExternal View

- The ROBUS broadcasts data back out to the PEs.

- ROBUS

- PE

- PE

- PE

- sender

- data out

- ...IC Protocol...

- data out

- data out

## Old Interactive Consistency ProtocolInternal View

- BIU1

- to PE

- to PE

- to PE

- BIU2

- BIU3

- RMU1

- RMU2

- sender

- data in

- ROBUS

- RMU3

## 1.   A BIU broadcasts data to the RMUs.	If the BIU is good, the same value is broadcast to all RMUs.

- BIU1

- to PE

- to PE

- to PE

- BIU2

- BIU3

- RMU1

- RMU2

- sender

- data in

- ROBUS

- data

- data

- RMU3

- data

## 2.   For each good RMU, if it receives data that isn't detectably faulty,        then it passes the data received back to each BIU.  Otherwise,               source_error is sent.

- BIU1

- to PE

- to PE

- to PE

- BIU2

- BIU3

- RMU1

- RMU2

- ROBUS

- RMU3

- similarly for
- RMUs 2 and 3

- data or
- source_error

- data or
- source_error

- data or
- source_error

- RMU1 good

## 3.   Each BIU eliminates from its EV those RMUs that sent detectably       faulty messages.  

- BIU1

- to PE

- to PE

- to PE

- BIU2

- BIU3

- RMU1

- RMU2

- ROBUS

- RMU3

- 2

- 1

- 3

- BIUs 2 and 3 do likewise

- d

- d

- garbage

- RMU1 good

- RMU2
- benign faulty

## 4.   For each BIU, it votes on the majority data sent from each                    RMU in its EV.

- BIU1

- to PE

- to PE

- to PE

- BIU2

- BIU3

- RMU1

- RMU2

- ROBUS

- RMU3

- 2

- 1

- 3

- BIUs 2 and 3 do likewise

- vote = d

- d

- d

## 5.   IF the majority of RMUs sent the same data, then it is sent to the      BIU's PE.      ELSE source_error is sent to the BIU's PE.

- BIU1

- to PE

- to PE

- to PE

- BIU2

- BIU3

- RMU1

- RMU2

- ROBUS

- RMU3

- BIUs 2 and 3 similarly send data

- vote = d

- d

## IC Protocol Guarantees

- Validity If the broadcasting BIU is good, not convicted, and sends data *d*, then the result of the vote for a good BIU is be *d*.
- Agreement Any two good BIUs vote the same result for the broadcasted value (even if the sender is asymmetric!).

## Old Assumptionsto ensure guarantees hold

**Environment Assumptions**
- The Maximum Fault Assumption (MFA):
-  There are more good BIUs than symmetric + asymmetric BIUs.
-  Similarly for the RMUs.
-  There are either no asymmetric BIUs or no asymmetric RMUs.

## Old Assumptionsto ensure guarantees hold

**Environment Assumptions**
- The Maximum Fault Assumption (MFA):
-  There are more good BIUs than symmetric + asymmetric BIUs.
-  Similarly for the RMUs.
-  There are either no asymmetric BIUs or no asymmetric RMUs.

**System Assumptions**
- Symmetric Agreement If a node is not asymmetric, then all good nodes assign it the same accusation.
- Good Trusting Good nodes aren't accused by good nodes.
- Conviction Agreement All good nodes have the same convictions.

## ValidityProof Sketch

- Assume the broadcasting BIU is good and sends data *d*.

- BIU1

- BIU2

- BIU3

- RMU1

- RMU2

- sender  good

- ROBUS

- d

- d

- RMU3

- d

## ValidityProof Sketch

- BIU1

- BIU2

- BIU3

- RMU1

- RMU2

- ROBUS

- RMU3

- similarly for
- RMUs 2 and 3

- d

- d

- d

- Thus, all good RMUs send* d* back to the BIUs.* *

- RMU1 good

## ValidityProof Sketch

- BIU1

- BIU2

- BIU3

- RMU1

- RMU2

- ROBUS

- RMU3

- 2

- 1

- 3

- d

- d

- Each good BIU filters out the bad messages received.  By the MFA, most of its EV then contains good RMUs.

- garbage

- similarly for
- BIUs 2 and 3

## ValidityProof Sketch

- BIU1

- BIU2

- BIU3

- RMU1

- RMU2

- ROBUS

- RMU3

- 2

- 1

- 3

- vote = d

- d

- d

- Since all good RMUs sent* d*, the result of the vote yields *d*.       **q.e.d.**

## AgreementProof Sketch

- BIU1

- BIU2

- BIU3

- RMU1

- RMU2

- sender  asym

- ROBUS

- d

- d''

- RMU3

- d'

- Either the broadcasting BIU is asymmetric or not.  Suppose it is.

## AgreementProof Sketch

- ROBUS

- BIU1

- BIU2

- BIU3

- RMU1

- RMU2

- RMU3

- Then no RMU is asymmetric, by the MFA.  So every RMU sends the same data to every BIU.

- 2

- 1

- 3

- x

- z

-  y

- BIUs 2 and 3 receive the same values

## AgreementProof Sketch

- ROBUS

- BIU1

- BIU2

- BIU3

- RMU1

- RMU2

- RMU3

- Since no RMU is asymmetric, by symmetric trusting, the EV of each BIU is the same.  Thus, the result of the vote for each BIU is the same.

- 2

- 1

- 3

- x

- z

-  y

- BIUs 2 and 3 receive the same values

## AgreementProof Sketch

- BIU1

- BIU2

- BIU3

- RMU1

- RMU2

- sender  not asym

- ROBUS

- d

- d

- RMU3

- d

- For the other case, suppose the sending BIU is not asymmetric.

## AgreementProof Sketch

- ROBUS

- BIU1

- BIU2

- BIU3

- RMU1

- RMU2

- RMU3

- Most of the RMUs are good, by the MFA.  Since all good RMUs received the same values, they send the same values.

- RMU1 good

- RMU3 good

- BIU1 good

- BIU3 good

- x

- x

## AgreementProof Sketch

- ROBUS

- BIU1

- BIU2

- BIU3

- RMU1

- RMU2

- RMU3

- By good trusting, no good BIU accuses a good RMU.  Since most RMUs are good, there are a majority of good RMUs in the EV of each good BIU, after filtering benign RMUs.

- RMU1 good

- RMU3 good

- 2

- 1

- 3

- x

- 2

- 1

- 3

- x

- x

- x

- BIU1 good

- BIU3 good

## AgreementProof Sketch

- ROBUS

- BIU1

- BIU2

- BIU3

- RMU1

- RMU2

- RMU3

- Thus, the result of the votes will be the same for all good BIUs.
-                                                                                             **q.e.d.**

- RMU1 good

- RMU3 good

- 2

- 1

- 3

- x

- 2

- 1

- 3

- x

- x

- x

- BIU1 good

- BIU3 good

## New Assumptionsto reason about reintegration

**Environment Assumptions**
- The Dynamic Maximum Fault Assumption (DMFA):
-  For each good BIU, its EV consists of more good RMUs than        symmetric + asymmetric RMUs.
-  Similarly for good RMUs.
-  Either no asymmetric RMU is in the EV of a good BIU or no         asymmetric BIU is in the EV of a good RMU.

## New Assumptionsto reason about reintegration

**Environment Assumptions**
- The Dynamic Maximum Fault Assumption (DMFA):
-  For each good BIU, its EV consists of more good RMUs than        symmetric + asymmetric RMUs.
-  Similarly for good RMUs.
-  Either no asymmetric RMU is in the EV of a good BIU or no         asymmetric BIU is in the EV of a good RMU.

**System Assumptions**
- Symmetric Agreement If a node is not asymmetric, then all good nodes assign it the same accusation.
- Good Trusting Good nodes aren't accused by good nodes.
- Conviction Agreement All good nodes have the same convictions.

## Agreement Breaks!Under the New Assumptions (courtesy of Wilfredo)

- ROBUS

- BIU1

- BIU2

- BIU3

- RMU1

- RMU2

- RMU3

- Suppose the sender is asymmetric, but is in no EV of all good RMUs.  Suppose there is an asymmetric RMU in the EV of both good BIUs.  This satisfies the DMFA.

- asym

- good &
- accuses BIU2

- good &
- accuses BIU2

- sender asym

- d

- d'

- d''

- good & trusts all

- good & trusts all

## Agreement Breaks!Under the New Assumptions

- ROBUS

- BIU1

- BIU2

- BIU3

- RMU1

- RMU2

- RMU3

- The two good RMUs relay the values received, and since RMU3 can relay arbitrary data, it sends *d* to BIU1 and *d'* to the other.

- asym

- good &
- accuses BIU2

- good &
- accuses BIU2

- sender asym

- 2

- 1

- 3

- d

- d

- 2

- 1

- 3

- d

- d'

- good & trusts all

- good & trusts all

- d'

- d'

## Agreement Breaks!Under the New Assumptions

- ROBUS

- BIU1

- BIU2

- BIU3

- RMU1

- RMU2

- RMU3

- The result of the votes of BIU1 and BIU2 differ.  Agreement is violated!

- asym

- good &
- accuses BIU2

- good &
- accuses BIU2

- sender asym

- 2

- 1

- 3

- d

- d

- 2

- 1

- 3

- d

- d'

- good & trusts all

- good & trusts all

- d'

- d'

- vote = d

- vote = d'

## Revised IC Protocol

- In the new IC Protocol, the RMUs relay source_error when
- They receive bad messages and
**They accuse the sender.**

## Revised IC Protocol

- In the new IC Protocol, the RMUs relay source_error when
- They receive bad messages and
**They accuse the sender.**

- The revised IC protocol satisfies both validity and agreement           (verified in PVS).

## Formal VerificationWhy Level 3 Verification?

- A math proof is proof enough, right?
- Level 3 verification can require significant time to complete.

- In other words...

## Using PVS

## Formal VerificationWhy Level 3 Verification?

- A math proof is proof enough, right?
- Level 3 verification can require orders of magnitude more time to complete than level 1 or level 2 verification.

- But...
- Proofs for fault-tolerant protocols for distributed architectures are tedious and large (there are nearly 400 lemmas & theorems in our current *unfinished* set of proofs).
- Proofs are not checked by a community of mathematicians like other mathematical results are.

- In other words...

## You don't have to be a Laurel or Hardy to make an oversight in an informal proof.

- Small changes in assumptions can obviate guarantees.

## Some Goals & Current Workin verifying SPIDER 

**Robust Specifications/Proofs**
- Hold for arbitrary configurations of SPIDER
- Hold for **all** accusation & conviction policies satisfying the system requirements

## Some Goals & Current Workin verifying SPIDER 

**Robust Specifications/Proofs**
- Hold for arbitrary configurations of SPIDER
- Hold for **all** accusation & conviction policies satisfying the system requirements
**Specification/Proof “Reuse” (Economic specs/proofs)**

## Some Goals & Current Workin verifying SPIDER 

**Robust Specifications/Proofs**
- Hold for arbitrary configurations of SPIDER
- Hold for **all** accusation & conviction policies satisfying the system requirements
**Specification/Proof “Reuse” (Economic specs/proofs)**
**Specification/Proof Hierarchy**
- Property specifications
- Relational specifications
**Functional composition specifications**
- State machine specifications

## References

- SPIDER Homepage:                          http://shemesh.larc.nasa.gov/fm/fm-now-spider.html.
- PVS Homepage:                                                    http://pvs.csl.sri.com/.
*Butler, Ricky et al.  NASA Langley's Research and Technology-Transfer Program in Formal Methods.  2000. * Available athttp://shemesh.larc.nasa.gov/fm/fm-welcome.html.
- Rushby, John.  *Formal Methods and Digital Systems Validation for Airborne Systems.  *NASA Contractor Report 4551.  1993.  Available at:  http://www.csl.sri.com/papers/csl-93-7/.