# Presentation: 509583

## SA @ WV(software assurance research at West Virginia)

**SA @ WV**(software assurance research at West Virginia)

**Kenneth McGill **
- NASA IV&V Facility Research Lead
- 304.367.8300
- Kenneth.McGill@ivv.nasa.gov
**Dr. Tim Menzies Ph.D. (WVU)**
- Software Engineering Research Chair
- tim@menzies,com

## Why, what is software assurance?

- Definition:
- Planned and systematic set of activities
- Ensures that software processes and products conform to requirements, standards, and procedures.
- Goals:
- Confidence that SW will do what is needed when it’s needed.

-     Why software assurance?
- bad software can kill good hardware.
- E.g. ARIANE 5: (and many others)
- Software errors in inertial reference system
- Floating point conversion overflow

**  **Ariane 5

## OSMA Software Assurance Research Program

- Office of Safety & Mission Assurance (Code Q- OSMA)
- Five million per year
- Applied software assurance research
- Focus:
- Software, not hardware
- SW Assurance
- NASA-wide applicability
- Externally valid results; i.e. useful for _MANY_ projects
- Organization:
- Managed from IV&V Facility
- Delegated Program Manager: Dr. Linda Rosenberg, GSFC

## Many projects

**Mega: highest-level perspective**
- e.g. project planning tools like ASK-PETE  [Kurtz]
**Macro:**
- e.g. understanding faults [Sigal, Lutz & Mikulski]
**Micro:**
- e.g. source code browsing [Suder]
**Applied to basic:**
- Applied:
- (e.g.) MATT/RATT [Henry]: support large scale runs of MATLAB
- Basic (not many of these)
- e.g. Fractal analysis of time series data [Shereshevsky]
**Many, many more**
- Too numerous to list
- Samples follow
- See rest of SAS!

**Horn of **
**plenty**

## Many more projects!

- Total proposals:	2.2
- NASA centers: 	1.5
- Industry: 		26
- University: 	 3.7

- Ratio
- FY02/FY01

**Notes:**
IV&V proposals include those by government PI only.  University PIs are included in the University category.  
WVU proposals are not included.

## A survey of 44 FY01 CSIPs

| project | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 to 44 |   |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AATT |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 2 |
| ISS |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 2 |
| Space Shuttle |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 2 |
| ST5 |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 2 |
| Aura |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
| CHIPS |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
| CLCS |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
| CM2 |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
| CMMI |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
| DSMS |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
| EOSDIS |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
| FAMS |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
| GLAST |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
| HSM4 |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
| HST |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
| Mars 07 |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
| Mars 08 |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
| PCS |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
| Space Station |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
| Starlight |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
| Stereo |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
| SWIFT |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
| X-38 |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |
|   | 5 | 4 | 3 | 2 | 2 | 2 | 2 | 2 | 1 | 1 | 1 | 1 | 1 | 0 |  |

- Need more
- transitions!
- (but don’t
- forget the
- theory)

- 75% with no
- claim for
- project
- connections

## Action plan- restructure CSIPS: more transitions!

- New (year 1)
- Fund many
- Renewed (year 2)
- Continue funding the promising new projects
- Recommended: letter of endorsement from NASA project manager
- Transition (year 3)
- Select a few projects
- Aim: tools in the hands of project folks
- Required: project manager involvement

-  Reality check:
-   Transition needs time
-   Data drought

## Long transition cycles

- CO2 + 2H2 —> CH4 + O2

- Marsatmosphere

- oxidizer

- fuel

- on-board

**(no photo)**
** **
**Carmen**
**Mikulski**
*JPL*

**Robyn Lutz**
*JPL, CS-Iowa*
*State*

**Pecheur & ****practical formal methods**
- In-Situ Propellant Production project
- Taught developers:
- Livingstone model-based diagnosis
- model-checking tool tools
- developed by Reid Simmons, (CMU)
- Technology to be applied to the  Intelligent Vehicle Health Maintenance (IVMS) for 2nd generation shuttles
**Lutz, Mikulski &  ****ODC-based analysis of defects**
- Deep-space NASA missions
- Found 8 clusters of recurring defects
- Proposed and validated 5 explanations of the clusters
- Explanations  changes to NASA practices
- ODC being evaluated by JPL’s defect management tool team

## The data drought

***Gasp...***
*** ******need data...***

## End the drought:bootstrap off other systems

**Find the enterprise-wide management information system**** **
**Insert data collection hooks**
- E.g. JPL adding ODC to their defect tracking system
- WVU SIAT sanitizer

## End the drought:Contractors as researchers  

**active data **
**repository**

- Buy N licenses of a defect tracking tool (e.g. Clearquest)
- Give away to projects
- In exchange for their data
- Build and maintain a central repository for that data
- With a web-based query interface
- Data for all

**take me to your data**

## End the drought:Contractors as researchers (2)  

**End the drought:****Contractors as researchers (2)  **

**abstraction**

**action**

**reflection**

**experience**

**1**

**2**

**3**

**4**

**Mark  Suder**
*Titan,  IV&V*

**Hypertext power browser for source code**

**4**

- _SIAT-1_

**}**

**For high-severity errors, recall what SIAT queries**
**lead to finding those errors**

**4’**

**2’**

**Assess each such “power queries”**
**Reject the less useful ones**

**3’**

**Procedures manual for super SIAT or**
**new search options in interface**

- _SIAT_2

**}**

**1’**

**Use it. **

- See also:
- Titan’s new ROI project
- Any contractor proposing an NRA
- Galaxy Global’s metric project

## End the drought:raid old/existing projects

**Cancelled projects with public-domain software**
- E.g. X-34
**Or other open source NASA projects **
- E.g. GSFC’s ITOS:
- real-time control and monitoring system during development, test, and on-orbit operations,
- UNIX, Solaris, FreeBSD, Linux, PC
- Free!!
- NASA project connections:
- Triana,
- Swift,
- HESSI,
- ULDB,
- SMEX,
- Formation Flying Testbed,
- Spartan

## End the drought:synergy groups

- N researchers
- Same task
- Different technologies
- Share found data
- E.g. IV&V business case workers
- E.g. monthly fault teleconferences
- JPL:
- Lutz, Nikora
- Uni. Kentucky:
- Hayes
- Uni. Maryland:
- Smidts
- WV:
- Chapman (Galaxy Global) & Menzies (WVU)

## End the drought:Tandem experiments

**End the drought:****Tandem experiments**

- “Technique X finds errors”
- So?
- Industrial defect detection capability rates:
- TR(min,mean,max)
- TR(0.35, 0.50, 0.65)
- Assumes manual “Fagan inspections”
- Is “X” better than a manual 1976technique?
- Need “tandemexperiments”to check
- I.e. do it twice
- Once by the researchers
- Once by IV&Vcontractors (baseline)

***fictional ***
***data***

## Alternatively:End your own drought

- Our duty, our goal:
- Work the data problem (e.g. see above)
- Goal of CI project year1: build bridges
- But the more workers, the better
- Myth: there is a “data truck” parked at IV&V
- full of goodies, just for you
- Reality: Access negotiation takes time
- With contractors, within NASA
- We  actively assist:
- Each connection is a joy to behold, an occasion to celebration
- We don’t celebrate much
- Bottom line:
- We chase data for dozens of projects
- Researchers have more time, more focus on their particular data needs
- Ken’s law:
- $$$ chases researchers who chase projects
- CI year2, year3: needs a project connection

## Alternatively (2), accept the drought and sieve the dust

**The DUST project:**
- Assumes a few key options control the rest
**Methodology:**
- Simulate across range of options
- Data dust clouds
- Too many options: what leads to what?
- Summarize via machine learning
- Condense dust cloud
- Improve mean, reduce variance
**Case studies:**
- JPL requirements engineering:
- Feather/JPL [Re02]
- Project planning:
- DART- Raque/ IVV;  Chaing/UBC;
- IV&V costing: Marinaro/IVV, Smith/WVU
- general:   Raffo, et.al/PSU [Ase02]
- An analysis of pair programming: Smith/WVU
- Better predictors for:
- testability: Cukic/WVU, Owen/WVU [Issre02, Ase02]
- faults: diStefano/WVU, McGill/IVV; Chapman/GG
- reuse :  diStefano/WVU [ToolsWithAI02]

**Each dot  = **
**1 random **
**project plan**

**The answer my friend,  is blowin’ in the wind**

**But wait:  the times they are changing**

## Other WVU SA research

**Other WVU SA research**

- Architectural
- descriptions

**Fault**,
**failure**
- data on
- components,
- connectors

- Software
- Specs & design
- (early life cycle)

**Code analysis**
**(iv&v,operational **
**usage)**

- Metrics(complexity,coupling,entropy** **)

**Failure data from testing**

- Severity of failures

- UML (sequence
-  diagrams,
- state charts)

**UML simulations**

- Static (SIAT,
- Mccabe, entrophy)

**Dynamic (testing, **
**runtime monitoring)**

-   **Testing & formal methods**
**     ******** ****Bayesian approach to reliability**
**        ******** ****Architectural metrics**
**            ********Risk assessment & dynamic UML**
**                 ******** ****Reliability & ****                           operational profile errors**

**Hany Ammar**

**Bojan Cukic**

**collaborator**** **

**Goal: accurate,**** stable, risk ****assessment **
**early in the ****lifecycle**

## More WVU research (FY02 UIs)

**Architectural metrics**
**Risk assessment & dynamic UML**
*Intelligent flight controllers*
**	**
**Testing & formal methods**
**Bayesian approach to reliability**
*Fractal study of resource dynamics*
**Reliability & operational profile errors**
**SE research chair**
**interns**
**DUST**

**Ammar**

**Cukic**

**Goseva-**
**Popstojanova**

**Menzies**

**new**
**renewed**

**c****	****= conference   ****w****	****= workshop**
**j****	****= journal**

- ISS hub controller,
- “Dryden application”

- F15

- “JPL deep space mission”
- DART
- “KC-2”
- IVV cost models

- SIAT

- X34

- ITOS

- X38

**jj**

**j, ccccccc, w**

**c**

**cccccc**

**jc**

**c**

**w**

- FY03 proposals = 2.2*FY02

## Function Point Metrics for Safety-Critical Software

**Thesis:**
- Traditional function-point cost estimation
- Incorrect for safety-critical software
**> 1 way to skin a cat**
- >1 way to realize a safety critical function:
- NCP=N-copy programming
- NVP= N-Version Programming ,
- NSCP= N Self-Checking Programming,
-  ...
- With, without redundancy,
**Method: **
**explore them all! **

**H2 and C2 : effort & cost, redundant system****H1 and C1: effort & cost, non-redundant system**

**Afzel  Noore**

## Pre-disaster warnings  [Cukic, Shereshevsky]

**Can we defer a maintenance cycle and keep doing science for a while longer?**

**Mark**** Shereshevsky**

- Crash

**Bojan Cukic**

**ARTS II**** **

## Intelligent flight controllers [Napolitano, Cukic] (and menzies)

**Marcello Napolitano****(Mechanical and****Aerospace****)**

**Bojan Cukic**
**(CSEE)**

**Lifecycle opportunities for V&V of neural network based adaptive control systems.**

## The road ahead: applied & theoretical research

**The road ahead: applied & theoretical research**

**CSIPs: applied  research**

**USIPs: applied + theoretical research**

**Need both**

**To boldly go...**