# Presentation: 885927

## State of the SARP

- NASA OSMA Software Assurance Research Program Overview
- 9 August 2005
- Kenneth McGill

## Overview

- Why SARP?
- Goals and Objectives
- Recent Evolution and Status
- Metrics
- Where we are going

## Why SARP?

- Because software is different
- Because software will bite you
- Safety
- Mission risk
- Cost and Schedule risk

## Software is Different

## Software is Different

- Component failure follows bathtub curve
- Will function the same over a range of operational scenarios
- Can be completely tested
- No hidden functionality

## Software is Different

## Software is Different

- Failure is event driven (Doesn’t follow a bathtub curve)
- Can have hidden functions
- Near infinite state space
- Can never fully test all scenarios

## Software is Different

- Fully IV&Ved system
- BTW, don’t think you are going to solve your software problems by reuse.  His code was designed for reuse.

## Software is Different

- Software failures are not a function of time but input
- Can’t determine reliability with finite testing
- FMEA and FTA don’t apply straight out of the box
-  Can never test all possible scenarios
- After years of use you can still have unexplored paths
- Safety and mission risks are harder to quantify
- Can’t simply make it more reliable by adding redundancy

## Software is Different

- Software cost and schedule requirements are often under estimated
- Software requirements often get refined (creep)
- Obviously results in development risk
- Also results in
- Shortcuts to make up time
- Truncated testing
- More safety and mission type risk

## Software is Different

- We have both the best and the worst in software development sitting side by side
- Things we knew 20 years ago still aren’t consistently applied
- Too many exceptions to the rules
- Too many scientists and too few engineers

## Software is Different

- We must:
- Define the envelope where we can depend on our software
- Integrate software into systems engineering
- Address both mission and development risk
- Apply what we know

## Software is Different

- In NASA, SARP is the only surviving program focused on software assurance
- There are programs which focus on a particular technology but when all have is a hammer.....

## Goals and Objectives

- Provide software assurance practices, methods, and tools needed to produce safe and reliable software for NASA
- Current and future needs
- Emphasis on transferring technology to NASA missions

## Goals and Objectives

- We use the Oceans, Fish, Fishermen, Hooks paradigm for selection research
- There was a time when we tried to make sure every center got something.

## Oceans

-   Oceans are the topic areas
- Must be relevant and promising
-   Must be software assurance
-   We don’t do project development
-   Must be externally valid
-   Some oceans are fished out

## Fish 

- Fish are data from real NASA projects
- We don’t like toy projects
- We like to fund research that is closely tied to a project
- Someone at a NASA Center has an advantage

## Fishermen

- The researchers
- Don’t have to be a big name
- Need to convince us that they are the right people for the job

## Hooks

- Must have the right tools for the job
- Mathematical techniques
- Formal approaches
- Exiting technologies which could be expanded

## Recent Evolution and Status

- Currently 34 SARP initiatives
- 25 Center Initiatives (CIs) - $150K average
- Centers: ARC, GSC, GSFC, IV&V, JPL, JSC
- Others: UMD, PSU, UVA, Triakis
- 9 University initiatives at WVU - $50K average

## Recent Evolution and Status

- Consecutive 14% cuts in FY05 and FY06 resulted in a total loss of $1.6 million
- 3 initiatives cancelled in FY05
- FY06 selection cancelled (No new starts.)
- This doesn’t mean no research infusion

## Recent Evolution and Status

- Seven initiatives from FY03 will end this year
- One FY03 initiative will be extended through FY07
- Seven FY04 initiatives should end in FY06
- This will make room for new initiatives in FY07

## Recent Evolution and Status

- New FY07 initiatives
- Call for proposal  in February 2006
- Proposals due EOM April 2006
- Selection at SAS06
- Start January 2007
- Note that funding comes through the SMA office at each center.
- Center SMA Directors and IV&V Facility Director should sign off on these in late April
- Will be a NASA only competition

## Recent Evolution and Status

- So what if you are a contractor?
- Most of our research is actually done by industry and universities in association with a NASA Center
- Is it in scope of an existing contract?
- What if you don’t have a contract?

## Recent Evolution and Status

- Make sure you make it very obvious that
- Your proposal is software assurance
- Not development
- You have targeted a NASA software development project to work with
-  You have a clear path to technology transition
- Your results will be externally valid
- Local SMA director has to approved your submission
- Proposal typically for 3 years
- 50K to 300K per year

## Recent Evolution and Status

- WVU is hiring a new NASA Liaison
- IV&V Facility funded at 75%
- $40% Research, 35% NASA interface, 25% Teaching

## Recent Evolution and Status

- Plan to hold mini-SASs at Centers
- I want to know which research results you could use at your center
- We plan to award a “Most Wanted” award at the end of SAS

## Metrics

- SARP Results Web Page
- Performance Reviews
- Penetration Factor
- Letter Grades

## SARP Results Web Site

- _[http://](http://sarpresults.ivv.nasa.gov/)__[sarpresults.ivv.nasa.gov](http://sarpresults.ivv.nasa.gov/)__[/](http://sarpresults.ivv.nasa.gov/)_

## SARP Results Web Site

| File | Downloads | Unique Downloads | Principal Investigator |
| --- | --- | --- | --- |
| Literature Survey of Current V&V Technology.pdf | 1,002 | 878 | Brian Taylor |
| Preliminary Methodology White Paper.pdf | 666 | 601 | Brian Taylor |
| Methods and Procedures for the Independent Verification and Validation of Neural Networks Presentation.ppt | 644 | 593 | Brian Taylor |
| Project Technical Documentation Description Document.doc | 602 | 540 | Ted Bennett |
| Software Metrics as Error Predictors Report.doc | 525 | 475 | Robert M. Chapman |
| Defining Software Quantification Models.doc | 520 | 469 | Carol Smidts |
| Achieving High Software Reliability Technical Report.pdf | 505 | 454 | Taghi M. Khoshgoftaar |
| Achieving High Software Reliability Measurement Data.pdf | 453 | 412 | Taghi M. Khoshgoftaar |
| Report Describing Changes to Space Station PRA.doc | 429 | 387 | Carol Smidts |
| Requirements and Specifications for SMART.doc | 426 | 387 | Taghi M. Khoshgoftaar |

## SARP Results Web Site

- When PRA = ?, researchers need to decide if it should be published
- When PRA = Required, Researchers need to complete a NF 1676

## Performance Reviews

## Penetration Factor

** **

**9****	**results actually
- 	 used by project
**8****	**data passed back 	to project
**7****	**data used by 	researcher
**6****	**data passed
**5****	**project agrees to 	passing data
**4****	**positive response 	to contact
**3****	**project contacted
**2****	**project targeted
**1****	**no project targeted

## Letter Grades

- SARP plus IV&V Facility Research
- “B”s 10
- “B+”s 19
- “A-”s 5
- “A”s 6
- All “C”s and many “B”s fell to budget cuts

## Were are we going

- More interaction with NASA projects
- More involvement from SMA organizations
- Ability to identify software risks like we do with hardware
- PRA
- Cost and schedule risks
- Software engineering vs code development