# Presentation: 748338

## Software Assurance Metrics and Tool Evaluation

- Paul E. Black
- National Institute of Standards and Technology
- _[http://www.nist.gov/](http://www.nist.gov/)_
- paul.black@nist.gov

## Outline

**Introduction**
**SRD project**
**SAMATE project structure**
**Future – where do we go now?**

- Paul E. Black

## What is Software Assurance?

**Activities that ensures that software processes and products conform to requirements. **
**after NASA Software Assurance Guidebook**
**Two legs of good software**
**Good Development**
**Good Checking**
**Testing (dynamic)**
**Analysis (static)**

- Paul E. Black

**Notes:**
None of these divisions are strict.  For instance, unit testing is used during development, and results of integration testing can feed into the next round of development.  Extreme programming in particular mixes checking and development even more intimately.  Even within checking, the line between running a program and analyzing it blurs.
We will concentrate on good checking, so our use of the term should be “post-development Software Assurance activities”.

## NIST’s role

**What is NIST?**
**A non-regulatory agency in Dept. of Commerce**
**3,000 employees in Maryland and Colorado**
**Primarily research, not funding**
**Why NIST?**
**Over a century of experience in standards and measurement**
**Involved in security: DES, AES, NVLAP, etc.**
**Trusted, neutral 3rd party**

- Paul E. Black

## The Two Projects

- SAMATE
- Software Assurance Metrics
- And Tool Evaluation

- SRD
- Standard Reference
- Dataset

- Paul E. Black

## Outline

**Introduction**
**SRD project**
**SAMATE project structure**
**Future – where do we go now?**

- Paul E. Black

##         SRD Project Goals

**Identify classes of security flaws and vulnerabilities**
**Identify classes of software security assessment techniques**
**Document state of the art**
**Develop a Standard Reference Dataset (SRD) of clean programs and programs with security flaws**

- Paul E. Black

##         SRD Characteristics

**Small test cases for each flaw**
**Separate “can detect” from speed issues**
**Flawed programs and their clean counterparts**
**Very large test cases**
**Confirm speed and maximum size**
**Test cases taken from actual code**
**Nobody can say it would never happen**
**Many different subsets**
**Java, C, web app, OS, Windows, Unix, etc.**
**Ongoing development and additions**
**Submissions from NIST, researchers, & vendors**
**Readily usable**

- Paul E. Black

##         SRD Project Plans

**Small workshop**
**     ****http://samate.nist.gov/softSecToolsSOA**
**10 & 11 August at NIST**
**Publish proceedings as NIST Special Publication & put in ACM Digital Library**
**Write journal article on **
**classes of known software security vulnerabilities and **
**the state of the art of security SA tools**

- Paul E. Black

## Outline

**Introduction**
**SRD project**
**SAMATE project structure**
**Future – where do we go now?**

- Paul E. Black

## DHS Software Assurance Plan

**1. ****	****PEOPLE  (Education/Training)**
***Software Developer focused training and education***
**2. PROCESS  (Lifecycle, Best Practices, Standards)**
***Security throughout the Software Development Life Cycle***
**3. TECHNOLOGY  (Tools and R&D)**
***SA tools identification, enhancement, and development***
**4. ACQUISITION (SOW / Procurement language)**
***Embed security requirements in procurement stage***

- Paul E. Black

## The SAMATE Project

**http://samate.nist.gov/**
**Compendiums (ongoing)**
**Tools**
**Researchers and companies**
**Workshops**
**Aids for tool evaluation**
**Software metrics**

- Paul E. Black

**Notes:**
The surveys help us with two things: (1) find what tools (and techniques) are available and (2) who to talk to. “Tools” eventually includes services, too. But services are harder to qualify. We are interested both in “academics” who are doing research and in commercial people and companies developing and promoting SA tools.

All this information will be available on a web site. Some may be published in journals, conferences, etc. Surveys will be kept up to date.

## Workshops

**Taxonomy of SA functions and techniques**
**Approach (code vs. spec, static vs. dynamic)**
**Software type (distributed, real time, secure)**
**Type of fault detected**
**Which are the most “important”?**
**Highest cost/benefit ratio?**
**Finds highest priority vulnerabilities?**
**Identify gaps in SA functions and write research agenda**
**Plan and initiate studies for metrics**
***      ******first workshop in Long Beach, Nov, w/ASE***

- Paul E. Black

## Purposes of SA Tool Evaluations

**Precisely document what a tool does (and doesn’t) do**
***         ... ******in order to ... ***
**Provide feedback to tool developers**
**Simple changes to make**
**Directions for future releases**
**Inform users**
**Match the tool to a particular situation**
**Understand significance of tool results**
**Guide research for next tool generation**

- Paul E. Black

## Developing a Specification

**After tool function selection approved by working group, ...**
**NIST develops a specification for the function with focus group input**
**Spec posted to web for public comment**
**NIST develops the tests**
**Detailed plans**
**Scripts**
**Standard Reference Dataset**

- Paul E. Black

**Notes:**
Focus Group DEFINES the specific behavior for that SA function in the tool function specification

## Outline

**Introduction**
**SRD project**
**SAMATE project structure**
**Future – where do we go now?**

- Paul E. Black

## Toward Software Metrics

**Qualitative comparison**
**Formally defined quantity**
**Unit and scale**
**Measured value**
**Derived units**

| warmer, colder | buggy, secure |
| --- | --- |

| temperature | quality? confidence? |
| --- | --- |

| degree, kelvin |             ? |
| --- | --- |

| Heat energy=smt | Software assurancept |
| --- | --- |

- Paul E. Black

**Notes:**
Maybe the unit of software confidence could be “lives” as in “I’d bet the lives of N people on this”.

Heat energy (Q) = specific heat (s) 



## Tool Effectiveness Metrics

**Do they really find vulnerabilities and catch bugs? **
**In other words, how much assurance does running a tool provide?**
**“****Create studies and experiments to measure the effectiveness of tools”**

- Paul E. Black

## Call for Participation

**Define classes of flaws and vulnerabilities**
**Contribute to collections of tools, researchers, ...**
**Help define classes of SA functions**
**Decide order of importance of functions**
**Participate in focus group to specify a function**
**Contribute to standard reference dataset**
**Develop metrics to assess software and tools**
**Help set research agenda**

- Paul E. Black

## Society has 3 options:

**Learn how to make software that **_***works***_
**Limit size or authority of software**
**Accept failing software**

- Paul E. Black

**Notes:**
“Size” means size, complexity, style, language features, etc.: anything which keeps us from assuring the quality of the software.

Subject: [sci.space.tech] Re: Delta vs TitanOn Sat, 08 Jan 2005 07:14:45 +0000, Henry Spencer wrote:> The software for the MOST> astronomy satellite doesn't get anywhere near such elaborate attention.> But then, the software basically can't do anything to break MOST.  And no,> that's not an accident -- we worked hard to make sure of that!> > For example, on a typical satellite, loss of attitude control is a> terrible emergency, because it means the solar arrays are no longer> pointed at the Sun and you've only got (at most) a few hours until the> batteries are drained.  Spacecraft have died that way. 
> > MOST was in a random tumble for a month after launch, while some other> problems were solved.  Nobody worried.  The spacecraft has solar arrays on> all six sides -- not enough for full power, but enough for crucial> functions (main computer and radio receivers).  And when the batteries> approach empty, the power system hardware switches off everything except> crucial functions.  (It tries to switch off everything, but there are no> power switches on the computer and the receivers.)> > The result is software that's *far* cheaper to develop.  Also a more> robust spacecraft:  even the inordinately expensive classical process can> *and does* make mistakes, which can kill spacecraft.  MOST is essentially> unkillable.  (Credit where due:  we got this approach from the Amsat> people, who do the amateur-radio satellites.)> > You couldn't quite do that for things with time-critical functions, like> aircraft and launchers.  Even there, though, it would be interesting to> see how far you *could* get in designing the total system to survive> imperfect software, rather than assuming the software must be perfect and> will be made so regardless of cost.