# Presentation: 434798

## Software Assurance with SAMATE Reference Dataset, Tool Standards, and Studies

- Paul E. Black
- National Institute of Standards and Technology
- _http://samate.nist.gov/_
- paul.black@nist.gov

## What is NIST?

**U.S. National Institute of Standards and Technology**
**A non-regulatory agency in Dept. of Commerce**
**3,000 employees + adjuncts**
**Gaithersburg, Maryland and Boulder, Colorado**
**Primarily research, not funding**
**Over 100 years in standards and measurements: from dental ceramics to microspheres, from quantum computers to fire codes, from body armor to DNA forensics, from biometrics to text retrieval.**

- Paul E. Black

## Software for Dependable Systems: Sufficient Evidence?

**Just published recomendations of 3 year study by National Research Council (NRC)**

**Certifiably dependable software needs assurance cases built on**
**Explicit claims, evidence, and**
**expertise.**
**System engineering including humans as components**
**Testing, static analysis, *****and***** formal methods**

- Paul E. Black

## The NIST SAMATE Project

**Software Assurance Metrics And Tool Evaluation (SAMATE) project is sponsored in part by DHS**
**Began 2004 to help improve software assurance**
**Current areas of concentration**
**Source code security analyzers**
**Studies of tool effectiveness**
**Web application scanners**
***Binary analyzers***
***Software labels***
**Web site     http://samate.nist.gov/**

- Paul E. Black

**Notes:**
Interagency agreement betwieen NIST and DHS

## Web Application Scanning

**Crawls web pages, performs automated penetration testing, and reports weaknesses and vulnerabilities found**
**Specification released**
**Test cases under development**

- Paul E. Black

**Notes:**
In the last 5 years, web application vulnerabilities reported in the NVD rose 5 fold

## Researching Risky Software

**Many people research malware, but there are no widely accepted protocols.**

**Biological research has defined levels with associated practices, safety equipment, and facilities.**
**Some approaches are **
**Weakened programs (auxotrophs)**
**Programs that ****ALERT**
**Outgoing firewalls**
**Isolated networks**

- Paul E. Black

## Static Analysis Tools

- Examine source code or binary for weaknesses, adherence to guidelines, etc.
*We need thousands of examples to assess tools ... *

- Paul E. Black

**Notes:**
Soundness may vary from program “proofs” to heuristics
Autonomy may vary from analysis assistant to fully automated

## SAMATE Reference Dataset (SRD)

- Paul E. Black

## Slide 9

- Paul E. Black

## Slide 10

- Paul E. Black

## SRD Has Search Facilities

- Paul E. Black

## Studies, e.g., Engler’s Question

**# vulnerabilities**

**time fixing weaknesses**
**reported by tools**

- Paul E. Black

## Seeking Participants

**Contribute test cases to **_**SRD**_
**Share software development data**
**Comment on specs and tests for**
**Source code security analyzers**
**Web application scanners**
**Join SAMATE email list with ideas on**
**Static binary analyzers**
**Software labels**
**Malware research protocols**
**contact **
**     ****Paul E. Black****	****	****	**
**     ****SAMATE Project Leader**
**     ****paul.black@nist.gov**

- Paul E. Black