# Presentation: 515841

## Is software still a problem?

- Kelly J. Hayhurst
- NASA Langley Research Center
- 1 South Wright Street, MS 130
- Hampton, VA  23681-2199
- k.j.hayhurst@larc.nasa.gov
- 757-864-6215

- Is software *still* a problem*?*

- 2003 Formal Methods Workshop
- October 22, 2003

**Notes:**
A lot of work in formal methods is devoted to verification of requirements, protocols, algorithms and such -- things that happen very often in concept development or very early in the product development process 

## System context

**System context**

- System Concept

- System Requirements

- Software
- 	Requirements
- 	Design
- 	Code
- 	Test

- Hardware
- 	Requirements
- 	Design
- 	Build
- 	Test

- System Test

- System Integration

## Context

- Growing dependence on software in safety-critical applications
- both airborne & ground-based systems
*just think about unmanned and personal air vehicles!*

- Yet, we have a difficult time
- getting software done
- through certification and into service
- getting software done right
- such that it does what we intend it to do
- establishing a credible level of confidence that the software will do what it is intended to do
- and that it won’t do anything that is unintended

## Getting it done & done right

**Getting it done & done right**

- - from *Status of FAA’s Major Acquisitions, *FAA, Report #: AV-2003-045, June 26, 2003

- FAA-sponsored programs to modernize have had significant cost & schedule overruns

**Notes:**
WAAS:  Ground-based navigation and landing system designed to improve the accuracy and integrity of GPS data
built by a CMM Level 5 organization
estimated 470K lines of delivered code
compliance with DO-178B, up to Level B

STARS:  New air traffic controller workstations with new color displays, processors and software
to replace outdated equipment for air traffic control at over 160 facilities
should allow for easy incorporation of new ATM tools

## We see it in the military

**Code Red Emergency**

- “With the threat of continued software problems derailing U.S. Air Force plans for the F/A-22 stealth fighter, prime contractor Lockheed Martin has devised a way to gradually overcome long-standing glitches that have led to repeated crashes of onboard computers.”

- - Robert Hall, Aviation Week & Space Technology, June 9, 2003

- “the greatest challenge in the F/A-22 development program is one of software integration, which has resulted in a software instability ...” a quote in the article from the Senate Armed Services Committee

- In March 2003, the GAO reported that the F/A-22 was running $1.3 billion over budget

**Notes:**
It is not just the FAA’s problem,
... that affected both the startup of the integrated weapons system and the continuity of the system while in operation 

New phrase -- instead of the traditional “mean time between failure” we now have “Mean time between interupt event”

## We see it at NASA

**We see it at NASA**

- NASA’s Checkout and Launch Control System (CLCS) for the space shuttle
- cancelled in Sept. ‘02 after 5 years and $300 million

- Original Cost Estimate
- $206 million

- Final Cost Estimate
- (before cancellation)
- $533 million

- Original Project Completion Date
- December 2000

- Final Projected Completion Date
- (before cancellation)
- 2005

- “the monumental task of developing more than 3 million lines of computer software and adapting hardware pushed the project years behind schedule and threatened to erode any potential savings” - Michael Cabbage, Orlando Sentinel, Sept. 17, 2002

## Achilles’ heel

- What all these systems have in common are difficulties with software development and human factors
- changing requirements
- more software than expected
- underestimated overall complexity of software development

**Notes:**
It is important to note that these examples are not the exception -- we are not talking about 1 failed program to 100 successful programs.  This is the norm. (failed programs are the norm -- successful ones are the exception)

## Fundamental software problem

**Notes:**
Every application begins as an informal concept
Every software solution is a formal specification of machine behavior
This boundary must be traversed
right now, that is very late in the development cycle

## Slide 9

**Notes:**
This is the evolution of a requirements document over a 4-year period for a safety-critical avionics system that resulted in 60,000 sloc

After several false starts, document 6 was used as the baseline for controlling change implementation into software.  On average, each system requirement changed twice from the document 6 baseline.  Two other revisions (not shown in the picture) were made to the system requirements prior to the initial certification. Explanation of the documents is as follows:

Document 1:  System requirements prior to contract award.

Document 2:  Original version of the system requirements.

Document 3:  System requirements after the first revision.

Document 3A:  Change packages leading to second revision of the system requirements.

Document 4:  System requirements after the second revision.

Document 4A:  Change packages leading to third revision of the system requirements.

Document 5:  System requirements after the third revision.

Document 5A:  Change packages leading to fourth revision of the system requirements.

Document 6:  System requirements after the fourth revision. 

Document 6A:  Change packages leading to fifth revision of the system requirements.

Document 7:  System requirements after the fifth revision.

Document 8:  Original version of the system requirements.

## Development & verification challenges

**Development & verification challenges**

**Ambiguous and changing requirements**
**Languages and design approaches that allow for ambiguity**
- UML (object orientation), C, C++
**Emphasis on flexibility and creativity**
- extreme programming, anyone?
**Lack of product measures**
- overemphasis on process
- Capability Maturity Model (CMM)

## Long term vision

- Increased Formality

**Notes:**
Will allow us to have less reliance on process -- more reliance on product

## Summary

- This future *will ****not**** be possible* until we resolve the software problems we struggle with today

- NASA’s Aeronautics Blueprint envisions a future with “on demand as well as scheduled air mobility, not just to hundreds, but to thousands of communities throughout the Nation and the world; travelling where we want, when we want, faster, safer, and with far fewer delays; having access to rural areas, no matter how remote; and having direct access to urban centers, no matter how congested,” using “air vehicles that pass overhead quietly with no emission of objectionable gases into our atmosphere”