# Presentation: 797566

## Quality Leadership Forum

**Software Quality Assurance at GSFC**
- Dr. Linda H. Rosenberg
- Chief Scientist for Software Assurance
- Office of Systems Safety and Mission Assurance
- 301-286-0087
- Linda.Rosenberg@gsfc.nasa.gov

## Discussion Areas

- NASA Software Activities
- 	Software Working Group
- 	NASA Software Initiative Implementation Plan
- GSFC Software Assurance Activities
- 	Software Process Improvement – CMMI
- 	Software Safety and Reliability
- 	Software Quality Metrics
- 	IV&V

- NASA’s Software Working Group

## SWG Initial Tasks

- 1 - Define criteria for use of IV&V on a project
- 2 - Standards evaluation
- Review IEEE 12207 for potential NASA use
- Review draft of NPG 2820
- Review draft NPD for IV&V
**3 - Prepare a plan for improving software process**
**Implementation of software metrics program**
**Implementation of process improvement  model**
**Establishment of Center Software  Engineering Process Groups (SEPG)**

## SWG Task 3 - Metrics

- Set of metrics finalized summer 2000
- Objectives
- 		Provide project managers with usable information
- 		Provide agency with information on software trends
- 		Provide a measure to assess improvement
- 2 test projects per Center started Fall 2000 for 1 year
- 		(GSFC projects AURA & AQUA)
- Developing database for metrics entry and analysis

## NASA Software Initiative Implementation Plan

- Goal:  Advance software engineering practices *(development, assurance, and management)* to effectively deliver the scientific and technological objectives of NASA.
- Strategies:
- 1.	Develop and implement Agency-wide and Center plans for continuous software process and product improvement in NASA and Contractor developed software; also establish infrastructure and measurement system
- 2.	Improve safety, reliability, and quality of software products through the integration of sound software engineering principles and standards.
- 3.	Provide input for research based on identified software problem areas and infuse research results
- 4.	Improve software engineering knowledge base in NASA,  and implement strategies for attracting, retaining software engineers

- GSFC Software Assurance Activities

## GSFC Software Development Process Improvement

- Purpose - improving the processes and practices in use at GSFC using the Capability Maturity Model Integrated (CMMI) levels of maturity (ML) as a measure of progress.
- Scope - process improvement effort that will be undertaken with the goal of raising GSFC from its current state to a CMMI Defined maturity level (L3).  All projects defined by NPG 7120.5 or otherwise identified by GSFC’s Center Director will participate in this effort.

## Capability Maturity Model Integrated (CMMI)

- Level		Process Areas

- Organization innovation and deployment
- Causal analysis and resolution
- Organizational process performance
- Quantitative project management
- Requirements development
- Technical solution
- Product integration
- Verification
- Validation
- Organizational process focus
- Organizational process definition
- Organizational training
- Integrated project management
- Risk management
- Decision analysis and resolution
- Requirements management
- Project planning
- Project monitoring and control
- Supplier agreement management
- Measurement and analysis

- 5 Optimizing
- 4 Quantitatively
- Managed
- 3 Defined
- 2 Managed
- 1 Initial

**Software**
**Development**
**SW**

**Systems**
**SE**

**Software Acquisition**
**SA**

**CMMI**

- _For Pilots:_
- Emphasis - SW CMM
- As appropriate - SE CMM
- 		   SA CMM

**GSFC**

## Pilot Project Selection

## Schedule

- GSFC Implementation plan to HQ July 2001
- Management Oversight Group
- Member identification July 2001
- Initial meeting August 2001
- Training in CMMI September (1/2 day or 3 day option)
- Engineering Process Group
- Member identification July 2001
- Training in CMMI September 2001(3 day course)
- Training in Risk Management October 2001
- Pilot Implementation
- Pilot identification by October 1, 2001
- Pilot study complete October 1, 2002
- Evaluation of Pilot and roll out January 1, 2003

## What is meant by “safety”

- A system/product is _***Safe***_** **when:
- There is little to no chance for it to blow up, break, malfunction, or otherwise fail in such a way as to _potentially_ injure someone
- Something is _***Critical ***_when there is a potential  for:
**Serious injury or death**
**Serious impact to the bottom line, or **
**Bad publicity, public reputation**
**Vital information is accessible to the wrong folks**
- A system/product is _***Not***_ Safe when:
- Someone _could_ die or be seriously injured
**NASA includes possible destruction of vital equipment as well**

**Notes:**
First let’s talk about just what is “safety”, what does it mean if something is safe or unsafe.  Each organization needs to define their definition and even what levels of safety to consider.  We will talk more on this later but as a quick point of reference, NASA usually defines 4 categories:     Catastrophic, Critical, Marginal, Negligible.  Each has a lesser degree of injury to personnel and equipment. But the basics are, will it or can it hurt some one? 
While, there is no defense against stupidity and ignorance but you try to be as safe as possible.  We all know of the cases where some genius decides his lawnmower is a good hedge clipper as well.
Some failures can impact your business (incorrect ATM functions, international bank transfers, etc.), your reputation, and can lead to liability. These may also be considered under your definition of Critical software, but they are not necessarily a “safety” concern.  Of course if your ATM doesn’t deliver the money to pay-off Guido, your loan shark, that might be classified as “safety critical”.
And, no matter what, you feed the lawyers – failure, of any type, often leads to liability and lawsuits! 
There are a lot of terms we use, and we need to be careful with them, notice the “could”, “potentially”, 
, “when used...”   We need to talk a common language and that language is ... next slide

## Standards

- NASA Standards (http://standards.nasa.gov/sitemap.htm)
- NPG 8715.3	NASA Safety Manual
- NSTS-1700-7B Safety Policy and Requirements for Payloads (Shuttle and ISS)
- NASA-STD-8719.13A	NASA Software Safety Standard
- NASA-GB-A302 Software Formal Inspections Guidebook
- NSTS-22254	 Methodology for Conduct of Space Shuttle Program Hazard Analyses
- SSP-50038 Computer-Based Control System Safety Requirements**, **ISS Program
- NPD/NPG 8730 “NASA IV&V Processes “
- IEEE Standards
- IEEE 12207  Information Technology - Software Life Cycle Processes
- IEEE 830-1998 Recommended Practice for Software Requirements Specifications
- IEEE 1016-1998 Recommended Practice for Software Design Descriptions
- IEEE 1228-1994 Standard for Software Safety Plans
- Other Standards
- MIL-STD-882D System Safety Program Requirements (C version January 19, 1993)
- DO-178B Software Considerations in Airborne Systems and Equipment Certification (Federal Aviation Administration).
- ISO 9000-3	Guidelines For The Application Of  ISO 9001 To The Development, Supply, Installation And Maintenance Of Computer Software

**Notes:**
Status of the new NASA Software Safety Guidebook:
	number assigned?
	published? If not, when?
	where available?

## Software Reliability

- The probability that software will not cause the failure of a system for a specified time under specified conditions. The probability is a function of the inputs to and use of the system, as well as a function of the existence of faults in the software.  The inputs to the system determine whether existing faults, if any, are encountered. [AIAA] [IEEE 982]
- IEEE 982.1-1988 Software Reliability Management : “The process of optimizing the reliability of software through a program that emphasizes _software error prevention_, _fault detection and removal_, and the _use of measurements_ to maximize reliability in light of project constraints such as resources, schedule and performance.”

## Hardware vs. Software Reliability

- Burn in        Useful Life       Wear out

- Integration    Useful Life       Obsolete
- & test

- Hardware Failure Rate			Software Failure Rate

- Hardware reliability == Software reliability

## Definitions: Safe vs. Reliable

**Definitions: Safe vs. Reliable**

- A system is:
**safe** if it doesn’t kill anyone, or the system itself, while either performing its normal operations or, when unable to perform correctly, “fails-safe” .
**reliable **if it performs the required functions within specified parameters/environment and within predicted working timeframe consistently
***Some consider Software to be ***_***very***_*** reliable, in that it does just what its programmed to do, over and over and over again.  It doesn’t wear out or ‘break’.  However, Linda will give you the real picture on Software reliability!***

**Notes:**
Now is probably a good time to straighten out this confusion.
While a reliable system is often safer, it is not necessarily so.  If the system can be counted on to always perform a certain way - that way can be made safe.
Example: A relay contact is usually used to signal elevators to return to the bottom floor of a building and open the doors in the event of a fire alarm.  This is reliable, but is it safe?  Yes, unless the fire is on the ground level - then it is lethal!
A safe system is not always reliable.  But there is a better chance of it being safe if there does not have to be safeguards for numerous unpredictable possibilities.
Example:  A pipe works within specified parameters (i.e.pressures) if regulator allows pressure to go outside pipes range, then the pipe becomes a hazard which may rip open, explode, whatever.  The regulator may, or may not, be reliable.  If regulator doesn’t properly regulate pressure according to specs, it is unreliable.  Pipe is only unreliable if it breaks even when proper pressure is within.
For software, while anyone working with Microsoft Windows will deny software’s reliability, is it really the software or something else?

Software is very reliable. It just does what its told to do!  It won’t break!
Note: - There are, of course, arguments that it is not possible to fail safe and that software, though reliable, does not always perform its function.  These are topics that can be debated (even in class!).

## How Do We Assure / Measure Software Quality?

**How Do We Assure / Measure Software Quality?**

- Design

- Coding

- Testing

*Can you test the requirements?*
*Are the requirements complete?*
*Are you testing each requirement?*

*How much testing is necessary?*
*How hard is it to fix the components?*
*Can I reuse any components?*
*What is the quality and complexity of the code?*

*How many errors still remain?*
*What is the reliability?*
*When can I stop testing?*

## IV&V Approach

- Req        Design	 Code   	       Test (Verification & Validation)
- 			Unit	 Integration	Acceptance

- Req        Design	 Code     Testing
- 			Unit

- Test (Verification & Validation)
-     Integration	 Acceptance

- Clean Room Approach

- Traditional Software Development

- V&V

- iV&V

- Req        Design	 Code   	       Test (Verification & Validation)
- 			Unit	 Integration	Acceptance

**IV&V Implementation**

- IV&V