# Presentation: 168195

## Injecting Faults for Error Evaluation

- Kalynnda Berens
*Kalynnda.Berens@grc.nasa.gov*
- Richard Plastow
*Richard.Plastow@grc.nasa.gov*

## Introduction

- Applications often consists of software components plus custom development, merged into a coherent package.
- COTS, GOTS, open source, etc.
- Source code is usually not available for review  of quality and reliability.
- Visibility into the component is only what’s available via a public interface
- What is the quality of that component?
- What faults lay inside the component?
- Applications interface with hardware and other software and can be influenced by failures in those systems.

**Notes:**
Developers today use many software components programmed by other software developers outside their organizations.
Examples of components are: 
3 GLs like C++, VB, and Java
4 GLs like SQL, Perl, and Python
RPCs like DCOM and CORBA
GUI design tools in IDEs tolls that create window frames, buttons , icons, and widgets.
Many commercial vendors  will not disclose their source code for review and many end users license prevent end users from disclosing any test result or errors and omissions they discover to other end users outside their organizations.
These software component are like black boxes that only expose public interfaces to access the remaining code inside.
Software designers are forced to accept the vendors idea of quality or lack of quality particularly if the vendor has an monopoly on the technology.
No software assurance department to date has been able to create a programming environment that allow developers to program fault free code, however several Software Assurance department have been able to create process that minimize the injection of faults into source code 
PSSST Software components have source code inside!

## Fault Injection on Interfaces

- Interfaces (hardware, software, human) are a major source of errors and induced faults
- Software and system testing looks at *anticipated* off-nominal situations, but often misses unusual situations or combinations of faults
- Mishap investigation has shown that multiple faults or unexpected anomalies are key players in accidents and mission failures

## Example System

- Application

- COTS
- Library

- COTS Operating System

- Other Applications on same system

- External Systems

- System Hardware

- Input Sensors

-  Control Outputs

## Fault Injection Flow Diagram

- No

## Interface Identification

- Artifacts and Documentation
- Software and System Requirements and Design specifications
- Interface Specifications
- User and Training Manuals
- Other project documentation
- Source code

## Error Research

- Sources of Error/Fault Information
- Vendor documentation
- Application or Hardware Interfaces
- White papers
- Defect list or open issue/problem reports
- Public bug list
-  Internet Sources
- Software logs
- Error databases
- Personnel Experience

## Evaluation and Scoping

- Determine level of effort, funding, time constraints
- If complete effort not possible
- Perform importance analysis of interfaces, software units
- Safety
- Complexity
- Use by other system elements
- Expected number or types of faults
- Prioritize and select by importance

## Testing

- Test case generation based on identified errors plus permutations on possible input values
- Consider multiple faults
- Consider faults while system is off-nominal from a previous fault
- Consider effects of system load/stress
- Consider state-specific effects
- Instrument software to observe effects of injected faults
- External or observable effects
- State changes (or lack of)
- Effects on safety-critical functions

## First Project: Tempest

- Written in Java 1.1
- Configurable
- Cross platform operability
- Implements HTTP GET and HEAD Request and Server Side Includes
- Has some Basic Security Features
- Debug Mode monitoring
- Commercially available

**Notes:**
Written in Java for portability.  HTTP Version 1.1 compliant.
In HTTP version 1.1 GET and HEAD request are required methods, the other methods are optional so they were not implemented.
Server side includes support dynamic Web pages.
There are several options at startup and initiation that are configurable.
Has a user/password security scheme. 
Under multitasking based VxWorks RTOS 3-4 msecs transaction on a local intranet.

## Requirement Database

- Documents found contained 80 requirements that vendor says he meets (vendor’s claims)
- Requirement table is the parent table for 5 other sub-tables
- Performance
- Specs
- HFE
- Security
- Misc

## Standards Database

- Parsed commercial standards into pseudo  functions and test scenarios
- Test scenarios included expected faults as well as on-the-edges and way-outside-the-box tests
- Each standard has its own table and own set of tests

## Tempest Results

- Inappropriate system operation with modified configuration file
- Non-compliance with HTTP standard
- System crash with invalid port numbers
- Port 49151.45 -> opened port 80
- File access in server machine outside of authorized directories
- System did not operate as per user documentation

## Second Project:Fluids and Combustion Facility

- Permanent, multi-user facility for ISS microgravity experiments
- Two racks (fluids/FIR, combustion/CIR)
- Operates for 10 years, so robustness important
- CANbus processors selected for fault injection
- Health and Status Monitoring
- Cannot be upgraded in flight
- Mature requirements, design, and interface definition
- Source code available

## CANbus Processors

**Air Thermal Control Unit (ATCU)**
**Color Camera Package (CCP)**
**Common IPSU Diagnostic Board**
**FOMA Control Unit**
**FSAP Diagnostic Board**
**Nd:YAG Laser Package **
**Water Thermal Control System (WTCS)**
**White Light Package**

## FIR System Diagram

- Laser Diode CAN Processor

- White Light CAN Processor

- DCM
- CAN Processor

- Nd:Yag CAN Processor

- PI Package

- ATCU CAN Processor

- WTCS CAN Processor

## CANbus Processor State Diagram

- Power On

- Power Off

- Power Down Cmd

- Error

- Success

- Operational Cmd

- Error

- Error

- Operational Cmd

- Power Down Cmd

## Next Steps

- Complete Interface Identification and prioritization
- Obtain hardware, source code for testing environment
- Error/Fault search on selected interfaces/components
- Test case generation, source code instrumentation, and test execution