# Presentation: 745528

## Maintaining System & SWVerification Test Consistency

- OSMA Software Assurance Research Program FY03: Initiative 583 	 FY04: CSIP2004-77

***TRIAKIS Corporation***

- July 2004

- By Ted Bennett & Paul Wennberg

- 2004 Software Assurance Symposium

- Testing Flight SW Response to Enhanced Failure Modes

## Problem

**Disconnect exists between System and software development loops**

- Most embedded SW faults found at integ. test traceable to Rqmts. & interface misunderstanding

**Notes:**
The status quo process for embedded systems development begins with a system design that is developed using any number of tools created in support of the process.  For example, the system designer may make use of modeling, simulation, prototyping, specification, and other tools to satisfy the need to validate control algorithms, component interactions, etc. and document the requirements for the implementation teams to follow.  The system architects verify and validate their design through analysis, possibly tests, and possibly by similarity with reused components.

The implementation teams take the requirements, interpret them the best they can - be they written in natural language, specification design language, or executable specifications, and derive a hardware and software design from them.  The software developers write their own tests to verify conformance to the requirements as they have interpreted them.  They may use some form of simulation, hardware development boards, inspection, analysis, or similarity comparison to facilitate the verification of their code.  When a major part of the system functionality has been coded, they create a build and begin the process of loading their software onto hardware connected to test equipment and perhaps other system elements to begin the process of system integration testing.  The faults discovered here are filed as problem reports and passed back to the development team to resolve.  The integration hardware test setup is generally very costly and consequently, it is rare to have more than one or two setups available for project use.  Not only does the high demand for integration test time cause create a resource bottleneck that often delays the project, but finding faults during this phase of development is by far the most costly. 

According to Lutz’s JPL spacecraft software study, 98% of the bugs discovered here “arise most commonly from (1) Discrepancies between the documented requirements specifications and the requirements needed for correct functioning of system and (2) Misunderstanding of the software’s interface with the rest of the system.” [1]   We feel this is largely due to what we see as a disconnect between the system development/verification and software development/verification loops.  If the software could be tested using the same tests developed to verify the system design, it might prove to be an effective means of reducing the number of software faults discovered during integration test.  

## Approach

**Test Results**

**Notes:**
Triakis has built its business over the past 8 years creating custom system simulators and using them for testing embedded avionics software.  Our approach has been to simulate the micro-controller hardware and the system environment in which it operates to create a Virtual System Integration Lab (VSIL) in which to execute and test the software.  We write verification tests according to the system requirements specifications, to exercise the object software in a system-driven manner.

A few years ago it occurred to us that our simulators could be used to develop and V&V the system design with Executable Specification (ES) type simulated parts. This approach would enable the specification and test of a complete system design in a highly coherent fashion with the finished product conveyed to the software developers as a complete package of executable specifications running within a VSIL with the system verification tests included.  If the ES’ in the VSIL were bounded exactly like their real-world counterparts, then the simulated controller hardware should be directly interchangeable with its ES part in the VSIL.  When the requirements of the controller ES have been implemented correctly then the software executing within the simulated controller hardware (referred to as the detailed executable or DE) should pass all the same system verification tests that the ES passed. In so doing we could greatly reduce the opportunity for the introduction of SW errors due to misspecification and misinterpretation of system requirements by the software development teams.  

Since we all know that you can never eliminate all bugs, we expect that this process will yield far fewer (and perhaps less malicious) bugs that pass through to the integration phase.

## Accomplishments

- Unmodified ES verification tests give same results when testing object SW running on simulated PowerPC-based SRMS

- 131 system-level verification tests written
- ES- and DE-based system simulations pass 129 tests
- 2 failed tests due to simulator bug

**Notes:**
These results demonstrate that we have achieved our primary objective of determining the viability of maintaining test consistency between system-level, and software-level verification.  We have shown that it is possible to test software with the same verification tests used to verify the system design.

## Importance/Benefits

- Early discovery of SW faults prior to HW integration testing

- Promotes SW assurance through close coupling of system and SW V&V

- Lower development $$ by finding more faults early, w/o lab or integration HW

- Unmodified system-level tests can be used to verify embedded object SW

**Notes:**
One of the potential benefits of using a VSIL simulator for software V&V is that it reduces the opportunity for misinterpretation and misimplementation of the system requirements.  Further, the software developers will gain a broader understanding of how the system functions and how the software is intended to interact with the various other system elements.  This close coupling of system and software V&V should serve to promote greater SW assurance.  Finding SW faults prior to system integration testing with the real hardware should save both time and money. 

## Relevance to NASA

- Potential to reduce project costs
- Improve project-level assurance
- Improve IV&V through SW *testing*
- Verify executable SW
- Manually generated software
- Auto-generated software
- Reused/modified software
- Real-time operating systems

## Empirical Assurance of Embedded SWUsing Realistic Simulated Failure Modes 

**Empirical Assurance of Embedded SW****Using Realistic Simulated Failure Modes**

**Problem: FMEA Limitations**
**Expensive & time-consuming**
**List of possible failure modes extensive **
**Focuses on prioritized subset of failure modes**
**Approach: Test SW w/sim’d Failures**
**Create pure virtual simulation of Mini-AERCam HW & flight environment running on PC**** **
**Induce realistic component/subsystem failures**
**Observe flight SW response to induced failures **

**Can we improve coverage by testing SW resp. to sim’d failures?**
**Compare results with project-sponsored FMEA:**

**#Issues uncovered?**

**#Failure modes evaluated?**

**Effort involved?**

## Mini-AERCam Project

**Importance/Benefits**
- Virtual environment testing requires no HW
- HW integration lab expensive, scarce
- Simulate more failure modes than analysis permits
- Beyond practical capability of HW integration labs
- More failure modes tested  more bugs found
- SMA use of simulator for orthogonal testing
- System & SW implementation tested – not just design

## Mini-AERCam Project

**Relevance to NASA**
- Addresses failure mode growth due to increasing system complexity & autonomy
- Mini-AERCam TBU for spacecraft inspections
- Space shuttle, ISS, CEV, et al
- Applicable to all NASA embedded systems
- Multiple uses for simulator
- Systems & SW Development
- Project-level Safety & Mission Assurance
- Independent Verification & Validation
- Post-launch support

## Mini-AERCam Project

**Accomplishments**
- Project begun in earnest April ’04
- Well into simulator development

**Next Steps**
- Complete Mini-AERCam simulator
- Write failure mode tests
- Test SW response to enhanced failures
- Compare empirical results with FMEA results

**Notes:**
FY04: NASA-JSC Mini-AERCam Project Abstract: Reliance on analysis alone to evaluate the effect of failure modes un-testable in an integration lab setup will not fully ameliorate the added risks due to ever-increasing system complexities and the use of intelligent behavior in software designs.  Few would argue that evaluating system behavior through dynamic testing in its native target environment is the best method of verification, however, this is almost always infeasible for NASA space hardware.  We will simulate the NASA Mini-AERCam nanosatellite, running its unmodified software, in its orbital environment and demonstrate how a high-fidelity virtual environment simulation can be used for thorough empirical assessment of system and software behavior in response to a wide range of enhanced system and component failure conditions.  
FY04: NASA-IVV STEREO Project Abstract: Sole reliance on static analysis of processes, specifications and software for IV&V activities neglects a substantial verification domain.  The task of V&V can hardly be considered complete until the behavior of the controlling software has been fully tested while dynamically executing in its target operational environment.  Historically, this domain has only been accessible to project development teams because of the high costs involved in putting together a full hardware integration & test laboratory.  In this case study, we will simulate the STEREO spacecraft hardware and orbital environment with sufficient fidelity to run the unmodified executable flight software.  Using this pure virtual environment we will develop a full suite of tests to verify that the software has correctly implemented the system and software design requirements.  Both nominal and off-nominal software path coverage will be monitored and all unexercised code will be held accountable.  Further, we will “stress test” the system to evaluate its robustness by varying instruction execution speed, propagation delays, data skew and other parameters readily adjustable in a virtual environment.  We expect to reveal many software, and possibly system “issues” discoverable in neither the traditional developer hardware integration & test environment nor the IV&V analysis domain.  The simulation results will be compared against the readily available IV&V analysis results.  The research team comprises the team lead for the STEREO IV&V and the Triakis Corporation – developers of the unique simulation technology that opens this verification domain to IV&V.  The ability to uncover system and software faults through the use of high-fidelity simulation will significantly enhance the value of IV&V and contribute to both safety and mission assurance.  
Candidate Topics: While there is certainly a cost and time savings associated with the early detection of software faults that our VSIL simulator approach can facilitate, there is a cost involved in creating the VSIL.  High reuse of simulator parts reduces costs while the creation of new parts takes time & effort.  The ultimate way to find out is to conduct a comparative assessment with status quo methods on a candidate project.  Alternatively, if there were 2 projects similar in complexity they might be used effectively for a comparative analysis without unnecessary duplication of effort.  
We could also conduct an analysis of the economics of both approaches through a study of a well-documented project or two.  We could create a VSIL for each, determine how many faults found during integration would have been detected in the VSIL, and estimate the costs involved to give us a ball-park number.  
An analysis made by one of our customers, of the impact that using a Triakis VSIL had on a small project concluded that the overall project duration was reduced by an estimated 24%.  Citing a reduced need for emulators, special test equipment, and other lab hardware along with the effect of the shortened project duration, this customer claimed a 15% reduction in overall project cost.