# Presentation: 375938

## VAL-COTSValidation of Real Time COTS Products

*Ricardo Barbosa, Henrique Madeira, Nuno Silva*
- Dependability & Embedded

## Overview - VALCOTS

- Introduction
- Objectives and Goals
- Developed Techniques
- Conclusions
- Future Work

## Introduction (1)

- VAL-COTS
- VALCOTS is an acronym for “Validation of Real Time COTS Products”
- COTS current trend
- Commercial Off-the-shelf products are being used broadly in space applications
- How to have full confidence in COTS components? (custom made components confidence can be controlled and assessed in-house)
- ... sometimes one may doubt of certain dependability characteristics of COTS

## Introduction (2)

- Why are COTS different from custom made software?
- Different requirements needs...
- COTS are designed to be general purpose (requirements such as timing and safety might not be given high priority)
- Wrong assumptions...
- Product engineers tend to assume that it is impossible to predict all possible component violations for each function
- Overlooked tests...
- Input tests are sometimes overlooked (insufficient) due to performance or schedule constraints

## Objectives and Goals (1)

- Goals
- The project main goal is to develop techniques and toolsets for the assessment of real time COTS products based on automated fault injection techniques
- Fault Injection...?
- Fault injection techniques have shown to be an excelent approach in the validation of COTS products
- Mentioned in Galileo Software Standard as an advisable technique to be used for the validation of space systems
- Benchmarks used fault injection techniques for validating real time constraints [DBench]
- Can simulate effects of radiation on hardware (space environment)

**Notes:**
RTEMS was a good project that shown the advantages of using fault injection
DBench Project showed that fault injection can also be used to validate real time constraints

## Objectives and Goals (2)

- Introduce the new concepts on fault injection and testing in NASA’s IV&V philosophy as an agency’s value added research and expertise
- Improve NASA’s IV&V processes, tools and techniques on dynamic testing

**Notes:**
Com estas actividades, a NASA pode ter acesso a métricas de software (particularmente de COTS) e com elas ponderar o seu uso.
Por exemplo, fornece dados acerca de uma missão em volta recebe métricas.
Passagem de conhecimento acerca de testes dinamicos na CSW/ESA para a NASA

## Objectives and Goals (3)

- Exchange of software metrics (on COTS and others)
- Collaborative knowledge exchange on IV&V, dependability evaluations, dynamic testing and automated testing  between CSW(European Space Industry) and NASA

## Developed Techniques (1)First Technique

- Three different techniques were developed during the project
- First Technique is focused on COTS robustness and stress capabilities
- Based on application requirements, API fault injection is performed to assess the COTS component in the presence of in and out of bounds values
- This process is accelerated and tunned up by performing software automated fault injection in upstream components of the target COTS

## Developed Techniques (2)First Technique

**mutex_change_priority (**
**	****TCB *p_tcb,**
**	****UINT oldpri,**
**	****UINT *p_newpri)**** **

| Type Name | Test Values |
| --- | --- |
| INT | -2147483648, -1, 0, 1, 2147483647 |
| UINT | 0, 1, 4294967295 |
| BOOL | true, false |
| FN | -2147483648, -1, 0, 1, 2147483647 |

**     ****__xception__ = 0;**
**     ****m_p_tcb = mutex_change_priority (**
**                                         ****p_tcb,**
**                                         ****oldpri,**
**                                         ****__xception__);**

**     ****__xception__ = 1;**
**     ****m_p_tcb = mutex_change_priority (**
**                                         ****p_tcb,**
**                                         ****oldpri,**
**                                         ****__xception__);**

**     ****__xception__ = 4294967295;**
**     ****m_p_tcb = mutex_change_priority (**
**                                      ****p_tcb,**
**                                      ****oldpri,**
**                                      ****__xception__);**

- Developed Techniques (2)First Technique

- Automatic Test Cases Generation

## Developed Techniques (3)Second Technique

- The second technique is double phased:
- (1) COTS component dependability is assessed individually at an earlier development stage
- (2) Software fault injection is performed inside the actual component to assess the component (and system) within the full integrated architecture, conducted at a latter developement stage
- Results may lead to architectural changes or component wrapping, or even COTS component replacement

## Developed Techniques (4)Second Technique

- Earlier Stages of SDP

- Latter Stages of SDP

## Developed Techniques (5)Third Technique

- A third technique was developed based on the second one
- To complement the technique, risk assessment calculations are also introduced to estimate failure probability of the component within the system
- Risk assessment calculations are based on software metrics from component and system complexity (e.g. cyclomatic complexity)

## Conclusions

- The case studies conducted have shown that the applicaiton of any of the techniques lead to architectural and component improvements
- Thus, robustness and dependability improvements
- Results identify that COTS components have intrinsic faults that are hard to find with traditional testing
- Usefulness of fault injection
- Xception based automated testing as proven to be very usefull particularlly considering fault injection definition and injection automation
- Some difficulties where encountered while evaluating injection run results as it is important to improve results analysis automation

## Future Work

- VAL-COTS II
- Improvement of the developed techniques
- Further testing to validate the techniques (hopefully on an on-going project)
- Adjust techniques for use in certification processes (follow specific standards like ISO 25051 or DO-178B)
- Improve the techniques based on the newly developed software fault injection G-SWFIT
- G-SWFIT is a technique developed at Universidade de Coimbra for emulating real faults in fault injection
- Using this technique, no source code is needed for performing fault injection, only binary files

## The end!

- Thank You!!
- Any questions?
- 			rbarbosa@criticalsoftware.com
- 			hmadeira@criticalsoftware.com
- 			nsilva@criticalsoftware.com