# Presentation: 066132

## Assessing Requirements Quality through Requirements Coverage

**Ajitha Rajan****	****	**University of Minnesota
**Mats Heimdahl****	**University of Minnesota
**Kurt Woodham****	**L3 Communications

## Model-Based Development

- Specification
- Model

**Notes:**
In model-based development, the development effort is centered around a formal description of the proposed software system -- the “model". This model
is derived from some high-level requirements describing the expected behavior of the software. For validation and verification purposes, this model can then be subjected to various types of analysis, for example, completeness and consistency analysis [6], model checking [3], theorem proving [1], and test-case generation [4,7]. This development paradigm is making rapid inroads in certain industries,
e.g., automotive, avionics, space applications, and medical technology. This shift towards model-based development naturally leads to changes in the verification and validation (V&V) process. The model validation problem -- determining that
the model accurately captures the customers' high-level requirements-- has received little attention and the sufficiency of the validation activities has been
largely determined through ad-hoc methods. Since the model serves as the central artifact, its correctness with respect to the users' needs is absolutely crucial.

## Model Validation problem

- Are the requirements sufficiently defined for the system?
- How well does the model implement the behaviors specified by the requirements?

**We propose a testing approach that explores the relationship between** ***requirements-based *****structural coverage** **and** ***model-based *****structural coverage**

**Notes:**
In our investigation, we attempt to answer the following two questions with respect to validation ...
The second question can be addressed using formal verification. Nevertheless, the size and complexity of many industrial systems make formal verification infeasible even if we have a formal model and formalized requirements. Thus, presently, there is no objective way of answering these two questions. To this end, we propose an approach based on testing that -- when given a set of formal requirements -- explores the relationship between requirements-based structural test-adequacy coverage and model-based structural test-adequacy coverage

The proposed technique uses requirements coverage metrics defined in [9] on formal high-level software requirements and existing model coverage metrics such as the Modified Condition and Decision Coverage (MC/DC) used when testing highly critical software in the avionics industry [8]. Our work is related to Chockler et al. [2], but we base our work on traditional testing techniques as opposed to verification techniques.

## Are the Requirements Sufficient?

- Run tests on the model and measure coverage achieved

**Informal Requirements**

**Model / Design Specification (MUT)**

**Requirements-Based Tests**

- Generate to provide requirements coverage

- Poor coverage of model implies one or more of the following
- Missing or implicit requirements
- Behavior in the model not derived from requirements
- Requirements-based tests are inadequate

- Measure

## Model Implements Requirements?

- Run tests on the requirements and measure coverage achieved

**Model/Design**
**Specification**

**Requirements**

**Model-Based Tests**

- Generate to provide model coverage

- Measure

- Poor coverage of requirements implies one or both of the following
- Model does not adequately implement behaviors specified in the requirements
- Model is correct and requirements are poorly written

## Experimental Setup

**Requirements Coverage Metric** – Unique First Cause (UFC) coverage defined over formal LTL requirements
**Model Coverage Metric** – Modified Condition/Decision Coverage (MC/DC)
- Four industrial case examples from the avionics domain
- Two kinds of assessments
- Generate test suites to provide requirements UFC coverage and measure MC/DC over the model
- Generate test suites to provide MC/DC over the model and measure UFC achieved over formal requirements.

**Notes:**
For each of these systems, we perform two kinds of assessment|(1) generate test suites to provide UFC coverage over the re-
quirements and measure MC/DC achieved over the model, and (2) generate test suites to provide MC/DC over the model and measure UFC coverage achieved over the formal requirements. We analyze the relationship between requirements coverage and model coverage to make an assessment of the quality of the sets of requirements as well as the models.

## Experiment Results

- MC/DC Achieved by Requirements-Based Tests

**Notes:**
Reason for UFC test suites provide good MC/DC of the model but 10 – 20% less than achievable 
This is somewhat expected since the requirements (representing
DO-178B high-level requirements) are typically less detailed than the model (representing DO-178B low-level requirements). Another reason may be that the UFC metric used for requirements coverage is not sufficiently rigorous and we thus have an inadequate set of requirements-based tests.

For  DWM2 case example in which the UFC suites achieved poor MC/DC over the model, the requirements were good, however, their structure was so that the complexity of conditions in the requirements were hidden. For such requirements, the UFC metric that we use is not effective since the structure of the formalized requirements effectively “cheated" the UFC metric. One solution to this would be to restructure the requirements to reveal condition complexity. Another possible solution is to use a requirements coverage metric that is not as sensitive to the structure of the requirements. We hope to investigate this issue further in our future work.

## Experiment Results

- MC/DC Achieved by Requirements-Based Tests
- Two of the four case examples
- UFC test suites provide **good** MC/DC of the model but 10 – 20% less than achievable
- Remaining two case examples
- UFC test suites achieved **poor** MC/DC on the model
- Many missing requirements on one example
- On the other example, structure of the requirements “cheated” the UFC metric

**Notes:**
Reason for UFC test suites provide good MC/DC of the model but 10 – 20% less than achievable 
This is somewhat expected since the requirements (representing
DO-178B high-level requirements) are typically less detailed than the model (representing DO-178B low-level requirements). Another reason may be that the UFC metric used for requirements coverage is not sufficiently rigorous and we thus have an inadequate set of requirements-based tests.

For  DWM2 case example in which the UFC suites achieved poor MC/DC over the model, the requirements were good, however, their structure was so that the complexity of conditions in the requirements were hidden. For such requirements, the UFC metric that we use is not effective since the structure of the formalized requirements effectively “cheated" the UFC metric. One solution to this would be to restructure the requirements to reveal condition complexity. Another possible solution is to use a requirements coverage metric that is not as sensitive to the structure of the requirements. We hope to investigate this issue further in our future work.

## Experiment Results

- UFC metric “cheated” by the structure of requirements

**LTLSPEC G( var_a > (**
**     ****case**
**	****foo : 0 ;**
**	****bar: 1 ;**
**     ****esac****	****+**
**     ****case**
**	****baz : 2 ;**
**	****bpr : 3 ;**
**     ****esac**
**));**

**LTLSPEC G( var_a > (**
**     ****case**
**	****foo & baz : 0 + 2 ;**
**	****foo & bpr : 0 + 3  ;**
**	****bar & baz : 1 + 2 ;**
**	****bar & bpr : 1 + 3 ;**
**     ****esac****	**
**));**

**Notes:**
Informally, the sample requirement states that var a is always greater than the sum of the outcomes of the two case expressions.

The above requirement may be restructured (to express the same behavior) so that the sum of two case expressions is expressed as a single case expression. Achieving UFC coverage over this restructured requirement will involve more obligations than before since the boolean conditions in the case expression are more complex.
Thus, the structure of the requirements has a signi ̄cant impact on the number and rigor of UFC obligations and hence the size of the test suite providing
UFC coverage.

as shown:

## Experiment Results

- Requirements UFC Coverage Achieved by Model-Based Tests

**Notes:**
We found that on all but one of the industrial systems, test suites providing MC/DC over the model achieved close to achievable requirements UFC coverage. This implies that the model exercises almost all the behaviors specified by the requirements for these systems.

Nevertheless, on one model the MC/DC test suites did poorly, only achieving 30% of the achievable requirements coverage.
This may either be because the model does not implement all the behaviors or the MC/DC metric is not rigorous enough.
At this time we have not been able to determine the cause more closely, but we hope to do so in our future work.

## Experimental Results

- Requirements UFC Coverage Achieved by Model-Based Tests
- On all except one case example MC/DC test suites achieved close to maximum possible requirements UFC coverage
- On one example, the MC/DC test suites did poorly (30% of maximum possible UFC)
- Model does not implement all the behaviors in the requirements
- MC/DC metric is not rigorous enough

**Notes:**
This implies that the model exercises almost all the behaviors specified by the requirements for these systems.
At this time we have not been able to determine the cause more closely, but we hope to do so in our future work.

## Conclusions

- Analyzing requirements coverage Vs model coverage provides a promising means of assessing requirements quality.
- Effectiveness is dependent on the rigor and robustness of the coverage metrics used
- UFC sensitive to the structure of the requirements

**Notes:**
For instance, in this experiment we found that the UFC metric was surprisingly sensitive to the structure of the requirements, and one has to ensure that the requirements structure does not hide the complexity of conditions for the metric to be effective.