+----------------------+-----------------------------------------------+
| **Topic:**           | 5\. Testing                                   |
+----------------------+-----------------------------------------------+
| **Need:**            | ## 5.3. Test coverage metrics                 |
|                      |                                               |
|                      | There is a need for better test coverage      |
|                      | metrics, and guidance on their application to |
|                      | NASA software.                                |
|                      |                                               |
|                      | Test coverage metrics play a prominent role   |
|                      | in V&V and certification of safety-critical   |
|                      | software outside of NASA. The purpose of      |
|                      | coverage metrics is to indicate when          |
|                      | sufficient testing has been performed and     |
|                      | where additional testing is needed.           |
|                      | Currently, commercial avionics software is    |
|                      | developed and tested to the DO178B standard.  |
|                      | This standard mandates requirements-driven    |
|                      | testing with MCDC test coverage for the       |
|                      | highest-criticality applications.             |
|                      |                                               |
|                      | NASA needs test coverage metrics appropriate  |
|                      | to its safety- and mission-critical software. |
|                      | It is unlikely that the MCDC coverage metric  |
|                      | by itself is appropriate for NASA             |
|                      | applications, for several reasons that are    |
|                      | discussed below. Therefore the need is        |
|                      | twofold: a better test coverage metric or     |
|                      | metrics and guidance on their application.    |
+----------------------+-----------------------------------------------+
| **Relevant           | High assurance (Class A and B) software.      |
| domain(s):**         |                                               |
+----------------------+-----------------------------------------------+
| **Project(s) that    | Constellation and NASA safety- and            |
| would use the        | mission-critical software. For example, the   |
| proposed             | Ares I Abort Fault Detection, Notification    |
| tool/solution:**     | and Response (AFDNR) Software system. Results |
|                      | could also influence the commercial aviation  |
|                      | standard.                                     |
+----------------------+-----------------------------------------------+
| **Current            | There is no NASA standard or detailed         |
| tool/solution in use | guidance on test coverage metrics comparable  |
| and its              | to that provided in the commercial aviation   |
| shortcomings:**      | world by DO178B and related documents. For    |
|                      | example, the Constellation Software           |
|                      | Verification and Validation Plan lists        |
|                      | several standard coverage metrics but does    |
|                      | not provide guidance in their application.    |
|                      |                                               |
|                      | The DO178B coverage metric for Class A        |
|                      | software, MCDC, is unlikely to be appropriate |
|                      | for NASA applications. The amount of testing  |
|                      | required to attain the coverage mandated by   |
|                      | this metric is onerously expensive, and       |
|                      | furthermore the survey by Kelly Hayhurst et   |
|                      | al. showed widely varying levels of           |
|                      | satisfaction with its effectiveness in        |
|                      | revealing software bugs.                      |
|                      |                                               |
|                      | There is a growing understanding of the       |
|                      | inadequacies of the MCDC coverage metric      |
|                      | itself. Like any metric, it is vulnerable to  |
|                      | deliberate approaches to thwart its intent    |
|                      | (notably by designing the program structure   |
|                      | so as to minimize the testing required to     |
|                      | attain the level of coverage, but at          |
|                      | increased risk of masking latent bugs). The   |
|                      | FAA provides guidance to mitigate this        |
|                      | problem. More worrisome are recently reported |
|                      | results showing that adoption of              |
|                      | well-intentioned program structures can also  |
|                      | lead to this same phenomenon. Furthermore,    |
|                      | MCDC does not address coverage issues         |
|                      | specific to reactive systems; and it is       |
|                      | unknown how it should be extended to other    |
|                      | non-standard forms of software (e.g., to      |
|                      | model-based reasoning systems, in which there |
|                      | is growing interest in Constellation and      |
|                      | NASA).                                        |
+----------------------+-----------------------------------------------+
| **Constraints that   | To the extent that the work plan included an  |
| might influence the  | effort to experiment with proposed metrics on |
| research work        | past NASA software developments, and          |
| plan:**              | especially if there was a desire to           |
|                      | retroactively estimate how adherence to the   |
|                      | proposed metric would compare with actual     |
|                      | past practices, then the following constraint |
|                      | would apply: Difficulty in obtaining          |
|                      | historical data including test plans and      |
|                      | results, in order to evaluate the             |
|                      | effectiveness of proposed metrics.            |
|                      |                                               |
|                      | The diversity of development processes,       |
|                      | development platforms, and software           |
|                      | architectures will likely preclude the        |
|                      | universal adequacy of a single coverage       |
|                      | metric.                                       |
|                      |                                               |
|                      | Unconstrained development methods can defeat  |
|                      | any coverage metric. Thus, we expect that     |
|                      | coverage metrics will impose constraints on   |
|                      | development methods.                          |
+----------------------+-----------------------------------------------+
| **Timeline that the  | For example, Ares AFDNR; related ground-based |
| proposed             | diagnostics systems (ISHM).                   |
| tool/solution must   |                                               |
| support:**           |                                               |
+----------------------+-----------------------------------------------+
| **Language           | Generally no specific language requirement.   |
| requirement:**       | The solutions (coverage metrics and guidance  |
|                      | on their application) should be               |
|                      | language-neutral, although specific tools for |
|                      | implementing the coverage metrics will be     |
|                      | language-specific. There may emerge a need    |
|                      | for different metrics depending on the        |
|                      | *class* of language -- e.g., graphical        |
|                      | languages and their accompanying code         |
|                      | generation may demand a different coverage    |
|                      | metric to that required by traditional        |
|                      | programming languages; model-based reasoning  |
|                      | systems may also represent a distinct class   |
|                      | with distinct metric needs.                   |
|                      |                                               |
|                      | Ideally, coverage metrics should also be able |
|                      | to take into account the results of other     |
|                      | verification techniques, such as static       |
|                      | analysis, so as to avoid duplication of       |
|                      | effort where coverage has already been        |
|                      | demonstrated.                                 |
+----------------------+-----------------------------------------------+
| **Size or scale      | The coverage metrics must be applicable to    |
| requirements:**      | real NASA or FAA applications. The concern is |
|                      | that in the area of testing, coverage metrics |
|                      | that demonstrate excellent results on "toy"   |
|                      | or small-scale applications may not scale to  |
|                      | real applications. Thus there is a need to    |
|                      | demonstrate applicability to applications of  |
|                      | at least 10KSLOC in size, preferably more.    |
+----------------------+-----------------------------------------------+
| **Required           | Clear, English-language definitions of new    |
| deliverables:**      | test coverage metrics;\                       |
|                      | Discussion of their range of applicability;\  |
|                      | Justification of their use in place of, or    |
|                      | together with, existing coverage metrics;\    |
|                      | Specific guidance on how to apply them\--for  |
|                      | example, through a tutorial;\                 |
|                      | Discussion of development techniques that     |
|                      | enable application of these metrics; and      |
|                      | factors that can defeat them;\                |
|                      | Indication of tools, technologies and         |
|                      | procedures that can implement them;\          |
|                      | What to do in the absence of a comprehensive  |
|                      | tool solution.\                               |
|                      | Ultimately we will need tools that implement  |
|                      | the coverage metrics for C, C++ and any other |
|                      | languages expected to be used for developing  |
|                      | safety- and mission-critical software.        |
|                      | However, we recognize that these products may |
|                      | be more appropriate for commercial            |
|                      | development following delivery of the needed  |
|                      | research results.                             |
+----------------------+-----------------------------------------------+
| **Other useful       | Specific guidance on how to apply them\--for  |
| information:**       | example, through a tutorial.                  |
+----------------------+-----------------------------------------------+