+------+---------------------------------------------------------------+
| ![   | FSW UNIT TEST STANDARD                                        |
| ](me |                                                               |
| dia/ | Flight Software Branch -- Code 582                            |
| imag |                                                               |
| e1.j | Version 2.00 -- 12/08/06                                      |
| peg) |                                                               |
| {wid | 582-2000-002                                                  |
| th=" |                                                               |
| 6.32 |                                                               |
| 4305 |                                                               |
| 5555 |                                                               |
| 5555 |                                                               |
| 5in" |                                                               |
| h    |                                                               |
| eigh |                                                               |
| t="8 |                                                               |
| .120 |                                                               |
| 8333 |                                                               |
| 3333 |                                                               |
| 3334 |                                                               |
| in"} |                                                               |
+------+---------------------------------------------------------------+

Forward and Update History

This standard defines the NASA Goddard Space Flight Center Flight
Software Branch software library requirements for unit tests of flight
software components, and gives guidance on unit test implementation.

  -------------------------------------------------------------------------
  Version   Date      Description                          Affected Pages
  --------- --------- ------------------------------------ ----------------
  1.0a      7 Sept    initial                              all
            2000                                           

  1.01      3 Sept    Convert source to Microsoft Word,    all
            2003      modify formatting                    

  1.02      3 Sept    Modify requirement for testing on    all
            2003      target machine. Formatting & layout  
                      changes; new cover page              

  1.03      25 Sept   Modify requirement 6 to describe     
            2003      waiver process                       

  2.00      08 Dec.   Implement DCR #241 (plus agreements  All
            2006      reached at the 11/29/06 Standards    
                      CCB meeting.                         
  -------------------------------------------------------------------------

Contents

[1 Introduction 4](#__RefHeading___Toc50518831)

[2 Requirements 5](#__RefHeading___Toc50518832)

[3 Guidelines 9](#__RefHeading___Toc50518833)

[]{#__RefHeading___Toc50518831 .anchor}1 Introduction

Finding problems in development is considered more cost effective than
finding problems during the later FSW phases; in addition testing is an
essential part of maintaining a software library. Users of code from the
library must be able to trust the code. Having easy to run, complete
unit tests contributes to this goal.

At the same time, having well tested library units allows projects to do
better testing on their specific target hardware. Since they are not
designing and writing code and unit tests from scratch, they have more
time to run the unit tests on their specific hardware, in order to
ferret out the last remaining compiler or target specific bugs.

A unit is defined as single interface file, together with the body files
that implement the unit. In C, the interface file is a header file. In
Ada, it is a package specification.

A unit test is defined to be a test that executes all of the code in a
unit, with the unit not embedded in a larger system. The unit test
proves that the unit executes correctly. Unit tests are always
white-box; they take advantage of knowledge of how the unit is
constructed. Unit tests may consist of more than one executable, if that
makes them more understandable. The unit source code, the unit test
input, the unit test source code, and unit test results must together
show the unit source code is properly implemented and free of
implementation errors.

A unit test is distinct from a functional test, which proves that a unit
meets its requirements as part of a larger system. Functional tests are
always "black box"; they use no knowledge of how the unit is
constructed - they can only use knowledge of the unit's interface and
requirements.

It may be necessary to break up large units into smaller ones in order
to satisfy the requirement for statement coverage and boundary value
testing. The large units then become components.

It is often the case that algorithm bugs are found during unit testing;
the unit developer may discover an ambiguity in the algorithm
definition, or it may become clear that the "known good" output is
actually wrong. While it is always useful to find such bugs, this is not
the primary goal of unit testing. Unit tests should show that the unit
implements the algorithm, not that the algorithm is "correct".

Another type of testing is "performance testing", where a full system
must meet some performance criteria, such as pointing a telescope with
some accuracy. Unit testing does not test performance, except that it
may be useful to measure execution times for various operations.

One issue not covered here is testing the effect of single event upsets,
which can change arbitrary portions of the code and data.

The following pages have requirements in bolded text and supporting
rationale in non-bolded text.

[]{#__RefHeading___Toc50518832 .anchor}2 Requirements

1\. Deleted

2\. Unit test shall execute every statement in the unit, including all
branches of conditional statements.

Statement Coverage means each line of source code has been executed and
tested.

We do not require covering all combinations of paths, since that is
often too cumbersome.

For example, consider the following code:

if (a=1)

{

if (b=1)

\... /\* line 1 \*/

else

\... /\* line 2 \*/

if (c=1)

\... /\* line 3 \*/

else

\... /\* line 4 \*/

}

\... /\* line 5 \*/

This code only requires 2 tests for \"statement coverage\". Lines 1, 3,
5 are executed by the case a=1, b=1, c=1, and lines 2, 4 are executed by
the case a=1, b=2, c=2.

Covering all combinations of paths would mean requiring the additional
cases a=1, b=1, c=2 (to get the total path line 1, line 4, line 5) and
a=1, b=2, c=1 (to get line 2, line 3, line 5). This standard does not
require all combinations of paths.

An exception to the rule would be for H/W related items that can't be
forced in a unit test.

This standard doesn't require a specific means of accomplishing this
task. The preferred method would be for an automated tool to show this
rule is accomplished. Another method would be for unit test
documentation and unit test comments to show which cases are being
tested. The end result should be 100% statement coverage.

3\. Each conditional branch in the unit shall be executed with data at
each side of the boundary of the condition.

This is sometimes called "boundary value testing". For example, if there
is a condition "n \< 3", with n an integer, the largest value of n that
will yield true is 2, and the smallest value that will yield false is 3.
Thus the data at the boundary is n = 2 and n = 3.

This standard doesn't require a specific means of accomplishing this
task. The preferred method would be for an automated tool to show this
rule is accomplished. Another method would be for unit test
documentation and unit test comments to show which cases are being
tested. The end result should be 100% boundary value testing.

4\. All operations which might cause erroneous execution, such as divide
by zero, taking the square root of a negative number, etc., shall be
tested explicitly.

The goal is to have the error occur for the first time in testing, not
on flight.

If the erroneous execution can only happen for certain combinations of
paths, those paths must be tested or proved impossible.

For example, consider the following code:

int x;

int y;

if (b=1)

x = 1; /\* line 1 \*/

else

x = 0; /\* line 2 \*/

if (c=1)

\... /\* line 3 \*/

else

y = 2 / x; /\* line 4 \*/

Since line 4 may be erroneous depending on whether line 1 or line 2 is
taken, these total paths must be tested explicitly.

Exceptions to the rule are permitted if the operation is proven
impossible.

5\. All parameters and inputs to subprograms shall be tested with
nominal values, and with values at the extremes allowed by the
algorithm. Elements of input arrays like vectors and matrices shall have
distinct values. When the order of inputs to an operation matters, the
inputs shall have distinct values.

In some cases, particularly for floating point values, there is no
identifiable extreme value, and this requirement may be relaxed.

Testing with nominal values makes it easier to see if the results are
reasonable.

Testing with extreme values makes it easier to catch overflow and
underflow errors.

Testing with distinct values in arrays makes it easier to catch indexing
errors.

Testing with distinct values makes it easier to catch order errors when
the order of inputs matters, such as for matrix multiply or cross
products.

Table parameters also fall under this rule. However, if there are ground
enforced restrictions placed on the table values then the table values
don't have to be unit tested with these restricted values.

6\. Unit tests shall not require any changes to the module source code.
If a preprocessor is used, exactly the same preprocessor values must be
set for the final release and unit test builds.

Otherwise you are not testing the same code. The compiler may optimize
things differently if debug code is present in the module code; this may
hide a compiler bug.

Debug code is often used to print intermediate results to make the unit
test output more understandable. If this is necessary, provide storage
for the intermediate results in a place the unit test code can access.
It will often be desirable to provide the same intermediate results in
telemetry, so a problem can be diagnosed in flight. Using a debugger to
access the intermediate results is not a good idea, since it is much
less portable to other target systems.

Sometimes stubs or alternate modules are required for parts of the full
system that the unit under test interfaces with. These alternates are
part of the unit test, not part of the unit, and as such can contain
code specifically for performing the unit test, including reading input
files and writing output files.

Software that writes directly to hardware is always difficult to unit
test. The full system should be structured so that there is a very thin
layer that writes to hardware; all other software can be tested with an
alternate body for this hardware layer, that writes to an output file to
show that the hardware write functions are called. Then only the thin
hardware layer needs to be tested with the real hardware.

In some situations, it will only be possible to test some functionality
in the unit by modifying the unit source. This requires a waiver from
the product development lead, and must be done using preprocessor
settings (not manual editing), following the language-specific coding
standards for such cases.

7\. Deleted

8\. Unit tests shall be repeatable, producing identical results on each
run.

If possible, a single output file should be generated, that can be saved
and differenced against subsequent results. When an output file is
generated, it should contain all inputs, parameters, and outputs for
each test case, clearly labeled. If a different format is needed to
facilitate plotting or comparing with known good output, a second output
file should be generated. In some cases, multiple output files will make
more sense.

Having an output file to diff makes it easy to tell if the unit test has
passed or not.

Note that ANSI C does not define how many digits are in the exponent in
'e' format; where possible, use 'f' format instead.

In many cases, the requirements documentation for the unit will contain
a file of known good results; a test output file should be written to
reproduce that file, so it is easy to tell if the unit passes the test.
Or, have the unit test read the known good file, subtract it from the
unit output, and define the pass/fail criteria based on the average
error or other appropriate measure.

In other cases, the unit will be the first implementation of the
algorithm; then the algorithm author must approve the unit test output
file as being correct.

Using a plot as the definition of expected results is usually not a good
idea, since it is labor-intensive and somewhat subjective to "compare
plots" to see if the unit passes the test. Plots are very useful while
debugging, but they are not useful in the final unit test. The ability
to produce a suitable plot file should be maintained in the final unit
test, to facilitate debugging changes to the algorithm.

9\. Unit tests shall run without user interaction, as much as possible.

Having tests run automatically means they will be run more often, since
it is easy. A desirable scenario is to run all unit tests after each
major build; having an automated unit test makes this much easier.

It is highly recommended that all unit tests be run from a single
command (the main makefile, for example).

10\. If input data files are used by the unit test, they shall be
treated as source code for the purpose of configuration management. Such
input data files shall be in human readable form; if this is not
possible, a separate tool shall be provided to generate the actual test
input files from some human readable form.

It is desirable to allow comments in input files, to document the
general purpose of that particular test, and to label the various input
values to make it easy to edit them. One simple commenting convention is
to ignore lines at the start of the file that begin with a suitable
comment character.

11\. Deleted

12\. Deleted

[]{#__RefHeading___Toc50518833 .anchor}3 Guidelines

It is often useful to have a library of unit test support code,
providing standard output facilities or stubs. Such code should be
treated as part of the library, rather than as part of each unit test.

When parameters are used in equations, it is often useful to run
separate tests with the parameters set to 0, 1, and some nominal value
from a real scenario. Using 0 may reveal divide-by-zero problems; using
1 makes it easy to compute the equation, showing that the output is
simply related to the input; using a nominal value makes it easy to see
if the output is reasonable.

Sometimes units depend on a particular operational sequence for correct
operation. For example, a filter must be initialized before it is used.
Ideally, the unit can detect that it is not initialized, and report an
error in some way. If so, this case must be tested. However, in a real
time system it is often preferable to just assume the initialization is
done, rather than wasting time and code space doing the check each time.
In these cases, the test documentation must clearly state this
operational requirement, as justification for why that particular path
is not being tested.

If it seems to be hard to construct test cases that adequately cover the
unit, it may be that the unit is badly designed. One of the requirements
for a well-designed unit is that it be testable, and that the test be
understandable.

Ideally, unit tests would be run on the target hardware (or equivalent
test hardware), compiled with the same configuration switches as the
final target build. That would provide the best assurance that any
compiler, OS, or hardware bugs are found by unit testing.

However, it is time-consuming to write unit tests that can run on some
target operating systems, and the target systems are a limited resource.

In addition, with modern processors, operating systems, and compilers,
there are few bugs left to be found on the target hardware.

Finally, integration, build, and system testing are done on the target
hardware; any bugs in the compiler, operating system, or hardware should
be found then. Bugs found in these phases are more expensive to fix, but
we have decided that in general, that cost risk is less than the clear
cost of running all unit tests on target hardware.

Therefore this standard does not require running unit tests on the
target hardware. Unit tests are typically only run on desktop hardware,
with desktop compilers and compiler configuration switches (for example
to enable saving debug information). This allows finding logic, design
or requirement bugs under a friendlier debugging environment.

However, some projects are using less modern processors and compilers,
and should consider requiring running unit tests on target hardware.
This is particularly true of the smaller processors typically used in
instruments.

Other projects have target operating systems that are near-desktop in
friendliness, and can easily support running unit tests on the target
hardware.

Note that some small systems do not support input files; unit tests that
must run on these must have all input data coded in the executable. Such
systems may impose other restrictions on unit tests.