April 19, 1999

William J. Hughes Technical Center Test Suite Analysis Work

for "RWG Approved Changes 1.0 to DO-85A"

During TCAS flight testing, nuisance failures of the barometric altitude
credibility monitor had been observed. They occurred during vertical
rate maneuvers approaching 1.0g in acceleration. Discussions with
Mitre/CAASD indicated that the barometric altitude credibility monitor
was designed to only handle approximately 0.5g accelerations maximum. It
was decided that the altitude credibility window should be modified to
avoid regular occurrences of TCAS system failures during climbouts and
during resolution advisories (RAs) where the pilot responds
aggressively.

The Requirements Working Group (RWG) agreed in March 1999 to widen the
credibility window of the alpha-beta tracker used to track own aircraft
altitude (airdata-tracker) to ensure that the tracker would be able to
maintain track on own aircraft during a 1.25g vertical acceleration.
This change prompted a re-examination of the credibility window used by
the 100-ft and 25-ft altitude trackers. It was agreed that the three
altitude trackers should perform similarly. These issues and the
corresponding modifications to the system requirements are documented in
Change Proposal 98 (CP98).

Personnel at the William J. Hughes Technical Center performed an
analysis of the effects CP98's resolution had on the existing Collision
Avoidance System (CAS) test suite. The test suite verifies coverage of
the requirements defined by the CAS Requirement Specification (CRS) of
DO-185A.

The Technical Center was tasked with analyzing what effects the
parameter changes implemented to resolve CP98 had on the test results.
Further, if requirements coverage was no longer attained, the Technical
Center was to make additions to the test suite to reacquire coverage.
Since the Technical Center's TCAS simulation is based on pseudocode that
is a verified implementation of the CRS rather than the CRS itself, some
assumptions were necessary to perform these tasks. Technical Center
personnel assumed that identification of lost test suite coverage in the
pseudocode due to implementing CP98 could be projected onto the CRS.
This was done by using the CRS-pseudocode mappings produced for the
verification and validation efforts performed for the baseline TCAS
requirements. This baseline is referenced by TSO C119B, and is comprised
of DO-185A specifications and CPs 1-92 that were approved by the RWG.
These mappings were assumed to still be valid with CP98 implemented. It
was further assumed that there was only a loss of CRS coverage where
there was a corresponding loss of pseudocode statement coverage.
Therefore, only those requirements specified in the CRS that
corresponded to lines of pseudocode that were no longer executed by the
test suite were examined. This work identified three Macro transitions
not executed by the test suite that were executed before CP98's
resolution was implemented. Tests were then developed that executed
those transitions. Those new tests were then added to the test suite so
that the test suite again has full requirements coverage.

Below is a more detailed breakdown of the process used at the Technical
Center for its test suite analysis:

1.  Ran entire test suite (306 tests) against Technical Center\'s
    pseudocode based simulation (FTEG) before and after the resolution
    to CP98 was implemented.

```{=html}
<!-- -->
```
2.  Identified which tests produced different outputs after CP98\'s
    resolution. Since CP98\'s resolution included reducing the allowable
    range of values for two fields (Min_Softness & Max_Softness), only
    tests that produced differences not involving the values of those
    fields were considered.

```{=html}
<!-- -->
```
3.  Performed a before and after CP98 pseudocode statement coverage
    analysis for the entire test suite.

```{=html}
<!-- -->
```
4.  Observed a total of six lines of pseudocode that were no longer
    executed after CP98\'s resolution was implemented. These six lines
    of pseudocode were found to be three logical code blocks in three
    separate routines. (See Attachment A.)

```{=html}
<!-- -->
```
5.  Identified which tests the lost pseudocode coverage previously was
    executed in. Confirmed that the same tests which produced
    substantial differences in results (Step 2) were also the tests
    which previously provided coverage for the three blocks of code not
    executed post-CP98. Four tests were identified (EN03TS17, EN03TS71,
    EN03TS75, EN04TS22). Two of these tests previously hit the same code
    no longer covered by the test suite. (EN03TS17, EN03TS75)

```{=html}
<!-- -->
```
6.  Used traceability matrices developed from the CRS and pseudocode as
    it is defined for TSO C119B to map the blocks of pseudocode not
    executed to their corresponding Macro transitions in the CRS. (See
    Attachment B.) As anticipated, it was observed here and in step 4
    that the requirements no longer executed by the test suite was the
    failure of the altitude credibility check and the subsequent
    coasting of the tracker being used.

```{=html}
<!-- -->
```
7.  Used identified tests (EN03TS17, EN03TS71, EN04TS22) as a basis for
    developing new tests that would reacquire lost pseudocode coverage.
    This was done by adding accelerations that would fail the
    credibility check and coast the various trackers.

```{=html}
<!-- -->
```
8.  Three new tests were developed (EN03TS79, EN03TS80, EN04TS49) which
    gave the test suite (now 309 tests) the same pseudocode statement
    coverage it had before CP98 was implemented.

```{=html}
<!-- -->
```
9.  Performed a visual inspection of two sets of TSIM output files from
    the three new tests and the three tests from which they were
    derived. The first set was from the TSIM that was verified and
    validated by CRC and distributed with TSO C119B on December 4, 1998.
    The second set was an unverified version of TSIM provided by CRC
    that included the resolution to CP98.

```{=html}
<!-- -->
```
10. Observed the loss of CRS coverage caused by CP98 in the original
    tests. Observed those Macro transitions being reacquired with the
    three new tests run in the updated version of TSIM.

```{=html}
<!-- -->
```
11. Concluded that the new test suite of 309 tests regained full
    requirements coverage of the CRS.

```{=html}
<!-- -->
```
12. Presented findings to RWG, which approved of the process followed
    and concurred with findings.

```{=html}
<!-- -->
```
13. Received verification from the TCAS manufactures that they obtained
    the same results in their simulations and observed the reacquisition
    of CRS coverage with the three newly developed tests.

[ROUTINE]{.underline} TWENTY_FIVE_FT_TRACKING

[IN]{.underline} (ZFLG, ZREPT, T)

[INOUT]{.underline} (\^pointer to N entry\^);

\^access N entry\^;

[IF]{.underline} (N.INITCOMPLETED [EQ]{.underline} \$FALSE)

[THEN]{.underline} [IF]{.underline} (ZFLG [EQ]{.underline} \$TRUE)

[THEN]{.underline} [IF]{.underline} (CREDIBLE(\^N entry\^, T, ZREPT --
N.Z) [EQ]{.underline} \$TRUE)

[THEN]{.underline} [PERFORM]{.underline} Rate_initialization;

[ELSE]{.underline} ZDCHANGE = 0;

[IF]{.underline} (ZFLG [EQ]{.underline} \$TRUE)

[THEN]{.underline} [IF]{.underline} (CREDIBLE(\^N entry\^, T, ZREPT --
N.ZSAVE) [EQ]{.underline} \$TRUE)

[THEN]{.underline} [IF]{.underline} ((ZREPT [EQ]{.underline} N.ZSAVE)
[AND]{.underline}

(T -- N.TTRAN [GT]{.underline} PN.DTLONG))

[THEN]{.underline} [PERFORM]{.underline} Level_track_update;

[ELSE]{.underline} [PERFORM]{.underline} Track_smoothing;

[IF]{.underline} (ZREPT [NE]{.underline} N.ZSAVE)

[THEN]{.underline} N.TTRAN = T;

N.ZSAVE = ZREPT;

[IF]{.underline} (N.ZD [EQ]{.underline} 0)

[THEN]{.underline} N.CLASS = \$LEVEL;

[ELSE]{.underline} N.CLASS = \$TREND;

N.TDAT = T;

[ELSE]{.underline} [CALL]{.underline} TRACK_COAST

[IN]{.underline} (\^pointer to N entry\^, T);

*[ELSE]{.underline} [CALL]{.underline} TRACK_COAST*

*[IN]{.underline} (\^pointer to N entry\^, T);*

[PERFORM]{.underline} Rate_bound_update;

N.TUPDT = T;

[END]{.underline} TWENTY_FIVE_FT_TRACKING;

[ROUTINE]{.underline} ONE_HUNDRED_FT_TRACKING

[IN]{.underline} (ZFLG, ZREPT, T)

[INOUT]{.underline} (\^pointer to N entry\^);

\^access N entry\^;

[IF]{.underline} (ZFLG [EQ]{.underline} \$TRUE)

[THEN]{.underline} DZM = ZREPT -- N.ZSAVE;

[IF]{.underline} (CREDIBLE(\^N entry\^, T, DZM) [EQ]{.underline} \$TRUE)

[THEN]{.underline} [SET]{.underline} N.ALT_CREDIBLE;

[CLEAR]{.underline} OSTOSS;

[IF]{.underline} (DZM [EQ]{.underline} 0)

[THEN]{.underline} [PERFORM]{.underline} No_transition_update;

[PERFORM]{.underline} No_transition_firmness;

[ELSE]{.underline} [PERFORM]{.underline} Transition_update;

[IF]{.underline} (OSTOSS [EQ]{.underline} \$FALSE)

[THEN]{.underline} N.TDAT = T;

*[ELSE]{.underline} [CLEAR]{.underline} N.ALT_CREDIBLE;*

*[CALL]{.underline} TRACK_COAST*

*[IN]{.underline} (\^pointer to N entry\^, T);*

[ELSE]{.underline} [CLEAR]{.underline} N.ALT_CREDIBLE;

[CALL]{.underline} TRACK_COAST

[IN]{.underline} (\^pointer to N entry\^, T);

N.TUPDT = T;

[END]{.underline} ONE_HUNDRED_FT_TRACKING;

[ROUTINE]{.underline} AIRDATA_TRACKING

[IN]{.underline} (INIF, ZOK, ZIN, TIN, ZDIN)

[INOUT]{.underline} (\^pointer to OAT entry\^);

\<This tracker is used to track own altitude data when the source of
that data is an airdata

computer or some other source of comparably finely quantized altitude
data\>

[IF]{.underline} (INIF [EQ]{.underline} \$TRUE)

[THEN]{.underline} \^create OAT entry\^;

OAT.Z = ZIN;

OAT.ZD = ZDIN;

OAT.TDAT = TIN;

OAT.TUPD = TIN;

OAT.SOFT = P.MAXSOFT;

[SET]{.underline} OAT.ALT_CREDIBLE;

\^Return address of OAT entry\^;

[ELSE]{.underline} \^access OAT entry\^;

ZPD = OAT.Z + OAT.ZD \* (TIN -- OAT.TUPD);

[IF]{.underline} (ZOK [EQ]{.underline} \$TRUE)

[THEN]{.underline} [IF]{.underline} (ABS(ZIN -- ZPD) [LE]{.underline}
P.CREDZADC\*OAT.SOFT)

[THEN]{.underline} [SET]{.underline} OAT.ALT_CREDIBLE;

OAT.Z = ZPD + P.ALFAO \* (ZIN -- ZPD);

OAT.ZD = OAT.ZD + P.BETAO \* (ZIN -- ZPD) /

(TIN -- OAT.TDAT);

OAT.TDAT = TIN;

OAT.SOFT = MAX(OAT.SOFT -- 1, P.MINSOFT);

*[ELSE]{.underline} [CLEAR]{.underline} OAT.ALT_CREDIBLE;*

*OAT.Z =ZPD;*

*OAT.SOFT = MIN(OAT.SOFT + 1, P.MAXSOFT);*

[ELSE]{.underline} [CLEAR]{.underline} OAT.ALT_CREDIBLE;

OAT.Z =ZPD;

OAT.SOFT = MIN(OAT.SOFT + 1, P.MAXSOFT);

OAT.TUPD = TIN;

[END]{.underline} AIRDATA_TRACKING;

  --------------------------------------------------------------------------------------------------------
  Process            Line   CRS       Interface, State, Macro, or        Transition   Column or
                            Section   Function                                        Abbreviation
  ------------------ ------ --------- ---------------------------------- ------------ --------------------
  AIRDATA_TRACKING   1      N/A       Does_Not_Map                       1            1

  AIRDATA_TRACKING   2      4.76      Own_Fine_Tracked_Alt               1            1

  AIRDATA_TRACKING   3      4.77      Own_Fine_Tracked_Alt_Rate          1            1

  AIRDATA_TRACKING   4      4.51      Last_Own_Fine_Track_Update_Time    1            1

  AIRDATA_TRACKING   5      3.57      Own_Altitude_Coast                 1            dt

  AIRDATA_TRACKING   5      4.76      Own_Fine_Tracked_Alt               1            dt

  AIRDATA_TRACKING   5      4.77      Own_Fine_Tracked_Alt_Rate          1            dt

  AIRDATA_TRACKING   6      3.57      Own_Altitude_Coast                 1            SOFTNESS_FACTOR

  AIRDATA_TRACKING   6      2.1.18    Own_Tracker_Softness               1            1

  AIRDATA_TRACKING   7      1.3.6     Own_Altitude_Credibility_Message   1            OWN_ALT_CREDIBLE

  AIRDATA_TRACKING   8      N/A       Does_Not_Map                       1            1

  AIRDATA_TRACKING   9      N/A       Does_Not_Map                       1            1

  AIRDATA_TRACKING   10     3.57      Own_Altitude_Coast                 1            1

  AIRDATA_TRACKING   10     3.57      Own_Altitude_Coast                 1            dt

  AIRDATA_TRACKING   10     3.57      Own_Altitude_Coast                 1            PREDICTED_ALTITUDE

  AIRDATA_TRACKING   10     4.76      Own_Fine_Tracked_Alt               1            dt

  AIRDATA_TRACKING   10     4.76      Own_Fine_Tracked_Alt               1            ZP

  AIRDATA_TRACKING   10     4.77      Own_Fine_Tracked_Alt_Rate          1            dt

  AIRDATA_TRACKING   10     4.77      Own_Fine_Tracked_Alt_Rate          1            ZP

  AIRDATA_TRACKING   11     1.3.6     Own_Altitude_Credibility_Message   1            OWN_ALT_CREDIBLE

  AIRDATA_TRACKING   12     4.76      Own_Fine_Tracked_Alt               1            1

  AIRDATA_TRACKING   13     4.77      Own_Fine_Tracked_Alt_Rate          1            1

  AIRDATA_TRACKING   13     4.77      Own_Fine_Tracked_Alt_Rate          1            CALCULATED_AR

  AIRDATA_TRACKING   14     4.51      Last_Own_Fine_Track_Update_Time    1            1

  AIRDATA_TRACKING   15     3.57      Own_Altitude_Coast                 1            SOFTNESS_FACTOR

  AIRDATA_TRACKING   15     2.1.18    Own_Tracker_Softness               3            1

  AIRDATA_TRACKING   15     2.1.18    Own_Tracker_Softness               3            2

  AIRDATA_TRACKING   16     1.3.6     Own_Altitude_Credibility_Message   1            OWN_ALT_CREDIBLE

  AIRDATA_TRACKING   17     3.57      Own_Altitude_Coast                 1            1

  AIRDATA_TRACKING   17     4.76      Own_Fine_Tracked_Alt               1            1

  AIRDATA_TRACKING   18     3.57      Own_Altitude_Coast                 1            SOFTNESS_FACTOR

  AIRDATA_TRACKING   18     2.1.18    Own_Tracker_Softness               2            1

  AIRDATA_TRACKING   19     1.3.6     Own_Altitude_Credibility_Message   1            OWN_ALT_CREDIBLE

  AIRDATA_TRACKING   20     3.57      Own_Altitude_Coast                 1            2

  AIRDATA_TRACKING   20     4.76      Own_Fine_Tracked_Alt               1            1

  AIRDATA_TRACKING   21     3.57      Own_Altitude_Coast                 1            SOFTNESS_FACTOR

  AIRDATA_TRACKING   21     2.1.18    Own_Tracker_Softness               2            1

  AIRDATA_TRACKING   22     3.57      Own_Altitude_Coast                 1            dt

  AIRDATA_TRACKING   22     4.76      Own_Fine_Tracked_Alt               1            dt

  AIRDATA_TRACKING   22     4.77      Own_Fine_Tracked_Alt_Rate          1            dt

                                                                                      
  --------------------------------------------------------------------------------------------------------

  -------------------------------------------------------------------------------------------------------------------
  Process                   Line   CRS       Interface, State, Macro, or        Transition   Column or Abbreviation
                                   Section   Function                                        
  ------------------------- ------ --------- ---------------------------------- ------------ ------------------------
  CREDIBLE                  1      3.98      VT_25ft_Coast                      1            2

  CREDIBLE                  2      3.99      VT_Credible_Report                 1            1

  CREDIBLE                  3      3.92      VT_100ft_Coast                     1            TDAT

  CREDIBLE                  3      3.92      VT_100ft_Coast                     1            TSTART

  CREDIBLE                  3      3.98      VT_25ft_Coast                      1            TDAT

  CREDIBLE                  3      3.98      VT_25ft_Coast                      1            TSTART

  CREDIBLE                  3      3.99      VT_Credible_Report                 1            2

  CREDIBLE                  3      3.99      VT_Credible_Report                 1            DRT

  CREDIBLE                  4      3.92      VT_100ft_Coast                     1            TDAT

  CREDIBLE                  4      3.92      VT_100ft_Coast                     1            TSTART

  CREDIBLE                  4      3.92      VT_100ft_Coast                     1            ZD

  CREDIBLE                  4      3.98      VT_25ft_Coast                      1            TDAT

  CREDIBLE                  4      3.98      VT_25ft_Coast                      1            TSTART

  CREDIBLE                  4      3.98      VT_25ft_Coast                      1            ZD

  CREDIBLE                  4      3.99      VT_Credible_Report                 1            3

  CREDIBLE                  4      3.99      VT_Credible_Report                 1            DRT

                                                                                             

  ONE_HUNDRED_FT_TRACKING   1      N/A       Does_Not_Map                       1            1

  ONE_HUNDRED_FT_TRACKING   2      2.3.3.2   Bin_Direction                      4            1

  ONE_HUNDRED_FT_TRACKING   2      2.3.3.2   Bin_Direction                      5            1

  ONE_HUNDRED_FT_TRACKING   2      1.3.6     Own_Altitude_Credibility_Message   1            DZM

  ONE_HUNDRED_FT_TRACKING   2      4.104     VT_100ft_Bins_Crossed              1            1

  ONE_HUNDRED_FT_TRACKING   2      3.92      VT_100ft_Coast                     1            DZM

  ONE_HUNDRED_FT_TRACKING   2      3.93      VT_100ft_No_Transition             1            1

  ONE_HUNDRED_FT_TRACKING   2      4.117     VT_100ft_Tracked_Alt_Rate          1            GUESS_TRACKED_ALT_RATE

  ONE_HUNDRED_FT_TRACKING   2      4.118     VT_100ft_Tracked_Alt_Rate_Inner    1            DELTA_Z

  ONE_HUNDRED_FT_TRACKING   2      4.119     VT_100ft_Tracked_Alt_Rate_Outer    1            DELTA_Z

  ONE_HUNDRED_FT_TRACKING   2      3.96      VT_100ft_Transition                1            1

  ONE_HUNDRED_FT_TRACKING   3      1.3.6     Own_Altitude_Credibility_Message   1            OWN_ALT_CREDIBLE

  ONE_HUNDRED_FT_TRACKING   4      3.94      VT_100ft_Spurious_Oscillation      1            1

  ONE_HUNDRED_FT_TRACKING   4      3.94      VT_100ft_Spurious_Oscillation      1            2

  ONE_HUNDRED_FT_TRACKING   5      N/A       Does_Not_Map                       1            1

  ONE_HUNDRED_FT_TRACKING   6      N/A       Does_Not_Map                       1            1

  ONE_HUNDRED_FT_TRACKING   7      N/A       Does_Not_Map                       1            1

  ONE_HUNDRED_FT_TRACKING   8      4.11      VT_100ft_Last_Alt_Report_Time      1            1

  ONE_HUNDRED_FT_TRACKING   9      1.3.6     Own_Altitude_Credibility_Message   1            OWN_ALT_CREDIBLE

  ONE_HUNDRED_FT_TRACKING   10     3.92      VT_100ft_Coast                     1            2

  ONE_HUNDRED_FT_TRACKING   11     1.3.6     Own_Altitude_Credibility_Message   1            OWN_ALT_CREDIBLE

  ONE_HUNDRED_FT_TRACKING   12     3.92      VT_100ft_Coast                     1            1

  ONE_HUNDRED_FT_TRACKING   13     N/A       Does_Not_Map                       1            1

                                                                                             
  -------------------------------------------------------------------------------------------------------------------

  ----------------------------------------------------------------------------------------------------------
  Process                   Line   CRS       Interface, State, Macro, or    Transition   Column or
                                   Section   Function                                    Abbreviation
  ------------------------- ------ --------- ------------------------------ ------------ -------------------
  TWENTY_FIVE_FT_TRACKING   1      N/A       Does_Not_Map                   1            1

  TWENTY_FIVE_FT_TRACKING   2      N/A       Does_Not_Map                   1            1

  TWENTY_FIVE_FT_TRACKING   3      4.127     VT_25ft_AB_Tracked_Alt_Accel   1            1

  TWENTY_FIVE_FT_TRACKING   4      N/A       Does_Not_Map                   1            1

  TWENTY_FIVE_FT_TRACKING   5      N/A       Does_Not_Map                   1            1

  TWENTY_FIVE_FT_TRACKING   6      4.139     VT_25ft_Transition_Time        1            1

  TWENTY_FIVE_FT_TRACKING   7      4.132     VT_25ft_Previous_Alt_Report    1            1

  TWENTY_FIVE_FT_TRACKING   8      2.3.4.3   25ft_Bin_Transition_Status     4            1

  TWENTY_FIVE_FT_TRACKING   9      2.3.4.3   25ft_Bin_Transition_Status     5            1

  TWENTY_FIVE_FT_TRACKING   10     4.13      VT_25ft_Last_Alt_Report_Time   1            1

  TWENTY_FIVE_FT_TRACKING   11     3.98      VT_25ft_Coast                  1            2

  TWENTY_FIVE_FT_TRACKING   11     3.98      VT_25ft_Coast                  1            DZM

  TWENTY_FIVE_FT_TRACKING   12     3.98      VT_25ft_Coast                  1            1

  TWENTY_FIVE_FT_TRACKING   13     N/A       Does_Not_Map                   1            1

  TWENTY_FIVE_FT_TRACKING   14     N/A       Does_Not_Map                   1            1
  ----------------------------------------------------------------------------------------------------------