June 20

NASA/

Fligh

Perfo

High

R. Micha

Boeing R

Brian T. 

Kara A. L

Langley R

013 

/TP–2013

ht Crew

forman

h-Dens

ael Norman 

Research and

Baxley, Cath

Latorella, an

Research Ce

3-218007 

w Wor

nce Wh

sity Te

d Technology

herine A. Ad

nd James R. 

enter, Hamp

rkload

hen Us

ermina

y, Hampton,

dams, Kyle K

Comstock, J

pton, Virginia

d, Acce

sing D

al Area

 Virginia 

K. E. Ellis, 

Jr. 

a

eptabil

Data Co

a Simu

lity, an

omm in

ulation

nd 

n a 
NASA STI Program . . . in Profile

Since its founding, NASA has been dedicated to the 

advancement of aeronautics and space science. The 

NASA scientific and technical information (STI) 

program plays a key part in helping NASA maintain 

this important role. 

The NASA STI program operates under the 

auspices of the Agency Chief Information Officer. 

It collects, organizes, provides for archiving, and 

disseminates NASA’s STI. The NASA STI 

program provides access to the NASA Aeronautics 

and Space Database and its public interface, the 

NASA Technical Report Server, thus providing one 

of the largest collections of aeronautical and space 

science STI in the world. Results are published in 

both non-NASA channels and by NASA in the 

NASA STI Report Series, which includes the 

following report types: 

 TECHNICAL PUBLICATION. Reports of 

completed research or a major significant phase 

of research that present the results of NASA 

Programs and include extensive data or 

theoretical analysis. Includes compilations of 

significant scientific and technical data and 

information deemed to be of continuing 

reference value. NASA counterpart of peerreviewed formal professional papers, but 

having less stringent limitations on manuscript 

length and extent of graphic presentations. 

 TECHNICAL MEMORANDUM. Scientific 

and technical findings that are preliminary or of 

specialized interest, e.g., quick release reports, 

working papers, and bibliographies that contain 

minimal annotation. Does not contain extensive 

analysis. 

 CONTRACTOR REPORT. Scientific and 

technical findings by NASA-sponsored 

contractors and grantees. 

 CONFERENCE PUBLICATION. 

Collected papers from scientific and 

technical conferences, symposia, seminars, 

or other meetings sponsored or cosponsored by NASA. 

 SPECIAL PUBLICATION. Scientific, 

technical, or historical information from 

NASA programs, projects, and missions, 

often concerned with subjects having 

substantial public interest. 

 TECHNICAL TRANSLATION. 

English-language translations of foreign 

scientific and technical material pertinent to 

NASA’s mission. 

Specialized services also include organizing 

and publishing research results, distributing 

specialized research announcements and feeds, 

providing information desk and personal search 

support, and enabling data exchange services. 

For more information about the NASA STI 

program, see the following: 

 Access the NASA STI program home page 

at http://www.sti.nasa.gov

 E-mail your question to help@sti.nasa.gov

 Fax your question to the NASA STI 

Information Desk at 443-757-5803 

 Phone the NASA STI Information Desk at 

443-757-5802 

 Write to: 

 STI Information Desk 

 NASA Center for AeroSpace Information 

 7115 Standard Drive 

 Hanover, MD 21076-1320
National A

Space Adm

Langley R

Hampton, 

June 20

NASA/

Fligh

Perfo

High

R. Micha

Boeing R

Brian T. 

Kara A. L

Langley R

Aeronautics and

ministration 

esearch Center

Virginia 23681

013 

/TP–2013

ht Crew

forman

h-Dens

ael Norman 

Research and

Baxley, Cath

Latorella, an

Research Ce

d 

r 

1-2199 

3-218007 

w Wor

nce Wh

sity Te

d Technology

hy A. Adams

nd James R. 

enter, Hamp

rkload

hen Us

ermina

y, Hampton,

s, Kyle K. E.

Comstock, J

pton, Virginia

d, Acce

sing D

al Area

 Virginia 

 Ellis, 

Jr. 

a

eptabil

Data Co

a Simu

lity, an

omm in

ulation

nd 

n a 
Available from: 

NASA Center for AeroSpace Information 

7115 Standard Drive 

Hanover, MD 21076-1320 

443-757-5802 

The use of trademarks or names of manufacturers in this report is for accurate reporting and does not 

constitute an official endorsement, either expressed or implied, of such products or manufacturers by the 

National Aeronautics and Space Administration. 
i

Foreword

This NASA Technical Paper is a refinement of a report provided on 28 July 2010 to Mr. Levent Ileri, 

of the FAA Data Comm Program office, titled “NASA / FAA Data Comm Airside Human-in-the-Loop 

Simulation.” That document served as the Final Report specified in the FAA/NASA Interagency 

Agreement IA1-973, Technical Direction 1, Paragraph 1.1.10 (FAA Agreement DTFAWA-09-A-80018). 

Generally FAA documents are not publically accessible, therefore with concurrence from both FAA and 

NASA, this document republishes that information as a NASA TP with administrative corrections and 

some formatting changes. No data was added or changed, however typographical errors were corrected, 

clarifying text and figures were added, and in a few instances different analysis was performed.

The authors thank Mr. Ileri and the FAA Data Comm Program for the opportunity to conduct this 

challenging human-in-the-loop simulation at NASA Langley Research Center.

The authors also express sincere thanks and gratitude to the many implementation team members in

the D-107 Simulation Development and Analysis Branch at NASA Langley Research Center that we 

worked with on a daily basis for a year. In particular, Paul Sugden wrote specialized software necessary 

to conduct this experiment and Jerry Karwac created the graphical displays that were key to this work. 

Others from D-107 that contributed significantly over many hours include Tom Feigh, Dennis Frasca, 

Chris Harrison, Sonia Herndon, Brian Hutchinson, Kemper Kibler, Wendy Pifer, Cassie Ruddiman, 

Darrel Sacra, Phil Smith, Joe Whiting, and Brittany Williams. Individuals from other Branches the 

authors wish to thank include William Lynn and Dan Burdette who made important contributions to the 

work required to install and use the oculometer system, and Regina Johns who did an incredible job 

recruiting subject pilots, as well as arranging their travel and accommodations.
ii
iii

Table of Contents

Abbreviations.............................................................................................................................................xiv

1 Introduction ........................................................................................................................................2

2 Data Comm Literature Review...........................................................................................................4

2.1 When to use Voice or Data Comm ...............................................................................................4

2.2 Prioritization of Voice and Data Comm........................................................................................4

2. 3 Use of Data Comm Reduces Need for Voice Communication.....................................................5

2.4 Data Comm Acceptability.............................................................................................................5

2.5 Understanding Communication ....................................................................................................6

2.6 Flight Crew Response Time..........................................................................................................7

2.7 Flight Crew Head Up Time...........................................................................................................7

2.8 Cockpit Graphical Displays..........................................................................................................7

2.9 Simultaneous Use of Voice and Data Comm................................................................................8

2.10 Experiment Design Decisions from Literature Review ................................................................9

3 Methodology..................................................................................................................................... 10

3.1 Experiment Hypotheses .............................................................................................................. 10

3.2 Independent Variables................................................................................................................. 10

3.2.1 Communication Modality ............................................................................................... 11

3.2.2 Display Methodology...................................................................................................... 11

3.3 Scenario Descriptions ................................................................................................................. 12

3.3.1 Airport............................................................................................................................. 12

3.3.2 Arrival Routes and Instrument Approaches.................................................................... 13

3.3.3 Taxi Routes..................................................................................................................... 14

3.4 Experiment Design...................................................................................................................... 15

3.5 Dependent Variables................................................................................................................... 16

3.6 Parameters and Data Analysis Techniques................................................................................. 16

3.6.1 Data Comm Message Response Time............................................................................. 16

3.6.2 Flight Technical Performance ......................................................................................... 17

3.6.3 Head Tracking................................................................................................................. 17

3.6.4 Biographical Data Questionnaire .................................................................................... 18

3.6.5 Post-Scenario Questionnaire ........................................................................................... 18

3.6.6 Post-Experiment Questionnaire ...................................................................................... 19

3.6.7 Post-Experiment Debrief................................................................................................. 19

3.6.8 Audio and Video Recordings.......................................................................................... 19

3.7 Research Facilities ...................................................................................................................... 20

3.7.1 Hardware and Software Configuration ........................................................................... 20

3.7.2 Additional Simulation Capabilities................................................................................. 21
iv

3.7.3 Oculometer Hardware and Software............................................................................... 21

3.8 Data Comm Messages and Displays........................................................................................... 21

3.9 Experiment Protocol ................................................................................................................... 27

3.10 Subject Pilots .............................................................................................................................. 27

3.10.1 Requirements .................................................................................................................. 27

3.10.2 Subject Pilot Experience Level ....................................................................................... 28

4 Results and Discussion ..................................................................................................................... 29

4.1 Data Comm Message Response Time......................................................................................... 29

4.1.1 Response Times Based On All Data ............................................................................... 29

4.1.2 Response Time Distributions.......................................................................................... 33

4.2 Flight Technical Performance..................................................................................................... 33

4.2.1 Power Spectral Density of NWS Rate ............................................................................ 34

4.2.2 Taxi Speed ...................................................................................................................... 36

4.3 Head Tracking............................................................................................................................. 38

4.3.1 Head Up Aggregate Results............................................................................................ 38

4.3.2 Head Up Time By Altitude Bands.................................................................................. 40

4.3.3 Head up Time During Taxi Operations........................................................................... 43

4.4 Post-Scenario Questionnaire Results .......................................................................................... 45

4.4.1 Post-Scenario Ratings on Workload ............................................................................... 45

4.4.2 Post-Scenario Ratings on Situation Awareness .............................................................. 48

4.4.3 Post-Scenario Ratings on Acceptability.......................................................................... 50

4.5 Post-Experiment Questionnaire Results...................................................................................... 53

4.5.1 Post-Experiment Ratings on Workload Comparison ...................................................... 53

4.5.2 Post-Experiment Ratings on Situation Awareness Comparison ..................................... 54

4.5.3 Post-Experiment Ratings on Acceptability of Expected D-TAXI Messages.................. 55

4.5.4 Post-Experiment Ratings on Crew Coordination............................................................ 55

4.5.5 Post-Experiment Summary Questions ............................................................................ 56

4.6 Verbal debrief comments............................................................................................................ 58

4.7 Operational errors ....................................................................................................................... 59

5 Synthesis of Results.......................................................................................................................... 60

5.1 Impact of Communication Modality on Flight Crew in the Terminal Area ............................... 60

5.2 Impact of Display Methodology on Flight Crew in the Terminal Area...................................... 61

5.3 Acceptability of Data Comm Use to Flight Crew in the Terminal Area..................................... 62

6 Exploratory Study: Rare Event Scenario and Trust Assessment..................................................... 64

6.1 Rare Event Scenarios.................................................................................................................. 64

6.2 Trust Questions and Results........................................................................................................ 65

7 Conclusion........................................................................................................................................ 68
v

8 Recommendations and Future Research........................................................................................... 69

8.1 Controller-Pilot Operational Procedure Recommendations........................................................ 69

8.2 Aircraft Avionics Implementation Recommendations................................................................ 69

8.3 Future Research Issues................................................................................................................ 70

References................................................................................................................................................... 72

Appendix A: FAA/NASA Interagency Agreement .............................................................................. 75

A.1 FAA/NASA Interagency Agreement .......................................................................................... 75

A.2 Assumptions Contained in Interagency Agreement and Addendum .......................................... 75

Appendix B: Scenario Descriptions ..................................................................................................... 78

B.1 Scenario Case Number by Display Type, Communication, and Flight Phase ............................ 78

B.2 Taxi Routes for Arrival Scenarios .............................................................................................. 80

B.3 Taxi Routes for Departure Scenarios.......................................................................................... 83

Appendix C: Biographical Questionnaire............................................................................................. 87

Appendix D: Post-Scenario Questionnaire ........................................................................................... 88

D.1 Workload During Scenario by Phase of Flight ........................................................................... 88

D.2 Situation Awareness by Phase of Flight ..................................................................................... 89

D.3 Sources of Information................................................................................................................ 90

D.4 Crew Interaction.......................................................................................................................... 90

D.5 Acceptability of “Expected Taxi” and “Taxi” Clearances.......................................................... 91

Appendix E: Post-Experiment Questionnaire ...................................................................................... 92

E.1 Workload Comparison ................................................................................................................ 92

E.2 Situation Awareness Comparison ............................................................................................... 92

E.3 Acceptability of “Expected Taxi” Data Comm message ............................................................ 93

E.4 Trust in the System ..................................................................................................................... 93

E.5 Crew Coordination Support ........................................................................................................ 94

E.6 Summary ..................................................................................................................................... 95

Appendix F: Oculometer Apparatus .................................................................................................... 96

Appendix G: Data Comm Message Format.......................................................................................... 98

Appendix H: Data Comm Uplink Messages......................................................................................... 99

Appendix I: Flight Crew Training Program...................................................................................... 102

Appendix J: Scenario Briefings......................................................................................................... 128

J.1 NORWICH3 (Arrival to Runway 33L)..................................................................................... 128

J.2 SCUPP4 (Arrival to Runway 27).............................................................................................. 128

J.3 Runway 27 (Departure to Runway 27) ..................................................................................... 129

J.4 Runway 33L (Departure to Runway 33L) ................................................................................ 129
vi

Appendix K: Response Time, Technical Performance, and Raw Data .............................................. 130

K.1 Message Response Time by Altitude........................................................................................ 130

K.2 Technical Performance ............................................................................................................. 131

K.3 Raw Data by Flight Crew.......................................................................................................... 132

K.3.1 Crew #1......................................................................................................................... 132

K.3.2 Crew #2......................................................................................................................... 135

K.3.3 Crew #3......................................................................................................................... 138

K.3.4 Crew #4......................................................................................................................... 141

K.3.5 Crew #5......................................................................................................................... 144

K.3.6 Crew #6......................................................................................................................... 147

K.3.7 Crew #7......................................................................................................................... 150

K.3.8 Crew #8......................................................................................................................... 153

K.3.9 Crew #........................................................................................................................... 156

K.3.10 Crew #10....................................................................................................................... 159

K.3.11 Crew #11....................................................................................................................... 162

Appendix L: Data Comm Response Time Distributions.................................................................... 165

Appendix M: Data Comm Response Time Tables for SC-214 ........................................................... 169

Appendix N: Oculometer Results....................................................................................................... 171

N.1 General Information.................................................................................................................. 171

N.1.1 Interpreting the ANOVA .............................................................................................. 171

N.1.2 Interpreting the Graphical Outputs................................................................................ 171

N.2 Arrival: High altitude messages............................................................................................... 172

N.3 Arrival: Medium altitude messages ......................................................................................... 175

N.4 Arrival: Low altitude messages................................................................................................ 179

N.5 Arrival: Taxi operations........................................................................................................... 182

N.6 Departure: entire scenario ........................................................................................................ 190

N.7 Arrival versus Departure ANOVA ........................................................................................... 193

Appendix O: Post-Scenario Questionnaire Results ............................................................................ 198

O.1 Workload (Bedford) rating........................................................................................................ 198

O.2 Situation Awareness.................................................................................................................. 204

O.3 Acceptability of “Expected Taxi” and “Taxi” clearances......................................................... 212

Appendix P: Post-Experiment Questionnaire Results........................................................................ 224

P.1 Workload Comparison .............................................................................................................. 224

P.2 Situation Awareness.................................................................................................................. 226

P.3 Acceptability of Expected Taxi Data Comm message.............................................................. 228

P.4 Trust in the System ................................................................................................................... 229

P.5 Crew Coordination.................................................................................................................... 231

P.6 Summary .................................................................................................................................. 233
vii

List of Figures

Figure 1. Boston Logan International airport diagram............................................................................... 12

Figure 2. KBOS airspace and arrival routes .............................................................................................. 13

Figure 3. Excerpt of NORWICH THREE arrival ...................................................................................... 14

Figure 4. Excerpt of SCUPP FOUR arrival ............................................................................................... 14

Figure 5. Integration Flight Deck simulator............................................................................................... 20

Figure 6. Control Display Unit (CDU) ...................................................................................................... 23

Figure 7. ATC Index (left) and ATC Request (right) pages ...................................................................... 23

Figure 8. ATC Log (left) and Downlink Response (right) pages .............................................................. 24

Figure 9. Open Expected D-TAXI message on CDU (left) and display on ND (right)............................. 24

Figure 10. Open D-TAXI message on CDU (left) and display on ND (right)........................................... 25

Figure 11. Accepted D-TAXI message on CDU (left) and display on ND (right) .................................... 25

Figure 12. Open Amended D-TAXI message on CDU (left) and display on ND (right) .......................... 26

Figure 13. Accepted Amended D-TAXI message on CDU (left) and display on ND (right).................... 26

Figure 14. Message response time by condition ........................................................................................ 30

Figure 15. Response time by display methodology ................................................................................... 31

Figure 16. Message response time by phase of flight ................................................................................ 31

Figure 17. Mean response time by message type....................................................................................... 32

Figure 18. Distribution of flight crew Data Comm response times........................................................... 33

Figure 19. NWS rate PSD during arrival (left) and departure (right) by condition ................................... 34

Figure 20. Taxi speed during arrival (left) and departure (right) by condition .......................................... 36

Figure 21. Percent head up time for PF (left) and PM (right) by location ................................................. 39

Figure 22. Percent head up time for PF (left) and PM (right) by altitude band and condition .................. 40

Figure 23. Percent head up time during arrival (left) and departure (right) taxi operations ...................... 43

Figure 24. Inflight (top) and Surface (bottom) workload ratings by condition.......................................... 46

Figure 25. Inflight workload ratings by position and by condition............................................................ 47

Figure 26. Surface workload rating by position and by condition ............................................................. 48

Figure 27. SART ratings for inflight operations by condition ................................................................... 49

Figure 28. SART ratings for only surface arrival operations by condition................................................ 49

Figure 29. SART ratings for only surface departure operations by condition ........................................... 50

Figure 30. Data Comm acceptability rating by condition .......................................................................... 52

Figure 31. Display preference for lowest workload by condition.............................................................. 54
viii

Figure 32. Display preference for highest situation awareness by condition............................................. 54

Figure 33. Display preference for effective crew coordination by condition ............................................ 56

Figure 34. Display preference for highest trust by condition..................................................................... 65

Figure 35. Runway 27 Arrival A ............................................................................................................... 80

Figure 36. Runway 27 Arrival B and C ..................................................................................................... 81

Figure 37. Runway 33L Arrival A, B, and C............................................................................................. 82

Figure 38. Runway 27 Departure A, C, and T ........................................................................................... 83

Figure 39. Runway 27 Departure B ........................................................................................................... 84

Figure 40. Runway 33L Departure A and C.............................................................................................. 85

Figure 41. Runway 33L Departure B......................................................................................................... 86

Figure 42. Bedford work scale................................................................................................................... 88

Figure 43. Oculometer and IR Flasher....................................................................................................... 96

Figure 44. Eye Gaze Vector....................................................................................................................... 96

Figure 45. Location of Oculometers and IR Flashers in IFD Simulator.................................................... 97

Figure 46. Mean response time to “Expected Taxi” message.................................................................. 130

Figure 47. Mean response time to other Data Comm messages .............................................................. 130

Figure 48. Flight director error by condition ........................................................................................... 131

Figure 49. Flight director error by altitude .............................................................................................. 131

Figure 50. Response time to Pushback message...................................................................................... 166

Figure 51. Response time to Start message.............................................................................................. 166

Figure 52. Response time to Expected Taxi message .............................................................................. 166

Figure 53. Response time to Taxi message.............................................................................................. 167

Figure 54. Response time to Amended Taxi............................................................................................ 167

Figure 55. Response time to Frequency change message ........................................................................ 167

Figure 56. Response time to ATIS message ............................................................................................ 168

Figure 57. Response time to Altimeter change ........................................................................................ 168

Figure 58. Residual plots for percent head up (16 – 14,000 feet MSL)................................................... 174

Figure 59. Main effects plot (fitted means) for percent head up (16 – 14,000 feet MSL)....................... 174

Figure 60. Interaction plot (fitted means) for percent head up (16 – 14,000 feet MSL).......................... 175

Figure 61. Residual plots for percent head up (10 – 8,000 feet MSL)..................................................... 177

Figure 62. Main effects plot (fitted means) for percent head up (10 – 8,000 feet MSL)......................... 178

Figure 63. Interaction plot (fitted means) for percent head up (10 – 8,000 feet MSL)............................ 178

Figure 64. Residual plots for percent head up (7 – 5,000 feet MSL)....................................................... 180

Figure 65. Main effects plot (fitted means) for percent head up (7 – 5,000 feet MSL)........................... 181

Figure 66. Interaction plot (fitted means) for percent head up (7 – 5,000 feet MSL).............................. 181
ix

Figure 67. Residual plots for percent head up (below 80 knots) ............................................................. 188

Figure 68. Main effects plot (fitted means) for percent head up (below 80 knots).................................. 188

Figure 69. Interaction plot (fitted means) for percent head up (below 80 knots) .................................... 189

Figure 70. Residual plots for percent head up (entire run) ...................................................................... 192

Figure 71. Main effects plot (fitted means) for percent head up (entire run)........................................... 192

Figure 72. Interaction plot (fitted means) for percent head up (entire run) ............................................. 193

Figure 73. Residual plots for percent head up (entire run) ...................................................................... 196

Figure 74. Main effects plot (fitted means) for percent head up (entire run)........................................... 196

Figure 75. Interaction plot (fitted means) for percent head up (entire run) ............................................. 197
x

List of Tables

Table 1. Experimental design matrix ......................................................................................................... 15

Table 2. Scenario types.............................................................................................................................. 16

Table 3. Data Comm messages per crew and entire experiment ............................................................... 22

Table 4. Subject pilot experience level in years and hours........................................................................ 28

Table 5. Mean response time and standard deviation by condition ............................................................ 30

Table 6. Pairwise comparisons of response time by condition (= .05, HSD = 5.47).............................. 30

Table 7. Mean response time and standard deviation by display methodology.......................................... 31

Table 8. Mean response time and standard deviation by phase of flight ................................................... 31

Table 9. Mean response time and standard deviation by message type..................................................... 32

Table 10. Pairwise comparisons of response time by message type (=0.05, HSD=6.09) ....................... 32

Table 11. NWS PSD for arrival scenarios by condition ............................................................................ 34

Table 12. Pairwise comparisons of arrival NWS PSD by condition (=0.05, HSD=44.6)....................... 34

Table 13. NWS PSD for departure scenarios by condition........................................................................ 35

Table 14. Pairwise comparisons of departure NWS PSD by condition (= .05, HSD = 44.88) .............. 35

Table 15. NWS PSD for all scenarios by day ............................................................................................ 35

Table 16. NWS PSD by phase of flight ..................................................................................................... 36

Table 17. Mean arrival taxi speed and standard deviation by condition.................................................... 37

Table 18. Pairwise comparisons arrival taxi speed by condition (=0.05, HSD=2.35) ............................ 37

Table 19. Mean departure taxi speed and standard deviation by condition ............................................... 37

Table 20. Pairwise comparisons of departure taxi speed by condition (=0.05, HSD=1.05).................... 37

Table 21. Aggregate head up time and standard deviation for PF and PM by condition........................... 38

Table 22. Pairwise comparisons of aggregate head up time by condition ................................................. 39

Table 23. Head up time for PF and PM in high altitude band by condition .............................................. 40

Table 24. Pairwise comparisons of High altitude band head up time by condition................................... 41

Table 25. Head up time for PF and PM in medium altitude band by condition ........................................ 41

Table 26. Pairwise comparisons of medium altitude band head up time by condition.............................. 41

Table 27. Head up time for PF and PM in low altitude band by condition................................................ 42

Table 28. Pairwise comparisons of low altitude band head up time by condition..................................... 42

Table 29. Head up time for PF and PM during arrival taxi operations by condition................................. 43

Table 30. Pairwise comparisons of arrival taxi head up time by condition ............................................... 44

Table 31. Head up time for PF and PM during departure taxi operations by condition ............................ 45

Table 32. Pairwise comparisons of departure taxi head up time by condition........................................... 45

Table 33. Confidence Data Comm message displayed properly on ND.................................................... 65
xi

Table 34. Verification of D-TAXI instruction feasibility .......................................................................... 66

Table 35. How often was D-TAXI instruction verified for correct display on MMD? ............................. 66

Table 36. How long did it take to notice data link message was incorrect? .............................................. 67

Table 37. Integrity, reliability, incompleteness, and ambiguity of Data Comm........................................ 67

Table 38. Scenario Case Number by Display Type and Communication Modality .................................. 78

Table 39. Scenario run order by crew........................................................................................................ 79

Table 40. Data Comm uplink messages (UM) and downlink messages (DM).......................................... 98

Table 41. Data Comm uplink messages by scenario.................................................................................. 99

Table 42. Data Comm response time for all display conditions .............................................................. 169

Table 43. Data Comm response time by Paper display condition ............................................................ 169

Table 44. Data Comm response time by MMD display condition .......................................................... 170

Table 45. Data Comm response time by MMD+Route display condition............................................... 170

Table 46. Workload ratings: Inflight operations during arrivals............................................................. 198

Table 47. Workload ratings: Surface operations during arrivals and departures.................................... 199

Table 48. Workload ratings: PF and PM mean Ranks............................................................................ 199

Table 49. Workload ratings: Binomial test of scale use.......................................................................... 199

Table 50. Workload ratings: Kruskal Wallis difference test for PF and PM .......................................... 200

Table 51. Workload ratings: PF and PM Friedman Ranks difference by condition ............................... 200

Table 52. Workload ratings: PF and PM test statistics ........................................................................... 200

Table 53. Legend for superscript in workload pairwise comparisons tables ........................................... 200

Table 54. Workload ratings: Pairwise comparisons Ranks of inflight operations during arrivals.......... 201

Table 55. Workload ratings: Pairwise comparisons Ranks for surface operations................................. 202

Table 56. Workload ratings: Pairwise comparisons test statistics (a) for inflight operations................. 203

Table 57. Workload ratings: Pairwise comparisons test statistics (a) for surface operations................. 203

Table 58. Workload ratings: By message altitude band during arrivals ................................................. 203

Table 59. Workload ratings: Difference by condition within each altitude band ................................... 203

Table 60. SART ratings: Inflight operations during arrivals .................................................................. 205

Table 61. SART ratings: Surface operations during arrivals and departures.......................................... 205

Table 62. SART ratings: Surface operations during departures only ..................................................... 206

Table 63. SART ratings: Surface operations during arrivals only .......................................................... 206

Table 64. SART ratings: PF and PM difference test............................................................................... 207

Table 65. SART ratings: PF and PM difference by condition ................................................................ 207

Table 66. SART ratings: Pairwise comparisons for inflight operations ................................................. 207

Table 67. SART ratings: Pairwise comparisons for all surface operations............................................. 208

Table 68. SART ratings: Pairwise comparisons for surface departure operations.................................. 209
xii

Table 69. SART ratings: Pairwise comparisons for surface arrival operations ...................................... 210

Table 70. SART ratings: During arrival scenario by message altitude................................................... 210

Table 71. SART ratings: During arrival scenario by message altitude, test on conditions..................... 211

Table 72. SART ratings: Binomial test for PF and PM by condition ..................................................... 211

Table 73. Acceptability ratings: Ownship helpful to understand clearance............................................ 214

Table 74. Acceptability ratings: Route helpful to understand clearance................................................. 214

Table 75. Acceptability ratings: Confidence in route depiction ............................................................. 214

Table 76. Acceptability ratings: Sufficient time to respond to Voice or Data Comm message.............. 215

Table 77. Acceptability ratings: Head down time acceptable for “Expected Taxi” messages ............... 215

Table 78. Acceptability ratings: Head down time for non-time-critical messages................................. 215

Table 79. Acceptability ratings: Overall acceptability of Data Comm................................................... 216

Table 80. Acceptability ratings: Operational risk imposed by communication mode ............................ 216

Table 81. Acceptability ratings: Taxi instructions considered accurate.................................................. 216

Table 82. Acceptability ratings: PF and PM differences ........................................................................ 217

Table 83. Acceptability ratings: PF and PM differences by condition ................................................... 217

Table 84. Acceptability ratings: Pairwise comparisons for sufficient time to respond by condition...... 218

Table 85. Acceptability ratings: Pairwise comparisons for “Expected Taxi” message head down time 218

Table 86. Acceptability ratings: Pairwise comparisons for non-time-critical message head down time 219

Table 87. Acceptability ratings: Pairwise comparisons for overall acceptability of Data Comm .......... 219

Table 88. Acceptability ratings: Pairwise comparisons for operational risk by condition...................... 220

Table 89. Acceptability ratings: Pairwise comparisons for taxi instruction accuracy by condition ....... 221

Table 90. Acceptability ratings: By message altitude band .................................................................... 222

Table 91. Acceptability ratings: Differences by altitude band................................................................ 222

Table 92. Acceptability ratings: Binomial test........................................................................................ 223

Table 93. Workload ratings: Levene's test of equality............................................................................ 224

Table 94. Workload ratings: Tests of between subjects effects.............................................................. 224

Table 95. Workload ratings: By Condition and PF and PM ................................................................... 224

Table 96. Workload ratings: By display condition ................................................................................. 225

Table 97. Workload ratings: By crew position ....................................................................................... 225

Table 98. Workload ratings: Multiple comparisons of display condition............................................... 225

Table 99. SA ratings: Levene's test of equality....................................................................................... 226

Table 100. SA ratings: Test of between subject effects.......................................................................... 226

Table 101. SA ratings: Means by display condition ............................................................................... 226

Table 102. SA ratings: Means by crew position ..................................................................................... 226

Table 103. SA ratings: Means by display condition and by crew position............................................. 227
xiii

Table 104. SA ratings: Mean differences by display condition .............................................................. 227

Table 105. Trust ratings: AHP preference analysis ................................................................................ 229

Table 106. Trust ratings: Weighted responses for consistency............................................................... 230

Table 107. Crew coordination: Levene's test of equality........................................................................ 231

Table 108. Crew coordination: Test for between subject effects............................................................ 231

Table 109. Crew coordination: Mean ratings by display condition ........................................................ 231

Table 110. Crew coordination: Mean rating by crew position................................................................ 231

Table 111. Crew coordination: Mean rating by display condition and crew position ............................ 232

Table 112. Crew coordination: Pairwise comparison by display condition............................................ 232
xiv

Abbreviations

ACARS Aircraft Communications Addressing and Reporting System

ADS-B Automatic Dependent Surveillance - Broadcast

AHP Analytical Hierarchy Process

ANOVA Analysis of Variance

ANSP Air Navigation Service Provider

APU Auxiliary Power Unit

A-SMGCS Advanced-Surface Movement Guidance and Control System

ATC Air Traffic Control

ATIS Automatic Terminal Information Service

ATO Air Traffic Organization

CDTI Cockpit Display of Traffic Information

CDU Control Display Unit

CPDLC Controller Pilot Data Link Communications

CRM Crew Resource Management

D-ATIS Digital ATIS

D-TAXI Data link Taxi instructions

DC Data Communications / Data Comm

DM Downlink Message

EFIS Electronic Flight Instrumentation System

EICAS Engine Indicating and Crew Alerting System

EMMA European Airport Movement Management by A-SMGCS

FAA Federal Aviation Administration

FAF Final Approach Fix

FANS-1/A Future Air Navigation System 1/A

FMS Flight Management System

HITL Human-In-The-Loop

HSD Honestly Significant Difference

IFD Integration Flight Deck

ILS Instrument Landing System

KBOS Boston Logan International Airport

KIAS Knots Indicated Air Speed

LaRC NASA Langley Research Center (Hampton, VA)

MMD Moving Map Display
xv

MSL Mean Sea Level

NASA National Aeronautics and Space Administration

ND Navigation Display

NextGen Next Generation Air Transportation System

NLR National Aerospace Laboratory of the Netherlands

NWS Nose Wheel Steering

OTW Out-The-Window

PF Pilot Flying (Captain in this experiment)

PFD Primary Flight Display

PM Pilot Monitoring (First Officer in this experiment)

PSD Power Spectral Density

RDHFL Research Development and Human Factors Laboratory

SA Situation Awareness

SART Situation Awareness Rating Technique

SID Standard Instrument Departure

STAR Standard Terminal Arrival Route

TCAS Traffic alert and Collision Avoidance System

UM Uplink Message

VMC Visual Meteorological Conditions

WJHTC William J. Hughes Technical Center (Atlantic City, NJ)
1

Abstract

This document describes a collaborative FAA/NASA experiment using 

22 commercial airline pilots to determine the effect of using Data Comm 

to issue messages during busy, terminal area operations. Four 

conditions were defined that span current day to future flight deck 

equipage: Voice communication only, Data Comm only, Data Comm 

with Moving Map Display, and Data Comm with Moving Map displaying 

taxi route. Each condition was used in an arrival and a departure 

scenario at Boston Logan Airport. Of particular interest was the flight 

crew response to D-TAXI, the use of Data Comm by Air Traffic Control 

(ATC) to send taxi instructions. Quantitative data was collected on 

subject reaction time, flight technical error, operational errors, and eye 

tracking information. Questionnaires collected subjective feedback on 

workload, situation awareness, and acceptability to the flight crew for 

using Data Comm in a busy terminal area. Results showed that 95% of 

the Data Comm messages were responded to by the flight crew within 

one minute and 97% of the messages within two minutes. However, post 

experiment debrief comments revealed almost unanimous consensus that 

two minutes was a reasonable expectation for crew response. Flight 

crews reported that Expected D-TAXI messages were useful, and 

employment of these messages acceptable at all altitude bands evaluated 

during arrival scenarios. Results also indicate that the use of Data 

Comm for all evaluated message types in the terminal area was

acceptable during surface operations, and during arrivals at any altitude 

above the Final Approach Fix, in terms of response time, workload, 

situation awareness, and flight technical performance. The flight crew 

reported the use of Data Comm as implemented in this experiment as 

unacceptable in two instances: in clearances to cross an active runway, 

and D-TAXI messages between the Final Approach Fix and 80 knots 

during landing roll. Critical cockpit tasks and the urgency of out-thewindow scan made the additional head down time to respond to Data 

Comm messages undesirable during these events. However, most crews 

also stated that Data Comm messages without an accompanying audio 

chime and no expectation of an immediate response could be acceptable 

even during these events.
2

1 Introduction

In the fall of 2008, the FAA Air Traffic Organization (ATO) Operations Planning, Air Traffic Systems 

Concept Development and Validation Group, prepared a document outlining research needs for 

implementing data communications (Data Comm) in the Next Generation Air Transportation System 

(NextGen) as it related to the flight crew in the aircraft. In particular, NASA Langley Research Center 

was to provide an analysis of the impact caused by Data Comm on the flight crew in a human-in-the-loop 

(HITL) simulation that aligned with the FAA William J. Hughes Technical Center‟s (WJHTC) Research 

Development and Human Factors Laboratory (RDHFL) simulation studying the impact of Data Comm on 

controllers. The FAA referred to this Langley experiment as the FAA/NASA Data Comm Airside Humanin-the-Loop Simulation. An excerpt from Paragraph 3 of the research request document states:

“The purpose of the airside research is to study Data Comm functionality and to determine 

how it can contribute to the ultimate NextGen goal of increased flight deck efficiency and 

capacity. To ensure that Data Comm successfully provides a digitally automated data 

communication system to support NextGen, the ATO Air Traffic Systems and Validation 

Group developed a Research Management Plan for Segment Two (FAA, 2008) that outlines a 

series of research initiatives and studies. These research initiatives and studies include 

cognitive walkthroughs, information flow models, part-task research studies, and high-fidelity 

HITL simulations of future operational concepts. The research efforts are pursued to help 

validate proposed Data Comm concepts and identify requirements that will be the basis for 

constructing the future air (and ground) Data Comm systems.” [1]

The request was codified in the FAA/NASA Interagency Agreement IA1-973, Technical Direction 1, 

and a NASA Langley Research Center document was submitted to the FAA Data Comm Program as the 

Final Report specified in Paragraph 1.1.10 of that document. [2] This document was titled “NASA/FAA 

Data Comm Airside Human-in-the-Loop Simulation,” and delivered on 28 July 2010. Key details and 

assumptions contained within that agreement and an Addendum are described in Appendix A. Additional 

requests for data analysis by the FAA after the experiment began have been accommodated in this report 

(e.g., message response time by message type, reformat of results for Special Committee 214 (SC-214), 

etc.).

The primary objective of the experiment was to determine the acceptability of Data Comm to the flight

crew during high traffic density operations in a complex terminal area with an operational environment 

appropriate to the FAA‟s Segment 2 timeframe (2017-2022). Of particular interest to the FAA was DTAXI, or the use of Data Comm messages to send taxi routes to the flight crew. Acceptability was

assessed in the context of expected, actual, and amended D-TAXI clearances during surface operations

and while on approach, as well as other Data Comm messages (frequency change, altimeter setting, etc) 

throughout the scenarios. Three types of D-TAXI messages sent via Data Comm were used in this HITL 

experiment:

1. Expected D-TAXI: informative; for flight crew planning only (not used in today‟s operation)

2. D-TAXI: directive; taxi route assigned by ATC to flight crew (unlike today‟s operation, it 

does not include clearance to begin moving the aircraft)

3. Amended D-TAXI: directive; change to existing taxi route by ATC (the same as changing the 

taxi route in today‟s operation)

Specific planning regarding the objective, scope, experimental design, scenario definitions, and 

assumptions for the experiment was based on the requirements of that IA1-973 agreement, refined by the 
3

literature search, and subsequent interagency communication. [2] Joint Planning and Development Office 

and FAA documents were used to define expected operations and Data Comm capabilities for that 

timeframe. [3][4][5] A paper by the FAA ATO Data Comm Human Factors Working Group (HFWG-08) 

identified that guidance must be defined on when not to send messages so crew distractions during critical 

phases of flight are minimized. [6] The paper further listed a range of research needs to be conducted, 

that included identifying the impact of mixed modes of communication (using both Voice and Data 

Comm) on controllers and pilots, what is the acceptable delay in responding to Data Comm messages, 

and what is the impact of aural cues. Further, a white paper from the FAA Human Factors Research and 

Engineering Group (AJP-61) identified specific research needs for Data Comm in the Segment 2 time 

frame (2017-2022). In particular, it asked what Data Comm procedures should be, and are there cases 

and places in which Data Comm use should be avoided. [7]
4

2 Data Comm Literature Review

There has been considerable research conducted in the United States and Europe regarding the use of 

Data Comm messages between pilots and controllers, the impact it has on the flight crew‟s workload and 

scan pattern, when it should or should not be used, and what characteristics are needed for it to be 

considered acceptable by the crew. This section outlines Data Comm findings by topic (with the relevant 

studies mentioned in the appropriate paragraphs), then describes the impact of the literature review on the 

experiment design in the final paragraph.

2.1 When to use Voice or Data Comm

The LINK2000+ Flight Crew Datalink Operational Guide specifies how Controller Pilot Data Link 

Communication (CPDLC) will be used for routine, non time-critical instructions and requests while in the 

European en-route environment. It also defined the response time required between ground and airborne 

equipment, as well as between controller and flight crew. These guidelines were based on a human-inthe-loop simulation using controllers and pilots in European enroute airspace. Operational review of the 

experiment led to the requirement that Voice be used for all time-critical and safety-related 

communication. Other findings include delay in communication response, lack of flexibility in 

composing Data Comm messages, and loss of situation awareness (SA) when not using party line 

communication (such as Voice). [8][9] 

Over 900 revenue flights at the Brussels airport from August 2006 through February 2007 participated 

in D-TAXI operational trials exploring procedures to improve productivity and safety while using nontime-critical messages for a medium to high taxi path complexity. Push-back, start-up, and taxi CPDLC 

messages were sent by ATC and responded to by the crew using CPDLC, however for operational and 

safety reasons, the crews also responded using Voice communication. Overall, pilot acceptance based on 

debrief comments was high and continued to increase as the experiment continued. Open issues from this 

research include: many of the crews did not respond to survey questions, existing equipage and 

procedures were used which were not optimized for a high-workload terminal area operations, and it was 

not clear what impact Data Comm caused to head up time or workload. [10]

2.2 Prioritization of Voice and Data Comm

Researchers from 24 multi-national partners (including air navigation service providers, airport 

operators, airlines, airframe, avionics, pilots, controllers, and research agencies) conducted a test called 

EMMA in 2004-2006, and EMMA2 in 2006-2009. EMMA (European Airport Movement Management 

by A-SMGCS) consolidated surveillance and conflict alert function for the controller, and EMMA2 

focused on advanced onboard guidance support to pilots and planning support for controllers (A-SMGCS 

is the Advanced Surface Movement Guidance and Control System conducted in Europe). Multiple 

simulation platforms and operational test locations were used to explore the holistic, integrated air-ground 

system. For the taxi tests in Prague, Milan and Toulouse, the flight crew had a moving map display as 

well as surface alerts for other traffic and runway incursions and CPDLC was used to transmit taxi 

instructions. Requirements and safety analysis conducted by the consortium resulted in the requirement 

that Voice communication always took precedence over Data Comm, and was reported as a key result. 

EMMA results indicated taxi time and Voice communication were reduced, while EMMA2 concluded 

that CPDLC for taxi operations under these conditions was technically and operationally feasible. No 

oculometer or other measure of head up time data were collected to independently and quantitatively 

measure these effects (and implicitly, quantify pilot workload and situation awareness). 

[11][12][13][14][15] (NOTE: this influenced the decision to include oculometers.)
5

A 2008 report about the operational use of Data Comm in Maastricht airspace states the use of 

CPDLC is continuing to grow at a steady pace, with controllers initiating communication (uplink) more 

than 70 times for every time pilots initiate communication (downlink). The messages are for routine, 

strategic situations and supplement Voice commands, and Voice instruction take precedence over Data 

Comm. Although the primary response to a message should be in the same mode it was received in, 

Voice will be used to resolve complex, safety, and time-critical issues, or resolve any confusion between 

the controller and pilot. [16]

2. 3 Use of Data Comm Reduces Need for Voice Communication

NASA flight tests conducted at Denver‟s Stapleton Airport in the late 1980s using 9 pilots flying a 

total of 54 scenarios (each scenario a 60 nautical mile long arrival procedure) concluded that Data Comm 

greatly reduced voice congestion and had lower pilot workload. [17] 

Research by Wright State and the FAA‟s William J Hughes Technical Center used eight pilots to 

explore the effects of CPDLC messages on controllers and pilots. While controller-pilot communication 

was decreased when using Data Comm, the amount of inter-crew communication was increased. The 

report further postulated that the increased discussion between pilots that occurs when using Data Comm 

communication may improve problem-solving and decision making within the cockpit. [18]

A conclusion from the LINK2000+ Real-Time Simulation Project was the benefit of a reduction in 

voice congestion. [9]

Another finding from the EMMA2 operational taxi trials in Europe found using CPDLC messages 

during taxi operations reduced the use of Voice communication by both controllers and flight crew. 

[13][14]

A report by Eurocontrol states the operational use of Data Comm within the Maastricht airspace has 

contributed to an increase in the safety of flight operations, as well as a reduction in controller-pilot voice 

communication congestion. [16]

2.4 Data Comm Acceptability

One of the early flight tests to explore the issue of Data Comm and flight crew interaction occurred in 

1991. NASA Langley used a Boeing 737 and seven crews flying in both enroute and terminal area 

environments, with scenarios using either Voice or Data Comm as the primary controller to pilot link. 

The flight test showed a reduction in workload and greater pilot acceptability when the ability to “autoload” the ATC instruction into the Flight Management System (FMS) was available. [19][20]

The National Aerospace Laboratory of the Netherlands (NLR) conducted research involving nine 

crews flying six gate-to-gate scenarios between London and Amsterdam. They identified that the ability 

to “auto-load” the Data Comm text message into the FMS substantially improved the crews rating of 

whether Data Comm was acceptable as a form of communication. Improvements in location (into the 

forward field of view on the center console) and the ability to “auto-load” information raised the 

acceptability rating from 56% to 94%. This research also concluded that the Control Display Unit (CDU)

was the optimum Data Comm interface. [21]

Research from a Human Factors study at the FAA William J Hughes Technical Center (FAA WJHTC) 

recommended that Data Comm reception and interface devices be in the forward field of view, and that a 
6

distinct aural alert should be used to indicate the presence of Air Traffic Control (ATC) messages. [22]

NASA research titled “The Human Factors of FMS Usage in the Terminal Area” had ten two-person 

crews fly a Boeing 747-400 simulator into the Dallas Ft-Worth terminal area, using manual, auto-pilot, or 

FMS coupled to auto-pilot operations. The results concluded that while use of the FMS is acceptable in 

the terminal area, the use of the FMS resulted in the highest workload and lowest pilot satisfaction as 

reported by the pilots. [23]

2.5 Understanding Communication

The D-TAXI operational trials at Brussels from August 2006 through February 2007 used push-back, 

start-up, and taxi CPDLC messages sent from ATC, while the crew responded via CPDLC and Voice. 

Results suggest the flight crew found that the messages were easy to understand and there were no 

incidents or errors. [10]

The general conclusion from the LINK2000+ Real-Time Simulation Project was that all controllers 

found Data Comm acceptable, easy to use, and assisted in increasing safety. They also stated it was 

beneficial to have Data Comm available as a second communication channel for routine messages. [9]

NASA flight tests conducted at Denver‟s Stapleton Airport concluded that Data Comm was more 

accurate than Voice and lowered pilot workload. Cockpit equipage included the ability to automatically 

load the ATC instruction from the CDU into the FMS. [17]

The 1991 flight test by NASA Langley with a Boeing 737 and seven crews showed a reduction in 

confusion, errors, and need for message repetition when the ability to “auto-load” the ATC instruction 

into the FMS was available. [19][20]

A 2009 FAA study interviewed 48 pilots from various US airlines, and concluded Voice

communication from non-native English speakers presents challenges to controllers and pilots on the 

receiving end of that transmission. In order to understand these challenges, a range of issues were 

identified to include pronunciation, syllable parsing, rate and timing of speech, and differences between 

ICAO and standard US phraseology. The study postulates employing Data Comm should significantly 

alleviate many of these problems. [24]

A NASA simulation study called “Integrating Datalink and Cockpit Display Technologies into Current 

and Future Taxi Operations” was conducted in 2002. Messages sent via Data Comm were found to 

reduce time spent writing clearances and improved the crews‟ ability to understand the message on the 

first attempt. [25]

A single pilot, general aviation study examined the effectiveness of three different Data Comm

interfaces, involving voice, visual, and redundant presentation of the ATC information. Oculometers 

were also used to measure pilot scan patterns and dwell time. Eighteen pilots flew multiple scenarios and 

responded to several ATC instructions while scanning outside for traffic. Results revealed that the visual 

display of ATC instructions (Data Comm) provided the greatest accuracy of communications read back, 

was less disruptive, and resulted in the least flight technical error (deviation from flight path). The 

auditory-only condition was the most disruptive of the conditions, with the redundant display condition 

providing many of the same benefits as visual-only, but never better than visual only. [26]
7

2.6 Flight Crew Response Time

The LINK2000+ Real-Time Simulation Project found that the response of the flight crew in 

responding to controllers was delayed when using CPDLC as compared to Voice communications. This 

study, conducted in European enroute airspace, did not include cockpit displays to assist in understanding 

text clearances; nevertheless, the overwhelming majority (> 95%) of the flight crew responses occurred 

within 60 seconds. [9]

The D-TAXI revenue flights at Brussels using CPDLC for push-back and taxi operations reported high 

pilot acceptance; response time was longer although it was stated that it was not operationally significant. 

[10]

The NASA “Integrating Datalink and Cockpit Display” simulation examined the impact on flight crew 

of using Voice or Data Comm in three different modes: Voice only, Data Comm with Voice, and Data 

Comm without Voice. Flight crews took the longest to respond to communications and instructions in the 

Data Comm without Voice mode. However, the benefit of Data Comm may extend to increased 

operational efficiency, increased communication efficiency, and reduced radio congestion. [25] 

Another NASA experiment evaluated flight deck procedures for Data Comm trajectory negotiations 

during cruise flight, and measured flight crew response time to the uplink messages as well as workload 

and acceptability. Results indicated workload did not have a significant impact on response times, 

response times were generally well within two minutes, and the procedures were deemed feasible. [27]

2.7 Flight Crew Head Up Time

The D-TAXI revenue flights at Brussels using CPDLC for push-back and taxi operations reported high 

pilot acceptance; however, head up time was decreased although it was stated that it was not an 

operationally significant factor. This result is based on pilot self-assessment during debrief, no 

independent measure was used. [10] 

Research sponsored by the FAA and conducted by NLR in the mid 1990s used 18 American and 

European crews flying a simulator into Schiphol Airport, and determined that Data Comm uplink 

messages decreased the head up time of both crew members. Another finding was “... the fact that 

uplinks had an effect on the scanning behavior of the crew member not responsible for the 

communication task.” The research stated the Pilot Flying (PF) had less head up time when Data Comm 

was being used due to interest in the message being received. [27]

2.8 Cockpit Graphical Displays

Research using 18 flight crews in a flight simulator by the FAA and NLR also reported the addition of

a Traffic alert and Collision Avoidance System(TCAS) or Cockpit Display of Traffic Information (CDTI)

would help maintain the awareness of the crew and offset the loss of Voice “party-line” information. [27]

The LINK2000+ Real-Time Simulation Project used controllers and pilots in European enroute 

airspace, and consisted of CPDLC Uplink and Downlink messages, to include heading and altitude 

changes, and frequency changes to the next controller. One issue identified was the flight crew‟s 

perceived loss of SA from the lack of party line communication when using Data Comm. However it 

should be noted that there were no cockpit displays to assess possible mitigations. [9]
8

The EMMA2 operational taxi trials in Europe found using CPDLC messages with cockpit displays 

while taxiing on the airport surface improved the flight crew‟s SA and their workload was maintained. 

[13][14]

A study by NASA called “The Effects of Advanced Navigation Aid and Different ATC Environments 

on Task-Management and Communication in Low Visibility Landing and Taxi” showed that an electronic 

moving map significantly enhanced SA when using Data Comm, and reduced both intra-cockpit and 

controller-pilot Voice communications. [29]

A study reviewed multiple research efforts in 1999 and identified the following problems with Voice

communication: data are transmitted sequentially, background noise and dialect, congestion, long or 

complex messages are prone to being misunderstood. The same study identified the following benefits of 

Data Comm communication: higher efficiency, unloads memory, improves message delivery time, 

improves transfer of information to other ATC and flight deck systems. However, the challenges Data 

Comm presents include: reduced SA due to loss of “party-line” Voice communication, inability to multitask while responding to Data Comm, decreased head up time, and that cockpit graphical displays appear 

to improve head up time when Data Comm is used. [30]

2.9 Simultaneous Use of Voice and Data Comm

The controllers in the LINK2000+ Real-Time Simulation Project reported “[i]t was difficult to mix the 

two ways of giving instructions (Voice and Data Comm)”. [9]

Results from the pilot debrief during the D-TAXI operational trials at Brussels from August 2006 

through February 2007 found the requirement for the flight crew to respond with both Data Comm and 

Voice was considered impractical. [10]

A NASA simulation in 2003 was conducted that compared how flight crews handled Voice and Data 

Comm messages in a single medium versus a mixed medium. The interval between messages was also 

varied to examine the influence of time pressure. Results indicated that for messages sent via Voice, 

transaction times were lengthened in the mixed media environment. Furthermore, when time pressure 

was introduced, the mix of Voice and Data Comm did not necessarily capitalize on the advantages of both 

media. [31]

A NASA simulation experiment using twenty-four experienced commercial pilots explored various 

communication modes to understand the impact on decision making, workload, and SA. “The Evaluation 

of Mixed Mode Data Link for NextGen” experiment used Voice redundant to Data Comm (ATC and pilot 

always use both), Voice supplement to Data Comm (pilot always uses both), Data Comm only, and Data 

Comm with display showing aircraft intent. This research indicated that Data Comm alone was not 

always the optimal solution. When pilots read back the Data Comm message over Voice, the pilots 

committed fewer errors and their SA was increased. This research looked at the pilot‟s performance, and 

did not examine the entire operational interaction with controllers. [32]
9

2.10 Experiment Design Decisions from Literature Review

The following experiment design decisions were made, driven by the literature review documented in 

the previous paragraphs of this section:

1. Data Comm will be used for normal communication (taxi clearances, altimeter settings, frequency 

change, etc.), and Voice for time-critical, safety-related, or non-normal situations (takeoff 

clearance, landing clearance, crossing an active runway, etc.).

2. Some events will occur during the experiment that have ATC simultaneously issuing both Voice

and Data Comm instructions, and the flight crew will respond to questions about this event.

3. Voice communication will have priority over Data Comm to ensure there is no ambiguity 

between the two communication modes.

4. The CDU will be the flight crew‟s interface for the Data Comm system.

5. The flight crew will be able to „auto-load‟ the Data Comm clearance into the FMS and display 

that route on the Multi-Function Display (MFD).

6. Data will be collected on flight crew interaction with Data Comm in terms of time to respond, 

workload, acceptability, and understandability.

7. Data Comm messages coupled to graphical displays for the flight crew will be an Independent 

Variable.

8. Loss of situation awareness due to use of Data Comm will be measured, to include the impact of 

graphical displays coupled to the Data Comm message.

9. Oculometers will be used to collect head up time for both the PF and the Pilot Monitoring (PM)

to create a more complete understanding of flight crew interaction with Data Comm. 

Independent oculometer systems will be used to accurately capture different cockpit tasks of the 

PF and PM.
10

3 Methodology

3.1 Experiment Hypotheses

The following high-level hypotheses drove selection of variables:

H1: Pilot workload and situation awareness will differ significantly between Voice and Data 

Comm communication modes.

– This hypothesis drove evaluation of the effect of Data Comm communications modality 

employment on flight crew workload and SA during taxi-in and taxi-out operations.

H2: Pilot workload and situation awareness will differ significantly between display modes 

when using Data Comm.

– This hypothesis drove evaluation of the influence of graphical display of airport and 

ownship route on crew workload and SA in a Data Comm environment.

H3: Pilots will rate the Data Comm used within this experiment as operationally acceptable.

– This hypothesis drove determination of the acceptability of Data Comm communications in 

the flight deck during operations in the terminal area. Acceptability was assessed in the 

context of expected, actual, and amended D-TAXI clearances during surface operations, 

and expected taxi clearances and other strategic CPDLC messages while on approach.

In addition to addressing the high-level hypotheses, the design of the study also permitted examining 

the following (specific metrics listed in Section 3.5):

 Message response times by type of Data Comm message

 Vehicle performance indices, such as Nose Wheel Steering (NWS) and taxi speed

 Workload and situation awareness of both PF and PM

 Acceptability of Data Comm messages at "High", "Medium", and "Low" altitude bands 

during arrivals

 Assessments of head up time for each crew member across the experimental conditions

 Objective data and subjective responses broken down by inflight and surface segments

 Objective data and subjective responses broken down by arrival and departure scenarios

3.2 Independent Variables

The literature review identified several key issues associated with Data Comm that could form 

independent variables. From these issues, two were selected by the FAA and NASA Team for inclusion 

in this study: Communication Modality (Voice, Data Comm) and Map Display Methodology (Paper, 

Moving Map Display (MMD), MMD+Route).
11

Thus, two component studies were chosen to efficiently and effectively incorporate both variables 

within time and resource constraints. The first of these (Study 1, or S1) assesses the differences in pilot 

acceptability of communications using two different modalities (Voice and Data Comm), and the second 

(S2) investigates the effect of map display methodology on the acceptability of Data Comm. The 

combination of communication modality and display methodology defines the four experimental 

conditions shown in Table 1 in Section 3.4.

3.2.1 Communication Modality

Communication modality addresses how information is transmitted from controllers to the flight crew. 

The use of Voice by exception was consistent with the Data Comm Tower Human-in-the-Loop (HITL) 

Simulation at the FAA Research Development & Human Factors Laboratory (RDHFL).

Two options were selected:

a. Voice only for controllers and flight crew. This condition serves as the baseline condition 

representing present-day operations.

b. Data Comm for controllers and flight crew, with Voice used by exception for time-critical or 

safety-related information. 

(1) Data Comm was used to issue:

 taxi, expected taxi, and amended taxi instructions, 

 gate pushback time,

 engine start clearance,

 notification of new altimeter or new Automatic Terminal Information Service (ATIS) 

information,

 radio frequency change to the next air traffic controller.

 

(2) Voice transmissions were used on departures to:

 initiate aircraft taxi (Ground), 

 hold short of an active runway (Ground),

 cross an active runway (Tower),

 provide clearance to position and hold on the takeoff runway (Tower).

 

(3) Voice transmissions were used on arrivals for:

 traffic call-outs during arrival (Approach),

 initial check-in on tower frequency (Tower),

 clearance to land (Tower), 

 initial check-in and clearance to taxi (Ground).

3.2.2 Display Methodology

Display methodology addresses the depiction of airport layout and taxi route with respect to ownship 

position on the Navigation Display (ND) in the Surface Depiction Mode. Three options were selected:

a. Paper where the flight crew had only a paper copy of the airport diagram.

b. MMD where taxiways, runways, signage, and ownship position was shown on a Moving 
12

Map Display.

c. MMD+Route included everything in Option b., as well as a graphical display of the 

expected and actual ownship route clearance.

3.3 Scenario Descriptions

The scenarios were arrival and departure operations at Boston Logan International airfield, and 

utilized a combination of current published instrument procedures and clearances given by controllers. 

Furthermore, the taxi operations were aligned with related research being conducted by the FAA at the 

WJHTC RDHFL. A complete list of the scenarios, run order by crew, altitude that Data Comm messages 

were sent, taxi routes, and arrival procedure (if appropriate) are described in Appendix B.

3.3.1 Airport

The Boston Logan International airport (KBOS) (Figure 1) was used to align this research of the 

impact of Data Comm to flight crews, with FAA research studying the impact of Data Comm to 

controllers.

Figure 1. Boston Logan International airport diagram
13

3.3.2 Arrival Routes and Instrument Approaches

Arrivals to Runway 27 and Runway 33L were created that provided realistic profiles and workload 

from 18,000 feet to landing. An overview of the routes (fixes associated with the NORWICH THREE 

and SCUPP FOUR Arrivals) is presented in Figure 2.

Figure 2. KBOS airspace and arrival routes

More specifically, portions of existing Standard Terminal Arrival Routes (STAR) were connected to a 

particular Instrument Landing System (ILS) approach based on current controller procedures. The routes 

were:

 NORWICH THREE Arrival, KENNEDY Transition (Figure 3).

o This procedure starts Southwest of the airport. The scenario itself started overhead 

Norwich and proceeded East to INNDY, then direct to the Initial Approach Fix 

(BENNN) for the ILS to Runway 33L. 

 SCUPP FOUR Arrival, KENNEDY Transition (Figure 4).

o This procedure starts East of the airport. The scenario itself started overhead 

ARMUN and proceeded West to SCUPP, then a clearance for the ILS to Runway 27.

SCUPP FOUR

NORWICH THREE
14

Figure 3. Excerpt of NORWICH THREE arrival

Figure 4. Excerpt of SCUPP FOUR arrival

3.3.3 Taxi Routes

Arrival taxi routes from Runway 27 ended at Terminal B, and arrival taxi routes from Runway 33L 

terminated at Terminal E (listed in Appendix B, Section 2). To align with research conducted by the FAA 

at the WJHTC RDHFL, departure taxi routes were selected from Terminal E-8A at the Northwest corner 

of Terminal E to the departure end of Runways 27 and 33L (listed in Appendix B, Section 3).
15

3.4 Experiment Design

Table 1 presents the experimental design matrix. Each of the four populated cells in the matrix had an 

associated arrival and departure scenario, creating the eight scenarios indicated in Table 2. Unpopulated 

cells (indicated by shading) were removed from the experiment since they were not essential in 

addressing the FAA questions, and to accommodate time and funding limitations. The column “Data 

Comm” refers to a communications modality where the primary mode is Data Comm; however, Voice

was used for initiation of aircraft movement, aircraft check-in on Tower and Ground frequencies, runway 

crossing, and position and hold clearances.

The populated cell for “Paper” and “Voice” was the baseline case in terms of workload and situation 

awareness, representing typical airline transport operations in a present-day (2010) environment. The 

crew saw approximately 15 static aircraft and 20 moving aircraft (with the appropriate Voice

communication between controller and pilot) during each 15 to 20 minute scenario, approximating a busy 

78 aircraft arrival day at KBOS. Thus, incorporation of expected or amended taxi clearances in a present 

day Voice communications environment, while not unheard of, represents atypical operations and 

therefore was not implemented in this experiment design.

During Data Comm scenarios, the experiment was specifically designed to present a worst-case 

operational scenario. Four Data Comm messages were sent within two minutes of each other (2 

“Expected Taxi” messages, 1 Altimeter, 1 change to the ATIS) while the crew was intentionally 

distracted. These distractions included ATC Voice call-out of factor traffic during arrival scenarios and 

researchers providing updated aircraft weight numbers during taxi that required the PM to use the FMS to 

recalculate takeoff speeds during departure scenarios. The second study held constant the modality of 

messages (Data Comm) but varied the display methodology over three display conditions: paper, moving 

map, and moving map with the graphical presentation of the ownship taxi route on the ND.

Table 1. Experimental design matrix

Voice Data Comm

Paper Baseline 1 (S1/S2) Baseline 2 (S1/S2)

MMD S2

MMD+Route S2

Baseline 1: Pilot performance with present-day Voice communications and paper airport diagram

Baseline 2: Pilot performance with Data Comm and paper airport diagram

Study 1 (S1): Assess and compare the acceptability of two communications modalities (Voice and 

Data Comm) while using paper airport diagrams.

Study 2 (S2): Assess and compare the effect of Map Display Methodology while using data 

communications on the acceptability of Data Comm.

Table 2 below contains the scenario types used for each study, and defines the flight phase, 

communication mode, and display methodology. 
16

Table 2. Scenario types

Type Flight Phase Condition Study S1 

(Comm)

Study S2 

Comm Mode Graphical Display (Display)

0 Arrival Voice Paper Baseline

1 Arrival Data Comm Paper X Baseline

2 Arrival Data Comm MMD X

3 Arrival Data Comm MMD+Route X

4 Departure Voice Paper Baseline

5 Departure Data Comm Paper X Baseline

6 Departure Data Comm MMD X

7 Departure Data Comm MMD+Route X

These eight scenario types were replicated so that each crew was exposed to 16 runs. Runs were 

ordered so that modality/display methodology conditions were not repeated within 3 runs, with arrivals 

and departures alternating as much as possible. Run conditions for the first eight runs were replicated 

exactly for the second set of eight runs for all crews. Different run orders were assigned to different 

crews to counterbalance the serial position of scenarios over the course of the experiment. In addition to 

these 16 runs, a rare event scenario run was conducted as the final run, unbeknownst to the crew. This 

last run supported an exploratory study described in Section 6.

3.5 Dependent Variables

The Dependent Variables for this experiment were:

 Workload 

 Situation awareness (SA)

 Acceptability

Metrics used to quantify the dependent variables included the following:

 Data Comm message response times

 Flight crew technical performance: NWS control rate, taxi speed, flight director error

 Workload: Bedford Workload Scale, pairwise comparisons of workload by display type

 Situation Awareness: Situation Awareness Rating Technique (SART), pairwise comparisons 

of SA by display type

 Acceptability: subjective, self-rating

 Crew Resource Management: flight crew errors, or mitigation of potential errors

 Head up time: both PF and PM

 Trust: model-based errors and response-time metrics, questionnaire items derived from 

previous research, and open format interviews

3.6 Parameters and Data Analysis Techniques

3.6.1 Data Comm Message Response Time

Data Comm message response times were calculated as the difference in seconds from the time that 
17

the message was initially received (chime annunciated, and “ATC MESSAGE” shown on upper Engine 

Indicating and Crew Alerting System display), and the time that a response button (“WILCO”, 

“ROGER”, or “UNABLE”) was depressed on the message ATC Uplink Page 2. Message response times 

were averaged for each crew, and for all crews, by modality and message type. Response times were also 

analyzed by distribution, and in categories requested by the FAA Data Comm group.

3.6.2 Flight Technical Performance

For both arrivals and departures, NWS rate in degrees per second was analyzed using the Power 

Spectral Density calculated for the frequencies of 0.1 to 2.0 Hz(i.e., over a frequency range where 

significant NWS corrections would be made). For arrivals, the calculations began when the aircraft taxi 

speed was first below 80 knots. For departures, the calculations began when the taxi speed was first 

above 0.5 knots.

Average taxi speeds were also calculated for arrivals - from when taxi speed was first below 30 knots 

through the end of run - and for departures, from when taxi speed was first above 0.5 knots through the 

end of run.

Since a precise path was not defined during periods where any Data Comm messages were being 

handled (messages were given prior to the Final Approach Segment during arrivals, and pilots are not 

required to precisely follow yellow taxi lines during surface operations), the PF flight technical 

performance was analyzed, rather than flight technical error. Flight technical performance during arrivals 

was defined as the average flight director deviation from null, determined over one of the three altitude 

bands where Data Comm messages were received. The specific altitude bands were „High‟ (16,000 -

14,000‟ MSL), „Medium‟ (10,000 - 8,000‟ MSL), and „Low‟ (7,000 - 5,000‟ MSL), and were chosen to 

represent various states of crew workload. Crew workload was not considered significantly different 

between these altitudes.

3.6.3 Head Tracking

Head tracking data were analyzed to determine each pilot‟s head up time over each test run. Pilot head 

up time was determined using a combined measure approach using both eye gaze and head position. If 

the eye gaze vector was present, head up was counted if the point of gaze was located out the window. If 

eye gaze was not available, determined by an eye gaze quality of less than 50%, head position was used to 

assess if the pilot was head up using a head pitch threshold specific to that subject on that run. To 

calculate the head pitch threshold, average head pitch was calculated when point of gaze was out the 

window. This approach was taken to maximize the number of data points usable for analysis due to head 

tracking being more stable across subjects than eye tracking. 

Head tracking analysis for each pilot was broken up into several phases. For all conditions, head up

time was calculated for four bands on arrival scenarios, aligning with the three Flight Technical 

Performance altitude bands (High, Medium, Low) and approach taxi (< 80 knots to end of scenario). 

Departure scenarios were analyzed from beginning of taxi (> 0.5 knots) to the end of the scenario. 

Statistical analyses were performed to identify significant difference across modality and between 

crew role (PF/PM), as well as the interaction of the modality and crew role. Used in conjunction with an 

Analysis of Variance (ANOVA), Tukey Honestly Significant Difference (HSD) pairwise comparison tests 

were performed to determine significant differences between multiple comparisons of modalities. These 

tests compare all possible pairs of statistical means of the individual modalities against the standard error 
18

of the data distribution, determining if the difference between means is significantly different from the 

general observations in the tested dataset. 

3.6.4 Biographical Data Questionnaire

The Biographical Data Questionnaire (Appendix C) was acquired detailed information about each 

pilot‟s experience. Questions focused on age, overall flight time in a cross section of aircraft, flight time 

in Boeing 757 or comparable aircraft, military time, experience with Data Comm messaging, and any 

flight experience flying into and out of KBOS. Results are shown in Section 3.10.2, Subject Pilot 

Experience Level.

3.6.5 Post-Scenario Questionnaire

The Post-Scenario Questionnaire (Appendix D) was given to both subject pilots after each scenario, 

and consisted of the Bedford Workload Scale [34], the Situation Awareness Rating Technique (SART)

[35], and questions addressing crew coordination, acceptability and trust. This questionnaire was given

electronically on a personal tablet computer while the subject was seated in the simulator. The Bedford 

Workload Scale is a uni-dimensional rating scale designed to identify operator's spare mental capacity 

while completing a task. The single dimension is assessed using a hierarchical decision tree (always 

completely visible to the subject) that guides the operator through a ten-point rating scale, each point of 

which is accompanied by a descriptor of the associated level of workload. It is simple, quick and easy to 

apply in situ to assess task load in high workload environments, but it does not have a diagnostic 

capability. 

A SART was also administered after each run. SART provides an assessment of the SA based on a 

pilot‟s subjective opinion. SART incorporates three dominant components: demand on the pilot‟s 

resources, supply of resources, and understanding of the situation. These were determined to be relevant 

to SA through an analysis with pilots. Pilots rated their perception of the impact of these components 

using bipolar scales from 1 to 7. These scales were then transformed using the formula:

SA = Understanding – (Demand – Supply)

to provide an overall SART score for a given system. The range of scores from the application of the 

formula is from -5 for extremely low SA to 13, extremely high SA.

Additional questions were also given on the Post-Scenario Questionnaire pertaining to where the crew 

received their Data Comm information, crew interaction, acceptability of receiving Data Comm 

messages, and their trust in the system. The trust questions were derived from previous research in which 

the issues of confidence, risk, accuracy, verification need, and time constraints were investigated. These 

elements of trust were found to be valid in several research efforts in which subjects were asked to 

identify concepts that they affiliated with the construct of trust. In addition, other research from which 

questions were drawn focused on the operators‟ perception of risk associated with too much trust in 

automation.

Six questions were developed to assess the Crew Resource Management (CRM), or interaction and 

coordination of crew members on the flight deck. Questions were formulated based upon FAA Advisory 

Circular 120-51E, Crew Resource Management Training, particularly the crew performance marker 

clusters. The questions ask pilots to assess themselves individually in terms of their performance during 

the scenario, their perception of their crewmembers performance, the level of communication and related 
19

SA throughout the scenario, and their subjective analysis of crew role responsibility adherence.

Levene‟s test for equal variances on six of the thirteen post-scenario questionnaire scales failed the 

assumption of equal variances (p≥0.10); and of those that did not fail Levene‟s test, all but two showed 

distribution distortions where either skew or kurtosis exceeded +/-2. Consequently, analyses for these 

items were conducted using non-parametric statistics. When necessary, results were analyzed separately 

to determine if display conditions differed for pilots, and then for copilots; and a separate analysis was 

performed to determine if, aggregated over display conditions, pilot and copilot ratings significantly 

differed.

3.6.6 Post-Experiment Questionnaire

The Post-Experiment Questionnaire (Appendix E) compared workload and SA between various 

scenarios and asked specific questions regarding the acceptability of using Data Comm at various 

altitudes. Additional questions were asked regarding crew coordination, the overall assessment of the 

experiment, the use of Data Comm, and suggestion for improvements to the messages or displays.

Pilot crews were asked at the completion of the experiment to compare their perceived support for 

effective CRM and crew coordination experienced among scenarios. Each pilot of the crew assessed the 

SA difference experienced through using one modality versus another, indicating which modality had the 

greatest effect on their ability to effectively coordinate as a team, distribute their attentional resources, 

and ensure shared SA. Responses to this qualitative questionnaire also provide insight into the interaction 

effect of crew role and modality, as well as basic pilot modality preference.

These data were analyzed according to the Analytical Hierarchy Process (AHP), which provides 

percentile preferences for options considered, as well as a consistency ratio of subject preferences. The 

preference percentiles averaged over participants in each crew role. Arcsin (square root) transformed 

percentile data was analyzed for equal variances among conditions. Levene tests for all dependent 

measures were non-significant (p>0.01), and all skew and kurtosis measures were within +/-1.05 (with 

standard errors of 0.257, and 0.508, respectively). Analyses of variance were conducted for each 

dependent measure testing for differences in preference by condition and crew role (PF and PM).

3.6.7 Post-Experiment Debrief

A semi-structured verbal debrief session was held after the Post-Experiment Questionnaire was 

complete. This session was recorded and generally lasted between 45 to 90 minutes, and loosely followed 

the format of the Post-Experiment Questionnaire and specific items the researchers had noticed during 

that particular crew‟s scenarios.

3.6.8 Audio and Video Recordings

Audio and video recordings were made for each of the runs for each crew. Audio recordings were 

made of the post-experiment crew debrief. Recordings were subsequently analyzed to assess crew 

performance, opinion, crew resource management, and crew errors.
20

3.7 Research Facilities

3.7.1 Hardware and Software Configuration

NASA Langley Research Center‟s Integration Flight Deck (IFD) Simulator (Figure 5) was used, with 

a Boeing 757-200 vehicle model, and an environmental simulation of KBOS, as well as navigation and 

communication facilities within an approximate 50 mile radius from the airport.

Figure 5. Integration Flight Deck simulator

The IFD full-mission simulator is a duplicate of a standard Boeing 757-200 aircraft cockpit and is 

driven by a Boeing 757-200 aircraft dynamics mathematical model. The cockpit includes standard ship‟s 

instruments representative of a line operations Boeing 757-200 aircraft. The main instrument panel 

contains the Primary Flight Display (PFD), ND, Engine Indicating and Caution Alerting System

(EICAS), flight instruments (airspeed, altitude, attitude, etc), as well as standby altimeter and gear lever. 

The center control stand consists of a typical B-757 throttle quadrant, flap and speed brake controls, 

reverse thrust, spoiler handles, dual FMS CDUs, several electronic panels for controlling the PFD and 

ND, as well as researcher specified systems. The IFD houses a standard Mode Control Panel (MCP) 

under the glare-shield, and a complete overhead panel. 

The cockpit‟s visual system is a panorama system using five video projectors that provide 200° 

horizontal by 40° vertical field-of-view, with 1440 x 1024 pixel resolution. The visual scene used for this 

experiment was the KBOS terminal environment in a day, Visual Meteorological Conditions (VMC)

setting. Up to 20 moving aircraft, and 15 static aircraft were depicted in the arrival and surface taxi 
21

scenarios, and this traffic was accurately projected in the out-the-window (OTW) displays, and shown on 

the moving map display, as appropriate for that run condition.

3.7.2 Additional Simulation Capabilities 

In support of this experiment, the following hardware and software additions to the IFD baseline 

configuration were incorporated:

 MMDs, presentable on the NDs at both crew stations, with the capability to display ownship 

cleared route.

 Electronic Flight Instrumentation System (EFIS) controls at both crew stations, to control 

scale and display mode for the NDs. Display mode selection allowed crews to see an airport 

depiction, with expected taxi route, while airborne during the simulated approach.

 The capability to trigger the playing of researcher-provided audio wave files, based on 

simulated aircraft position, range to traffic, and/or specified cockpit control actuation (such as

microphone transmit release).

 Additional selectable pages on both FMS CDUs, to support a hierarchical Data Comm uplink 

and downlink capability, as well as the capability to selectively load expected or cleared 

routes into the MMDs.

 The capability to simulate (visually OTW) push-back from the terminal gate.

A Rockwell Collins EP-1000 KBOS database was used for OTW projection of the airport surface, 

taxiways, runways, buildings, obstructions, signs, and airport terrain and cultural features. The IFD 

simulation also used the appropriate database to provide accurate location and frequency of navigation 

aids, in particular the ILS RWY 27 and ILS RWY 33L. Frequencies aligned with published charts and 

pre-recorded Automatic Terminal Information Service (ATIS) messages were used based on 

environmental conditions and airport status for the particular scenario. The IFD employed a navigation 

and communications simulation that permitted realistic voice communication, as well as accurate 

navigation and flight crew position awareness during standard arrivals, appropriate to each scenario.

3.7.3 Oculometer Hardware and Software

A ten-camera oculometer system (Appendix F) was installed in the IFD to support unobtrusive 

collection of eye tracking and head position data for both flight crew subjects. This Smart Eye Inc. eye 

tracker uses a remote eye tracking system with facial recognition to calculate the position of defined 

points on a subject‟s head relative to the calibrated position of two or more cameras. The cameras used

the facial features to locate the corners of each of the subject‟s eyes and digitally zoomed to enhance the 

image of the eye.

3.8 Data Comm Messages and Displays

The general Data Comm message format and content is documented in Appendix G and was derived 

from Section 5 of Reference [33]. The specific Data Comm uplink messages based on those documents 

used in this experiment are listed in Appendix H. Each of the 11 crews received 96 Data Comm uplink 

messages (1056 total for all crews), and the crews had to respond with a downlink message to each one. 

The aggregate count of these messages per crew and over the entire experiment is tabulated in Table 3.
22

Table 3. Data Comm messages per crew and entire experiment

Departure Data Comm messages Per Crew Total

 Push back 6 66

 Start 6 66

 Expected D-TAXI 12 132

 D-TAXI 6 66

 Amended D-TAXI 6 66

 Cross Active Runway 6 66

 ATIS 6 66

 Altimeter 6 66

Departure sub-total: 54 594

Arrival Data Comm messages Per Crew Total

 Expected D-TAXI 12 132

 ATIS 6 66

 Altimeter 6 66

 Frequency change 6 66

 D-TAXI 6 66

 Amended D-TAXI 6 66

Arrival sub-total: 42 462

Total: 96 1056

 

Data Comm message format and page architecture were modeled after the Boeing 747-400 Future Air 

Navigation System 1/A (FANS-1/A) implementation. Display shapes, sizes, and colors on the ND were 

based on on-going research at NASA Langley, the proposed Data Comm standards [33], and discussions 

between members of the FAA and NASA Data Comm team.

The flight crew accessed Data Comm messages by depressing the CDU button labeled „ATC‟ (located 

on the top row of the CDU menu page selections, Figure 6), which caused the „ATC Index‟ page to be 

displayed on the CDU screen (left side of Figure 7). The „Prev Page‟ and „Next Page‟ CDU buttons 

(fourth row of Figure 6) were used by the flight crew to access the different pages of the CPDLC 

message, with the ability to send a CPDLC response always on the last page of the message (right side of 

Figure 8).

 Depressing the „Request‟ key on the „ATC Index‟ page (left image of Figure 7) accesses the ATC 

Request page (shown on the right side of Figure 7). 

 Depressing the „Log‟ key on the „ATC Index‟ page accesses the „ATC Log‟ page (left side of 

Figure 8). 

 Depressing any button on the right side of the ATC Log (left side of Figure 8) brings up the 

respective Data Comm message, such as the D-TAXI messages in the next three figures (Figures 

9-11). 

 Depressing the “Next Page” button (a separate button on the CDU panel) from the ATC Log 

page, reached the second page of the Data Comm message where the downlink response could be 

sent by the crew (right side of Figure 8).
23

Figure 6. Control Display Unit (CDU)

Figure 7. ATC Index (left) and ATC Request (right) pages
24

Figure 8. ATC Log (left) and Downlink Response (right) pages

Three types of routes are shown on the MMD in Figure 9, Figure 10, and Figure 11 that correspond to 

the three types of Data Comm D-TAXI messages. For these scenarios (MMD+Route display 

methodology), taxi routes were loadable on each crew‟s MMD, when either FMS CDU load button was 

pressed (left side of Figure 9, bottom-left key). Once loaded, the routes were not removable, except by 

replacing them with a new route. 

Expected D-TAXI routes for flight crew planning purposes were labeled “Expect Taxi” on the CDU, 

and depicted in dotted cyan on the ND (Figure 9). The CDU shows the message has been received and 

loaded, but a flight crew response has not yet been sent, so the status is depicted as “OPEN” on the CDU. 

Runway hold short bars were intentionally not shown with Expected D-TAXI route to differentiate them 

from D-TAXI uplink message. After the response was sent by depressing “Next Page” and then the line 

select key for “ROGER” (Page 2 shown in Figure 8 right side), the expected taxi route depiction on the 

ND did not change however the status on the CDU Page 1 changed from “OPEN” to “ROGER”.

Figure 9. Open Expected D-TAXI message on CDU (left) and display on ND (right)
25

Taxi instructions sent by Data Comm that the flight crew had not yet responded to were called 

Proposed D-TAXI, and were depicted as a dotted white line with runway hold short bars shown in red

(Figure 10). The remainder of the route after the red hold short bar was shown in dotted cyan. For this 

message, the message text on the CDU read “Taxi To” instead of “Expect Taxi To.” In this case, the 

route displayed (indicated by a dotted white line) changed to a cleared taxi route once a “WILCO” 

response had been sent (Figure 11). After a “WILCO” downlink was sent by the crew, the cleared taxi 

route was depicted in solid magenta to the first red hold short bar, with the remainder of the route after the 

red hold short bar remained in dotted cyan. Page 1 of the CDU was also changed to show “WILCO”. 

(Note: “ROGER” was used as the flight crew response for the “Expected D-TAXI” message since that 

message was informative, as were messages for altimeter settings and weather information. “WILCO” 

was used for “D-TAXI” and radio frequency change messages since they are directive.)

Figure 10. Open D-TAXI message on CDU (left) and display on ND (right)

Figure 11. Accepted D-TAXI message on CDU (left) and display on ND (right)
26

Amended D-TAXI instructions followed the same protocol as Proposed D-TAXI instructions. The 

text on the CDU displayed “Amended Clearance” and status as Open, while the ND displayed the current 

taxi route was a solid magenta line, with any proposed changes as solid white lines (Figure 12). 

Figure 12. Open Amended D-TAXI message on CDU (left) and display on ND (right)

The status of the message would change from OPEN to WILCO when the flight crew sent a Data 

Comm response, and the ND displayed the new route as a solid magenta line (Figure 13).

Figure 13. Accepted Amended D-TAXI message on CDU (left) and display on ND (right)
27

3.9 Experiment Protocol

Prior to the experiment, the subject pilots were scheduled and paired with others from the same flight 

organization. This tended to minimize adverse effects from differing standard operating crew procedures 

or crew resource management principles inherent in different airlines. To the maximum extent 

practicable, all crews used standardized, pre-briefed procedures. During the experiment, the pilot 

qualified as a Captain performed the role of the PF in the left crew station and was responsible for control 

of the simulated aircraft throughout the experiment. The pilot qualified as a First Officer performed the 

role of PM in the right crew station and had primary responsibility for Data Comm messages for the 

duration of the simulation experiment. The First Officer was the PM throughout the entire experiment to 

increase the statistical significance of collected data.

Subject pilots arrived at the research facility by 0745 on the first day and completed required 

paperwork. At 0800 the formal briefing began with completing the informed consent form required by 

NASA‟s Institutional Review Board, followed by a two hour training program (Appendix I). The training 

covered the purpose of the experiment, interactive practice sending and responding to Data Comm 

messages, a walk-through of each scenario, and practice completing the electronic questionnaires. From 

approximately 1000 to 1230, the subject pilots were in the IFD for part-task training and completed the 

four training scenarios. After lunch, the first group of eight runs was accomplished, usually finishing by 

about 1730. The second day began at 0800 with the second group of eight scenarios (replicate of the first 

group), and finished by 1200. Prior to the beginning of each scenario, the crews were given a verbal 

briefing about the upcoming scenario (Appendix J). After each scenario (departures lasted about 15 

minutes and arrivals about 20 minutes), five to ten minutes were required for the crew to answer the 

electronic questionnaire and the researchers to reconfigure the cockpit for the next scenario. After every 

third or fourth scenario, a break was taken to ensure the subjects were well rested. Following the last 

scenario, the subject pilots were brought back to the briefing room where they completed the postexperiment questionnaire on paper, generally taking 20 to 30 minutes. Following that, a semi-structured 

verbal debrief was held with the research team and the subject pilots, frequently lasting up to 90 minutes.

3.10 Subject Pilots

3.10.1 Requirements

NASA recruited subject pilots in support of this simulation experiment, and complied with all 

applicable procedures and laws relating to protection of human participants as specified by the 

Institutional Review Board. The following were specific requirements for all participant pilots:

 A US citizen or Permanent Resident status

 A valid FAA Airline Transport Pilot certificate

 Currently employed by a Part 121 air carrier or manufacturer

 Preference was given to pilots that held a Boeing 757 or 767 type-rating, however, other type 

ratings with CDU/FMS incorporation that is similar to the 757 / 767 were considered

 Preference was given to pilots familiar with the FANS-1/A CDU controls, displays, and 

functionality through flight experience. However, pilots not meeting this preference were

familiarized with FANS-1/A CDU during the training program portion of the experiment.

 All pilots had current or recent flight experience in the crew role they were assigned for the 

experiment (i.e., Captain or First Officer).

 A preference was given to subjects without hard edge bi-focal or tri-focal glasses.
28

Pilots were instructed to wear glasses only if/when absolutely necessary as there were detrimental 

effects to oculometer eye tracking ability depending on the type of glasses worn, specifically glasses with 

bi/tri-focal lenses. Head tracking was unaffected by the presence of glasses.

3.10.2 Subject Pilot Experience Level

Eleven crews of two pilots each participated in the study, with each crew comprised of a Captain and 

First Officer (FO) from the same airline. on the pilot experience data collected from the Biographical 

Questionnaire (Appendix C) are summarized in Table 4. All pilots were male with an average age of 48.6 

years, and their total flying time ranged from 6000 to 24,000 hours with a mean of 13,832.5 hours. In the 

Boeing 757 or comparable aircraft type, their time ranged from 1000 hours to 15,000 hours with a mean 

of 7768.6 hours. Nineteen of the 22 pilots had conducted flight operations into and out KBOS, and 

approximately half of the pilots had some prior experience with Data Comm. Six Captains and four FO 

pilots wore glasses during the experiment.

Table 4. Subject pilot experience level in years and hours

Mean

Age

Low 

Age

High 

Age

Std Dev 

Age

Mean

Years

Flying

Low 

Years

Flying

High 

Years

Flying

Std Dev 

Years

Flying

Captain 52.5 46.0 58.0 4.0 23.9 19.0 33.0 3.9

FO 44.2 37.0 56.0 5.6 15.0 10.0 26.0 4.8

Mean

Total 

Hours

Low 

Total 

Hours

High 

Total 

Hours

Std Dev 

Total 

Hours

Mean

B757 

Hours

Low 

B757 

Hours

High 

B757 

Hours

Std Dev 

B757 

Hours

Captain 17614 13750 25000 3784 7255 1100 10000 3139

FO 11242 6600 19460 3391 5036 1100 10000 3032
29

4 Results and Discussion

This first part of Section 4 presents a summary of results of flight crew response time to Data Comm 

uplink messages (complete data in Appendix K), and the distribution of those response times (complete 

data in Appendix L). Section 4.2 presents flight crew technical performance results for the rate of Nose 

Wheel Steering inputs and aircraft taxi speeds (complete data in Appendix K). Section 4.3 discusses 

results from the two independent oculometer systems (complete data in Appendix N). Section 4.4

presents a summary of results from the Post-Scenario Questionnaire (complete data in Appendix O). 

Section 4.5 is a summary of results from the Post-Experiment Questionnaire (complete data in Appendix 

P). Section 4.6 presents a summary of results from the verbal debrief session held at the end of the 

experiment. In response to RTCA Special Committee 214 (SC-214), the flight crew CPDLC response 

times are published in a particular format to support their analysis in Appendix M.)

„N‟ in this paper is used as the number of events that occurred (for example, number of times the flight 

crew responded to a Data Comm message, or the number of responses received on a question). 

4.1 Data Comm Message Response Time

Results from flight crew response to all Data Comm messages, excluding those that the crew took 

longer than two minutes to respond or were not responded to at all, are described in Section 4.1.1. The 

time distribution of these responses is presented in Section 4.1.2. The beginning of Appendix L contains 

the rationale for the two minute limit for flight crew responses to Data Comm messages, and lists by 

category the number of events that were excluded from analysis in the paper.

4.1.1 Response Times Based On All Data

Individual time to respond in seconds to Data Comm uplink messages are listed by crew in Appendix 

K. Figure 14 shows the mean of all flight crew Data Comm message response times by condition. 

Results show that the majority of analyzed response times were well under a minute (Mean = 20.7 

seconds, SD = 17.6 seconds across all conditions). There were a few occasions wherein crews reviewed a 

message and agreed to its content but did not respond to the message within two minutes (5 of 369 (~1%) 

directive Data Comm messages, and 27 of 660 (~4%) informative Data Comm messages). Video review, 

researcher experience, and verbal debrief with subject pilots suggests that these long response or nonresponse events were cases wherein the crew simply forgot that the message had not been responded to, 

rather than workload prioritization and shedding. This result suggests that improving the operational ease 

of answering a message over the FANS-1/A standard would improve crew response to messages, and/or 

implementation of ATC message timeouts and re-sends should be considered in Data Comm 

implementation.
30

Figure 14. Message response time by condition

Table 5 and Table 6 show mean, standard deviation, and paired mean differences for message response 

time in seconds by condition. The analysis indicated a significant effect (F(5,1010)=3.777, p=0.0022), 

but is only evident between the two lowest (Arrive/MMD and Arrive/MMD+Route) and the highest 

(Depart/MMD+Route) times on the figure (=0.05, HSD=5.47). However, this difference is not 

operationally significant. The remaining response times by Condition exhibited no statistically significant 

difference.

Table 5. Mean response time and standard deviation by condition

Arrival /

Paper

Depart /

Paper

Arrival /

MMD

Depart /

MMD

Arrival /

MMD+Route

Depart /

MMD+Route

Mean (seconds) 18.7 22.9 17.8 21.0 18.2 24.0

Standard Deviation 14.6 20.4 15.7 16.6 12.8 21.2

N 146 188 153 197 147 185

Table 6. Pairwise comparisons of response time by condition (= .05, HSD = 5.47)

Depart /

Paper

Arrival /

MMD

Depart /

MMD

Arrival /

MMD+Route

Depart /

MMD+Route

Arrival / Paper (seconds) 4.2 0.9 2.3 0.5 5.3

Depart / Paper 5.1 1.9 4.7 1.1

Arrival / MMD 3.2 0.4 6.2 *

Depart / MMD 2.8 3.0

Arrival / MMD+Route 5.8 *

NOTE: statistical significance indicated by *

Figure 15 shows a plot of the mean data, and Table 7 shows mean and standard deviation for message 

response time by display methodology (arrival and departure aspects of the conditions collapsed). The 

analysis indicated no significant differences (F(2,1013)=1.027, p=0.36) between any of the groups. 
31

Figure 15. Response time by display methodology

Table 7. Mean response time and standard deviation by display methodology

Paper MMD MMD+Route

Mean (seconds) 21.0 19.6 21.4

Standard Deviation 18.2 16.3 18.2

N 334 350 332

Additional analysis was requested by the FAA, and one of those requests was message response time 

by phase of flight. Figure 16 shows a plot of the mean data and Table 8 the mean and standard deviation 

message response times by arrivals (inflight and surface) and departures (surface only). The analysis 

indicated a significant effect (F(1,1013)=15.85, p<0.001), (=0.05, HSD=2.81, Mean Difference=4.4) 

between the two groups. Results of the analyses indicate departure operations had a statistically 

significant longer response times than arrival operations (although 4 seconds would not be operationally 

significant), with variations in response time due to display methodology being not significant.

Figure 16. Message response time by phase of flight

Table 8. Mean response time and standard deviation by phase of flight

Phase of Flight Arrival Departure

Mean (seconds) 18.2 22.6

Standard Deviation 14.4 19.5

N 446 570
32

Figure 17 shows mean Data Comm message response times by message type. Statistical analysis of 

the mean response time by message type indicated a significant message type effect (F(5,6)=12.683, 

p=0.004), but only between certain message types (Information versus Frequency; Frequency versus 

Expected Taxi; Pushback and Start versus Expected Taxi; Expected Taxi versus Taxi; and Expected Taxi 

versus Amended Taxi). Table 9 shows mean and standard deviation by message type and Table 10 the

difference between means for paired response times by message type. Analysis indicated a significant 

effect (F(5,1010)=7.602, p<0.0001), (=0.05, HSD=6.09, p<0.05) for all the comparisons involving 

Expected Taxi messages. The remaining response times by condition did not differ significantly.

Figure 17. Mean response time by message type

Table 9. Mean response time and standard deviation by message type

Message Type Info Frequency Pushback 

and Start

Expected

Taxi

Taxi Amended

Taxi

Mean (seconds) 19.0 16.0 19.5 26.5 19.8 18.6

Standard Deviation 19.3 12.6 17.0 20.7 16.6 12.2

N 250 65 127 253 132 190

Table 10. Pairwise comparisons of response time by message type (=0.05, HSD=6.09)

Frequency Push back 

and Start

Expected

Taxi

Taxi Amended

Taxi

Info (seconds) 3.1 0.5 7.4 * 0.8 0.5

Frequency 3.6 10.5 * 3.9 2.6

Pushback and Start 6.9 * 0.3 1.0

Expected Taxi 6.6 * 7.9 *

Taxi 1.3

NOTE: statistical significance indicated by *

Researcher experience suggests that Expected Taxi message response times were somewhat longer 

because they were delivered during times of relatively high workload for the PM, and there was a 

perceived absence of operational urgency in responding to them. Taxi and Amended Taxi message 
33

response times benefited from the perception of operational urgency. The frequency change message was 

delivered during low workload for the PM (after configuration changes and checklists were complete).

4.1.2 Response Time Distributions

To address the FAA‟s request for time required for flight crew response, additional analysis was 

conducted that removed responses greater than two minutes and the results are presented in this Section 

with complete data available in Appendix L. Thirty-nine of the 1056 Data Comm uplink messages 

(approximately 4%) were not responded to within 120 seconds. In all cases, it appeared to the researchers 

that the root cause was that the pilots read the uplink message, mentally processed it, and were complying 

if appropriate, but believed either they had acknowledged the message or forget to acknowledge the 

message on the second page of the FANS-1/A implementation. This statistic and researcher observation 

is collaborated by crew debrief comments where they commented having to proceed to a separate page to 

respond led to occasional mistakes. 

For Data Comm message response time within two minutes (N=1017) the distribution seen in Figure 

18 shows that response times are not normally distributed, which is expected due to the left hand limit of 

zero seconds for response time. Heavily-peaked, positively-skewed distributions indicate that regardless 

of message type, pilots attempt to answer the message as soon as operationally possible, with rare 

situations arising when a message cannot be immediately answered, or the pilots believed they had

acknowledged the message (see Appendix L for distribution by message type). From an operational 

standpoint, the distributive shape of response times suggest that the flight crew attempted to answer all 

Data Comm messages in an expeditious manner.

Figure 18. Distribution of flight crew Data Comm response times

4.2 Flight Technical Performance

The FAA requested additional data analysis to explore results based on arrival and departure 

scenarios; therefore, Section 4.2.1 and 4.2.2 include these analyses although they are not part of the 

original hypotheses. The PF Flight Director error data was collected and analyzed for the flight portion of 

the arrival scenarios; however, no statistical correlation or significance was found. Therefore, the 

summary of that data is presented only in Appendix K.
34

4.2.1 Power Spectral Density of NWS Rate

A statistical analysis was conducted on NWS rate Power Spectral Density (PSD) during arrival and 

departure taxi (individual crew performance shown in Appendix K). NWS PSD is a measure of control 

activity during taxi, which may be qualitatively related to PF (physical) workload. Analyses were 

conducted separately for arrivals and departures by condition, for Day 1 and 2 of the experiment, and for 

all arrivals and all departures. Figure 19 shows an average of NWS Rate Power Density Spectrum

(degrees per second squared times frequency in Hz) for Arrival and Departure Scenarios, as a function of 

condition. 

Figure 19. NWS rate PSD during arrival (left) and departure (right) by condition

NWS PSD analysis by condition for arrival scenarios:

Table 11 shows mean and standard deviation and Table 12 the paired mean differences for NWS PSD 

during arrivals by experimental condition. The Tukey HSD analysis indicated no significant difference

(F(3,80)=3.388, p =0.022), (=0.05, HSD=44.63, p<0.05) for any of the pairwise comparisons. Results 

during arrival operations show no statistically or operationally significant differences in NWS activity by 

condition. 

Table 11. NWS PSD for arrival scenarios by condition

Voice /

Paper

DataComm /

Paper

DataComm /

MMD

DataComm /

MMD+Route

Mean ((Deg/Sec)2 Hz) 196.6 164.5 195.1 152.6

Standard Deviation 69.9 37.8 56.2 51.1

N 21 21 21 21

Table 12. Pairwise comparisons of arrival NWS PSD by condition (=0.05, HSD=44.6)

Data Comm /

Paper

Data Comm /

MMD

DataComm /

MMD+Route

Voice / Paper 32.1 1.5 44.0

DataComm / Paper 30.6 11.9

DataComm / MMD 42.5

NOTE: no cell comparisons were statistically significant
35

NWS PSD analysis by condition for departure scenarios:

Table 13 shows mean and standard deviation, and Table 14 the paired mean differences for NWS PSD 

during departures by experimental condition. The Tukey HSD analysis indicated a significant difference

(F(3,80)=4.959, p=0.0033), (=0.05, HSD=44.88, p<0.05) between the Voice/Paper and Data 

Comm/MMD+Route conditions. No other paired comparisons were significant. 

Results show an increase in NWS activity when going from Voice to Data Comm modality. 

Departure routes were relatively long (about 15 minutes) and complex (5-10 turns). It is possible that the 

decrease in head up time associated with reading and interpreting Data Comm clearances, as well as the 

compelling nature of the MMD and loadable routes, contributed to less time available for head up precise 

path control, and thus, greater NWS activity in making fine corrections.

Table 13. NWS PSD for departure scenarios by condition

Voice /

Paper

DataComm /

Paper

DataComm /

MMD

DataComm /

MMD+Route

Mean 207.5 236.1 248.8 271.9

Standard Deviation 54.6 56.0 48.8 61.0

N 21 21 21 21

Table 14. Pairwise comparisons of departure NWS PSD by condition (= .05, HSD = 44.88)

Data Comm /

Paper

Data Comm /

MMD

Data Comm /

MMD+Route

Voice / Paper 28.6 41.3 64.5 *

Data Comm / Paper 12.7 35.8

Data Comm / MMD 23.2

NOTE: statistical significance indicated by *

NWS PSD analysis by day effect:

Table 15 shows mean and standard deviation for NWS PSD for all conditions and phases of flight, by 

day (i.e., which day of the experiment the event occurred) to investigate training effects. Results of the 

Analysis of Variance (ANOVA) indicated no significant difference (F(1,166)=2.2, p=0.14), between the 

two days of the experiment (for each crew).

Table 15. NWS PSD for all scenarios by day

Day 1 Day 2

Mean 217.1 201.9

Standard Deviation 71.1 61.1

N 80 88
36

NWS PSD analysis by phase of flight:

Table 16 shows mean and standard deviation for NWS PSD for all conditions and test days, by phase 

of flight. Results of the ANOVA indicated a significant difference (F(1,166)=50.62, p<0.0001), between

arrival and departure scenarios, supporting the discussion above concerning increases in NWS activity 

because there are more turns required, by design, for the departure than for the arrival taxi tasks. This is 

additional analysis requested by the FAA.

Table 16. NWS PSD by phase of flight

Arrivals Departures

Mean 177.2 241.1

Standard Deviation 57.3 59.1

N 84 84

4.2.2 Taxi Speed

Figure 20 shows taxi speed during Arrival and Departure Scenarios, with respect to communications 

modality and display methodology (data by crew in Appendix M). Table 17contains the mean and

standard deviation of taxi speed, and Table 18 the paired mean differences by condition. The analysis 

indicated a significant effect (F(3,80)=10.01, p<0.0001), (=0.05, HSD=2.35, p<0.05) for Voice/Paper 

and Data Comm/Paper, Voice/Paper and Data Comm/MMD+Route, and Data Comm/MMD and Data 

Comm/MMD+Route.

Figure 20. Taxi speed during arrival (left) and departure (right) by condition
37

Table 17. Mean arrival taxi speed and standard deviation by condition

Voice /

Paper

DataComm /

Paper

DataComm /

MMD

DataComm /

MMD+Route

Mean (knots) 13.5 16.2 14.3 18.0

Standard Deviation 1.9 2.0 2.0 4.7

N 21 21 21 21

Table 18. Pairwise comparisons arrival taxi speed by condition (=0.05, HSD=2.35)

Data Comm /

Paper

Data Comm /

MMD

Data Comm /

MMD+Route

Voice / Paper (knots) 2.7 * 0.8 4.5 *

Data Comm / Paper 1.8 1.8

Data Comm / MMD 3.7 *

NOTE: statistical significance indicated by *

Table 19 shows mean and standard deviation, and Table 20 shows the mean differences for departure 

taxi speed by condition. The analysis indicated a significant effect (F(3,80)=11.6, p<0.0001), (=0.05, 

HSD=1.05, p<0.05), for Voice/Paper and any of the other conditions.

Table 19. Mean departure taxi speed and standard deviation by condition

Voice /

Paper

DataComm /

Paper

DataComm /

MMD

DataComm /

MMD+Route

Mean (knots) 14.5 12.3 13.0 12.8

Standard Deviation 1.8 1.1 0.8 1.2

N 21 21 21 21

Table 20. Pairwise comparisons of departure taxi speed by condition (=0.05, HSD=1.05)

Data Comm /

Paper

Data Comm /

MMD

Data Comm /

MMD+Route

Voice / Paper (knots) 2.2 * 1.5 * 1.7 *

Data Comm / Paper 0.7 0.6

Data Comm / MMD 0.1

NOTE: statistical significance indicated by *

For arrivals, the results show a slight (2 knot) but significant increase in taxi speed with Data Comm 

modality over that in Voice modality. For departures, the results show a slight (2 knot) but significant 

decrease in average taxi speed of DataComm/MMD+Route over Voice/Paper. The increase in taxi speed 

on arrivals may be due to increased situation awareness for the PF when routes were presented more 

clearly in those scenarios, where shorter and simpler routes required less attention to turns than in 

departure scenarios. Though overall taxi speeds were higher for arrivals (Mean=15.5 knots, SD=3.3 

knots) than departures (Mean=13.2 knots, SD=1.5 knots), it is important to realize that arrival scenario 

data analysis began at 30 knots, there were no active runway holds, there were fewer turns, taxi times 

were shorter, and scenario was terminated with the aircraft still moving.
38

4.3 Head Tracking

Some variability in head and eye tracking behavior existed across subjects during data collection. The 

main factor in variance was due to some pilots requiring the use of reading glasses as pilots tended to 

wear them low on the bridge of their nose to look over them when focusing outside the flight deck. This 

behavior often reversed the head up/head down pitch behavior when compared to pilots not wearing 

glasses in this manner, with pilots wearing reading glasses pitch their head down to look out the window. 

Although the use of glasses was discouraged in the attempt to maintain data integrity, pilots were still 

allowed to perform the tasks as they would in real world operations. If eye tracking data was available, 

there was no impact as the gaze vector was true regardless of head pitch angle. However, if only head 

tracking was available the impact of wearing reading glasses had to be accounted for. (Note: analysis of 

eye tracking data collected showed the software was able to maintain track when pilots transitioned 

between cockpit instruments and looking out the window.)

4.3.1 Head Up Aggregate Results

Aggregate head tracking analysis (Table 21 and Table 22) indicated that the overall effect observed 

was a statistically insignificant decrease in the PF head up time in scenarios involving Data Comm, with 

significant difference across conditions for the PM (F(1,3)=4.03, p=0.008). For head tracking data, 

differences in the „N‟ value is due to some data not being usable (lost calibration, interference from 

glasses, etc.). [Note: data tables in Section 4.3 have a slightly different format than in Section 4.1 and 4.2 

due to different approaches to calculating pairwise comparisons. In this section, =0.05 is used and the p

value is only listed if it is significant.]

It is postulated that the increased requirement for the PM to interface with Data Comm messages in 

these scenarios using a CDU mounted in a relatively low location in the cockpit reduced the capacity for 

frequent lookout tasks. Display methodology conditions showed a small magnitude effect on pilot head 

up time, with no greater than 10% variance across the means for each crew role. Further research should 

be conducted to test display methodology conditions combined with Voice and not simply Data Comm in 

order to remove the effect of the Data Comm head up time impact.

Table 21. Aggregate head up time and standard deviation for PF and PM by condition

PF Voice / 

Paper

Data Comm /

Paper

Data Comm /

MMD

Data Comm /

MMD + Route

Mean (percent) 47.076 45.193 42.622 41.434

Standard Deviation 26.751 23.465 23.062 22.558

N 40 41 35 40

PM

Mean 41.152 33.388 31.861 34.267

Standard Deviation 17.126 13.622 10.883 14.46

N 42 42 40 39
39

Table 22. Pairwise comparisons of aggregate head up time by condition

PF Data Comm /

Paper

Data Comm /

MMD

Data Comm /

MMD+Route

Voice / Paper (percent) Not significant Not significant Not significant

Data Comm / Paper Not significant Not significant

Data Comm / MMD Not significant

PM

Voice / Paper Not significant p=0.0187 Not significant

Data Comm / Paper Not significant Not significant

Data Comm / MMD Not significant

Whether or not a decrease in head up time is acceptable may depend on the phase of flight and 

associated task loading during which the decrease is observed. In-flight, head tracking analysis indicated

that regardless of condition pilots focused a majority of their attention inside the cockpit. The PF spent 

less time head up than the PM in flight since the use of the auto-pilot was not allowed, however during 

surface operations, the PM spent less time head up than the PF due to cockpit tasks involved with running 

checklists, programming the flight management computer, and answering Data Comm messages. When

the pilots‟ attention is required outside the flight deck, such as during taxi, head tracking analysis 

observed the greatest variation across modalities and crew role. Of note, crew qualitative data presented 

in Appendix N indicated that the decrease in head up time associated with Data Comm employment was

not unacceptable.

Figure 21 shows the percentage of time the PF and PM were head up by arrival altitude bands, arrival 

taxi, and departure taxi (the complete data by crew is available in Appendix N). These results stand in 

sharp contrast to the recommendation given in Section 8-1-6(c) of the Aeronautical Information Manual. 

[36] The paragraph titled “Scanning for Other Aircraft” states 2⁄3 to 3⁄4 of the pilot‟s scan should be 

outside the aircraft, whereas this experiment showed current commercial pilots operating in busy terminal 

airspace scanned outside the cockpit approximately 10% of the time while hand-flying the aircraft, and 

approximately 50% outside the cockpit while operating on the surface.

Figure 21. Percent head up time for PF (left) and PM (right) by location
40

4.3.2 Head Up Time By Altitude Bands

The FAA specifically requested flight crew head up data be analyzed in terms of altitude, which this 

section addresses. Figure 22 presents a summary of PF and PM head up time, as a function of altitude 

band and scenario condition. Following this data, the results are presented for each altitude band are 

presented (High: 16,000 – 14,000 MSL; Medium: 10,000 – 8,000 MSL; Low 7,000 – 5,000 MSL).

A statistical analysis of the head up percentage (%) in the high altitude band (Table 23 and Table 24) 

indicated a significant condition effect, F(1,3)=8.10, p<0.001. The Tukey HSD pairwise comparison tests 

indicated a significantly lower head up percentage with the Data Comm/Paper condition than the 

Voice/Paper diagram condition (T=-2.826, p=0.0303) for the PF. The remaining head up percentages by 

display methodology did not differ significantly for the PF. The HSD pairwise comparison tests for the 

PM indicated a significantly higher head up time in the Voice/Paper condition than the Data 

Comm/MMD (T=-3.888, p=0.0012) and the Data Comm/MMD+Route (T=-2.644, p=0.0447). No 

statistically significant difference in head up time was found to exist between crew members nor was a 

statistically significant interaction found to exist between crew member and condition.

Results in the high altitude band indicated that there was a statistically significant change in pilot head 

up behavior in the high altitude Data Comm scenarios. The Voice/Paper condition showed significantly 

higher head up percentage than the Data Comm/paper condition, suggesting that the effect was due to the 

Data Comm. No significant difference within the Data Comm conditions across display methodologies 

was observed. 

Figure 22. Percent head up time for PF (left) and PM (right) by altitude band and condition

Table 23. Head up time for PF and PM in high altitude band by condition

PF Voice / 

Paper

Data Comm /

Paper

Data Comm /

MMD

Data Comm /

MMD + Route

Mean (percent) 15.788 6.849 5.443 11.931

Standard Deviation 13.472 7.905 4.504 14.565

N 20 20 17 20

PM

Mean (percent) 23.047 12.273 5.659 11.222

Standard Deviation 16.13 16.047 9.042 14.641

N 21 21 20 20
41

Table 24. Pairwise comparisons of High altitude band head up time by condition

PF Data Comm /

Paper

Data Comm /

MMD

Data Comm /

MMD+Route

Voice / Paper (percent) Not significant p=0.0303 Not significant

Data Comm / Paper Not significant Not significant

Data Comm / MMD Not significant

PM

Voice / Paper Not significant p=0.0012 p=0.0477

Data Comm / Paper Not significant Not significant

Data Comm / MMD Not significant

Table 25 and Table 26 list head up time for scenarios with Data Comm messages presented in the 

medium altitude band. Tukey HSD pairwise comparison tests indicated no statistically significant 

differences between the Voice/Paper and Data Comm/Paper communication modalities or across any of 

the three display methodologies. No statistically significant difference in head up time was found to exist 

between crew members nor was a statistically significant interaction found to exist between crew member 

and condition.

Table 25. Head up time for PF and PM in medium altitude band by condition

PF Voice / 

Paper

Data Comm /

Paper

Data Comm /

MMD

Data Comm /

MMD + Route

Mean (percent) 16.18 13.763 8.556 6.012

Standard Deviation 18.852 12.929 13.327 7.424

N 20 20 17 20

PM

Mean (percent) 22.849 11.093 12.139 12.443

Standard Deviation 19.11 11.797 14.929 15.707

N 21 21 20 20

Table 26. Pairwise comparisons of medium altitude band head up time by condition

PF Data Comm /

Paper

Data Comm /

MMD

Data Comm /

MMD+Route

Voice / Paper (percent) Not significant Not significant Not significant

Data Comm / Paper Not significant Not significant

Data Comm / MMD Not significant

PM

Voice / Paper (percent) Not significant Not significant Not significant

Data Comm / Paper Not significant Not significant

Data Comm / MMD Not significant

A statistical analysis of the head up percentage in the low altitude band (Table 27 and Table 28) 

indicated a significant condition effect F(1,3)=4.46, p=0.005. Tukey pairwise comparison tests indicated 

the Data Comm/MMD+Route condition had significantly greater head up time than the Data 

Comm/Paper (T=2.740, p=0.0378) for the PF, and the Voice/Paper condition had significantly greater 
42

head up time than the Data Comm/Paper condition (T=-2.732, p=0.0382). Statistical analysis indicated a 

significant crew role effect, indicating the PM showed greater head up than the PF F(1,3) = 4.09, p = 

0.045.

Results for the low altitude band suggest that both the communication modality and display 

methodology had a significant effect on the percentage of PF head up time. Data Comm/MMD and Data 

Comm/MMD+Route were both associated with significantly more eyes-out when compared to the Data 

Comm/Paper condition, but not with each other. This suggests that the MMD and MMD+Route displays 

allow the PF‟s attention to be out the window more than the Data Comm/Paper condition. There was also 

a significant difference between Data Comm/Paper and Voice/Paper conditions. Data Comm/Paper had a 

significantly lower head up percentage than all other three conditions. This is partially explained by 

pilots having to have attention inside the flight deck to locate the paper maps and prepare for expected 

taxi accordingly, which may not have been as necessary with the MMD conditions.

Table 27. Head up time for PF and PM in low altitude band by condition

PF Voice / 

Paper

Data Comm /

Paper

Data Comm 

/ MMD

Data Comm /

MMD + Route

Mean (percent) 10.568 3.553 11.044 13.44

Standard Deviation 11.691 4.739 15.687 11.562

N 20 20 17 20

PM

Mean (percent) 19.377 7.189 16.189 12.662

Standard Deviation 17.521 11.42 13.729 14.464

N 21 21 20 20

Table 28. Pairwise comparisons of low altitude band head up time by condition

PF Data Comm /

Paper

Data Comm /

MMD

Data Comm /

MMD+Route

Voice / Paper (percent) Not significant Not significant Not significant

Data Comm / Paper Not significant p=0.0378

Data Comm / MMD Not significant

PM

Voice / Paper (percent) Not significant Not significant Not significant

Data Comm / Paper Not significant Not significant

Data Comm / MMD Not significant

In general, during inflight arrival conditions, a finding of significant difference between 

communication modality suggests that the introduction of the Data Comm drives pilots‟ attention inside 

the cockpit reducing their head up time. In the low altitude band, the display methodology impacted the 

head up percentage, combined with significance existing between crew role suggesting the introduction of 

a MMD or MMD+Route allows pilots to spend more time head up. This difference in crew role is largely 

explained by the crew role responsibility differences during this phase of flight. As the aircraft 

approaches the runway, the PF will bring his/her attention out the window, relying less on instruments. 

This is especially true in daytime VMC, the weather condition for all scenarios in this experiment. 

However, this behavior was not observed in the head tracking results when contrasted to head up

percentages of the other in-flight altitude bands. This change in behavior may be due to the PF hand 
43

flying the aircraft and spending significant time monitoring the PFD glide slope and course deviation 

markers that become increasingly difficult to track as the aircraft closes in on the runway, keeping his 

attention inside the flight deck. The PM also spends a slightly increased amount of time head up

compared to the PF during this phase, sharing time on the instruments to make call-outs to the PF. 

4.3.3 Head up Time During Taxi Operations

As in Section 4.2, the FAA also requested additional data analysis of head up time during taxi or 

surface operations as a function of arrival and departure scenarios. Statistical analysis comparing the 

variance between arrival and departure taxi scenarios indicated the departure taxi scenario yielded 

significantly greater head up time than the arrival taxi scenario, F(1,3)= 2.09, p<0.001. No significance 

was found in the interaction of arrival or departure conditions, suggesting that conditions varied similarly 

for both arrival taxi and departure taxi. Figure 23 summarizes the findings.

Figure 23. Percent head up time during arrival (left) and departure (right) taxi operations

As shown in Table 29 and Table 30, a statistical analysis of the head up percentage in the arrival taxi

scenarios indicated a significant condition effect, F(1,3)=4.47, p=0.005. There were no significant 

differences across conditions for the PF. There was significantly greater head up time in the Voice/Paper 

condition than the Data Comm/Paper condition (T=-3.144, p=0.0125), greater head up time in the 

Voice/Paper than the Data Comm/MMD (T=-3.867, p=0.013), and greater head up time in the 

Voice/Paper than the Data Comm/MMD+Route (T=-5.278, p<0.0001) for the PM. There was statistical 

significance between crew role, F(1,3)=89.89, p<0.001, and significant interaction between crew role and 

condition, F(1,3)=4.16, p=0.007.

Table 29. Head up time for PF and PM during arrival taxi operations by condition

PF Voice / 

Paper

Data Comm /

Paper

Data Comm /

MMD

Data Comm /

MMD+Route

Mean (percent) 53.717 59.517 51.008 53.864

Standard Deviation 20.374 14.677 12.68 17.591

N 20 20 16 19

PM

Mean (percent) 44.272 31.627 28.315 22.491

Standard Deviation 14.945 12.948 10.871 12.826

N 21 21 19 19
44

Table 30. Pairwise comparisons of arrival taxi head up time by condition

PF Data Comm /

Paper

Data Comm /

MMD

Data Comm /

MMD+Route

Voice / Paper (percent) Not significant Not significant Not significant

Data Comm / Paper Not significant Not significant

Data Comm / MMD Not significant

PM

Voice / Paper (percent) p=0.0125 p=0.0013 p<0.0001

Data Comm / Paper Not significant Not significant

Data Comm / MMD Not significant

Analysis of the arrival taxi phase represents the pilots‟ behavior from rollout (below 80 knots) to the 

end of the run. Comparison tests indicated a significant difference between Voice and Data Comm

coupled with a display methodology. This effect was pronounced in the case of the PM, with little 

variance existing across conditions for the PF. The PM with Data Comm/MMD+Route condition

indicated significantly lower head up time compared to the PM with Voice/Paper condition, suggesting 

not only did the Data Comm decrease head up time, but Data Comm combined with any display 

methodology did as well. This effect was only observed with the PM. 

Statistical significance between crew role and the interaction of crew role and scenario condition

indicated a difference in crew behavior between the PF and PM that is dependent on the communications 

modality and display methodology. The taxi phase of arrival scenarios was considerably shorter than the 

departures phase (2-3 minutes versus about 15 minutes). It is postulated that the faster pace of the taxi 

portion of arrival scenarios necessitated a tactical focus on the part of the PF, with the primary task being 

recognition and execution of upcoming turns requiring their attention out the window increasing their 

head up time. This requirement remained essentially unchanged across conditions, as SA was essentially 

provided entirely verbally by the other crew member, rather than by displays or messages. The PM had to 

assume more of a strategic role (interpreting the rapidly changing clearances and providing directive 

commentary to the PF), driving their attention inside the flight deck. The presence of Data Comm and the 

low location of the CDU interface exacerbated the effect of decreased head up time for the PM on the 

arrival taxi.

A statistical analysis of the head up percentage in the departure taxi scenario indicated a significant 

condition effect, F(1,3)=11.08, p<0.001 (Table 31 and Table 32). Tukey pairwise comparison tests 

indicated significantly more head up in the Voice/Paper condition compared to the Data Comm/MMD 

condition, (T=-3.346, p=0.0069) and (T=-4.341, p=0.0003) for the PF and PM respectively. There was 

significantly more head up time with the Voice/ Paper condition than the Data Comm/ Paper condition 

(T=-3.506, p=0.0042) for the PM. Also observed was significantly more head up time with the 

Voice/Paper condition than the Data Comm/MMD+Route condition (T=-3.809, p=0.0016) for the PF. 

The remaining head up percentages by condition did not differ significantly. There was statistical 

significance between crew role, F(1,3)=195.70, p<0.001. No statistically significance difference was 

found in the interaction between crew role and condition.
45

Table 31. Head up time for PF and PM during departure taxi operations by condition

PF Voice / Paper Data Comm /

Paper

Data Comm /

MMD

Data Comm /

MMD+Route

Mean (percent) 71.281 66.094 62.607 61.668

Standard Deviation 9.196 7.665 7.099 7.738

N 20 21 18 20

PM

Mean (percent) 53.175 42.577 39.89 46.759

Standard Deviation 12.307 11.106 6.72 7.574

N 21 21 20 19

Table 32. Pairwise comparisons of departure taxi head up time by condition

PF Data Comm /

Paper

Data Comm /

MMD

Data Comm /

MMD+Route

Voice / Paper (percent) Not significant p=0.0069 p=0.0016

Data Comm / Paper Not significant Not significant

Data Comm / MMD Not significant

PM

Voice / Paper (percent) p=0.0042 p=0.0003 Not significant

Data Comm / Paper Not significant Not significant

Data Comm / MMD Not significant

In departures, the presence of Data Comm decreased the head up percentage relative to the Voice

communication condition, significantly so for the PM. No significant difference in Data Comm 

conditions was observed across display methodology, suggesting pilot behavior was not significantly 

affected by ownship being presented on the head down displays versus the use of a paper map. 

Significance between crew role but not the interaction of crew role and condition indicates that there was 

a similar behavior change across conditions for each pilot, with variance in head up percentage being 

derived from differences in crew tasks during the departure taxi phase. In contrast to the arrival 

scenarios, departure scenarios afforded more time for decision making for both crew members, and 

allowed the PF to assume a more strategic role. Thus, the method of information delivery had a greater 

effect on head up time for this crew role. The decreased pace of decisions also afforded greater head up

time for both crew members in these scenarios.

4.4 Post-Scenario Questionnaire Results

Section 4.4 presents a summary of results from the questionnaires. Complete data from the PostScenario Questionnaires is in Appendix O, and from the Post-Experiment Questionnaire in Appendix P.

For the Post-Scenario Questionnaire, workload and situation awareness (SA) responses were categorized 

into „inflight‟ (when the aircraft was airborne, and occurred only during arrival scenarios), and „surface‟ 

(aircraft movement on the ground, occurred in both arrival and departure scenarios).

4.4.1 Post-Scenario Ratings on Workload 

Subjects used Bedford scale to rate the workload associated with inflight and surface operations. Full 

results are in Appendix O, Section O.1. Figure 24 presents results from post-scenario questionnaires 

regarding workload, and indicates a perception of relatively low workload for all conditions (both inflight 
46

and surface operations). Along the x-axis, a rating of 1 indicates “workload insignificant”, 5 “reduced 

spare capacity”, and 10 “task abandoned” (the higher count for „Surface‟ is due to that operation 

occurring in both arrival and departures scenarios.)

Figure 24. Inflight (top) and Surface (bottom) workload ratings by condition

PF ratings of workload in flight operations (Table 46 and Table 48) during arrivals were significantly 

higher than PM (

2

(1)=9.094, p= 0.003), but were not for surface operations (Table 47 and Table 48) 

during arrivals and departures (

2

(1) =2.339, p=0.126). A binomial test with a cutpoint of 3 (“Enough 

spare capacity for all desirable additional tasks”) and test proportion of 75% showed that most ratings 

were significantly on the low workload side of the scale for both PFs and PMs, for inflight and surface 

operations (Table 49). For inflight operations, over 84% of PF ratings and over 94% of PM ratings were 

3 or less; and of the 88 rating opportunities, PF rated workload at 7 (“Very little spare capacity, but 

maintenance of main task not in question”) or greater in only 5 case and PM only once. For surface 

operations, over 78% of PF ratings and over 85% of PM ratings were 3 or less; and of the 176 rating 

opportunities, PF ratings were 7 or more in only 6 cases, and for PMs only in 5 cases.

Figure 25 shows the median responses for PF and PM workload ratings for flight portions of the 

arrival scenarios. Both PF and PM rated mean workload significantly different during flight operations
47

(Table 51 and Table 52) among the display conditions (PF: 

2

(3)=28.525, p<0.001, PM: 

2

(3)=25.245, p< 

0.001). While medians appear fairly constant, pairwise Wilcoxon Signed Rank tests (Bonferroni adjusted 

=0.05, whereby significance is p<0.008) show that PF rated workload in flight operations with the 

DataComm/Paper condition as significantly different than any other condition (Table 51, Table 52, and 

Table 54). Mean ranks suggest that the DataComm/Paper ratings are higher (Table 46 and Table 54). PM 

rated workload for flight operations as significantly different between Voice/Paper and both 

DataComm/Paper and DataComm/Route; where Voice/Paper mean ranks are lower than either of these 

other conditions. PM ratings for inflight workload for the DataComm/Route condition were not only 

significantly higher than those for Voice/Paper, but also for DataComm/MMD. The DataComm/Paper 

condition was significantly different from (and had higher average workload ratings than) both the 

Voice/Paper, and the DataComm/MMD conditions.

Figure 25. Inflight workload ratings by position and by condition

Figure 26 shows the median responses for PF and PM workload ratings for surface (taxi) operations 

occurring in both arrival and departure scenarios. Both PF and PM rated workload (Table 51 and Table

55) during surface taxi operations significantly different among display conditions (PF: 

2

2(3)=43.603, 

p< 0.001; PM: 

2

(3)=34.875, p< 0.001). Post-hoc comparisons show the same patterns as that of inflight 

ratings (Table 51and Table 54). For PF, the DataComm/Paper condition appeared to have significantly 

higher ratings than all other conditions. PM rated workload for flight operations as significantly different 

between Voice/Paper and both DataComm/Paper and DataComm/Route; where Voice/Paper mean ranks 

are lower than either of these other conditions (Table 46 and Table 54). PM ratings for inflight workload 

for the DataComm/Route condition were not only significantly higher than those for Voice/Paper, but 

also for DataComm/MMD. The DataComm/Paper condition was significantly different from (and had 

higher average workload ratings than) both the Voice/Paper, and the DataComm/MMD conditions. With 

PF and PM ratings combined, there were no significant differences in workload among display condition 

for any of the tested altitude bands in either arrival or departure operations (Table 58 and Table 59).
48

Figure 26. Surface workload rating by position and by condition

Regardless of the scenario segment, display condition, or role of the respondent, workload ratings 

were rarely extreme. The PF experienced higher workload in the flying portions of the arrival scenarios 

than did the PM. While the patterns differed, depending on whether the rater was PF or PM, the pattern 

for each type of crew member‟s ratings as affected by display conditions was the same for in flight and 

surface operations. For PF, in all cases, workload was rated highest for the DataComm/Paper condition; 

but interestingly, there were no significant differences between the Voice/Paper condition and when 

DataComm was augmented with the MMD, and Route. For PM, the Voice/Paper condition appeared to 

induce lower workload than either the DataComm/Paper or the DataComm/Route conditions, but did not 

significantly differ from the DataComm/MMD condition. The DataComm/MMD condition was also 

rated as having lower workload than either the DataComm/Paper or DataComm/Route conditions. 

Regardless of operation or crew role, the Voice/Paper and the DataComm/MMD conditions are never 

associated with significantly greater workload conditions than the DataComm/Paper and 

DataComm/Route conditions, and the DataComm/MMD condition was associated with the lowest 

workload ratings of all the DataComm conditions for PM.

4.4.2 Post-Scenario Ratings on Situation Awareness

SA scores were obtained for both the inflight and surface/taxi operations of the scenarios (inflight 

assessments were only available for arrivals). Full results are in Appendix O, Section O.2. The SART 

technique results in a score that can range from 13 (highest SA) to -5 (lowest SA). SART ratings were 

collected for inflight operations (only during arrival scenarios), arrival surface operations only, and 

departure surface operations only. Binomial tests were conducted for both PF and PM, and results 

indicate that the preponderance of the data for both pilot roles and for each SART measure favored the 

high end of the scale, demonstrating high SA on the whole. 
49

PF and PM SART ratings significantly differed for inflight scenario segments (

2

(1)=16.341, p<0.001)

(Figure 27), surface/taxi operations in arrival scenarios (

2

(1)=4.450, p=0.035) (Figure 28), but not 

surface/taxi operations in departure scenarios (

2

(1)=0.872, p=0.351) (Figure 29). Both PF 

(

2

(1)=10.342, p=0.016) and PM (

2

(1)=15.459, p=0.001) SART scores significantly differed by 

condition for surface/taxi operations during arrival scenarios. Dunnett‟s C statistics indicate that for PF, 

the scenarios with a Voice/Paper condition had ratings that were significantly higher than scenarios with 

the Data Comm/Paper condition. For PMs, the ratings for the Voice/Paper condition were higher than all 

Data Comm conditions. Analysis across all conditions did not differentially affect SART ratings for the 

flight segments for either PF (

2

(1)=2.723, p=0.436) or PM (

2

(1)=5.205, p=0.157), or for the surface/taxi 

operations in departure scenarios (PF: 

2

(3)=2.982, p=0.394; PM: 

2

(3)=1.875, p=0.599). 

Figure 27. SART ratings for inflight operations by condition

Figure 28. SART ratings for only surface arrival operations by condition
50

Figure 29. SART ratings for only surface departure operations by condition

In summary, results from the post-scenario questionnaire indicate reduced SA due to Data Comm. 

During surface/taxi operations in arrival scenarios, PMs experienced significantly higher SA when using 

the Voice/Paper condition than any other Data Comm condition. PFs showed the same SA reduction

from Voice/Paper only over the Data Comm/Paper. During departure, SA differences were not shown. 

However, post-test debriefing comments seem to indicate that the crews still considered the SA 

acceptable regardless of communication modality, display methodology, or the altitude at which Data 

Comm messages were given to the crew. SA scores were generally high, and PM rated SA higher during 

flight operations and during surface/taxi operations in arrival scenarios than PF. Experimental conditions 

did not differentially affect SA ratings for either crew role during the flight scenario segments.

4.4.3 Post-Scenario Ratings on Acceptability

Appendix D, Section D.5 contains the nine questions on the post-scenario survey that addressed the 

acceptability of different aspects of the conditions, and some of these were only relevant for some 

conditions. The rating scale for all nine questions was weighted so that 1 represented an improvement or 

very high acceptability, a 4 represented acceptable or no change to current operations, and a 7 represented 

an operationally unacceptable or unsafe condition. Full results are in Appendix O, Section O.3.

Question 1: Did the display of the OWNSHIP POSITION on the navigation display make the taxi 

clearance easier to understand and to carry out? (1 – easier to understand, 7 – not easier to understand)

Mean and Standard Deviation values are listed in Table 69, differences by crew position in Table 78, 

differences by Condition in Table 79, and the Binomial Test in Table 88. Both PFs and PMs

overwhelmingly stated displaying the ownship position on the ND made taxi clearances easier to 

understand and carry out. This question was only relevant for the Data Comm/MMD and Data 

Comm/MMD+Route conditions. 

Question 2: Did the display of the ROUTE on the navigation display make the taxi clearance easier to 

understand and to carry out? (1 – easier to understand, 7 – not easier to understand)
51

Mean and Standard Deviation values are listed in Table 70, differences by crew position in Table 78, 

differences by Condition in Table 79, and the Binomial Test in Table 88. Both PFs and PMs

overwhelmingly stated display of route on the ND made taxi clearances easier to understand and carry out 

(

2

(1) = 0.058, p = 0.809).

(Note: Question 3 was analyzed with #9; please see the end of this Section.)

Question 4: Did you have a sufficient amount of time to respond to the Voice or Data Comm 

transmitted messages? (1 – more than enough time, 7 – not enough time)

Mean and Standard Deviation values are listed in Table 72, differences by crew position in Table 78, 

differences by Condition in Table 79, pairwise comparisons in Table 80, by altitude band in Table 86, and 

the Binomial Test in Table 88. PF ratings were significantly worse than PM ratings when asked whether 

they had sufficient amount of time to respond to the Voice or Data Comm transmitted messages (

2

(1) =

12.639, p < 0.001). Both PF and PM ratings were significantly affected by display condition (PF: 

2

(3) =

27.635, p < 0.001; PM: 

2

(3) = 18.974, p < 0.001). Both PFs and PMs rated the Voice/Paper condition as 

more acceptable than any other condition. Although statistical differences were observed, the PF (mean 

2.5) and PM (mean 2.1) ratings indicate that generally flight crews felt there was enough time for the 

flight crew to respond. Pilots always indicated there was sufficient time in the Voice/Paper condition. 

While the preponderance of ratings indicated sufficient time for Data Comm conditions, there were a few 

ratings where the pilots did not have sufficient time. 

Question 5: Was the amount of head down time required to receive and respond to just the “Expected 

Taxi” Data Comm messages acceptable in this scenario? (1 – minimal increase in Head Down time, 7 –

too much Head Down time)

Mean and Standard Deviation values are listed in Table 73, differences by crew position in Table 78, 

differences by Condition in Table 79, pairwise comparisons in Table 81, and the Binomial Test in Table 

88. The PFs rated the acceptability of the head down time required to receive and respond to “Expected 

D-TAXI” messages significantly worse than the PMs (

2

(1) = 12.159, p < 0.001). Neither PFs nor PMs

showed differences as a function of Data Comm condition (paper, moving map, route). Although 

statistical differences were observed, the PF (mean 3.1) and PM (mean 2.5) ratings indicate that 

operationally the amount of head down time was acceptable to the crew.

Question 6: Was the amount of head down time required to receive and respond to other non-timecritical Data Comm messages acceptable in this scenario? (1 – minimal increase in Head Down time, 7 –

too much Head Down time)

Mean and Standard Deviation values are listed in Table 74, differences by crew position in Table 78, 

differences by Condition in Table 79, pairwise comparisons in Table 82, and the Binomial Test in Table 

88. The PFs rated the acceptability of the head down time required to receive and respond to other nontime-critical Data Comm messages (e.g., frequency changes or new altimeter setting) significantly worse 

than the PMs (

2

(1) = 24.162, p < 0.001). Neither group showed differences as a function of Data Comm 

condition (paper, moving map, route) (PF: 

2

(2) = 1.822, p = 0.402); PM: 

2

(2) = 0.556, p = 0.757). 

Although statistical differences were observed, the PF (mean 3.1) and PM (mean 2.2) ratings indicate that 

operationally the amount of head down time was acceptable to the crew.

Question 7: Overall, was the communications mode (Voice or Data Comm) for receiving Expected 

Taxi and Taxi clearances acceptable during this scenario? (1 – completely acceptable, 7 – completely 
52

unacceptable) [NOTE: this question was presented to the subjects only during Data Comm scenarios]

Mean and Standard Deviation values are listed in Table 75, differences by crew position in Table 78, 

differences by Condition in Table 79, pairwise comparisons in Table 83, by altitude band in Table 86, and 

the Binomial Test in Table 88. Figure 30 presents a histogram of ratings regarding the acceptability of 

using Data Comm to receive Expected D-TAXI and D-TAXI messages. Overall results indicate high 

acceptability for Data Comm to be used to issue taxi route clearances. The PF (mean 2.7) and PM (mean 

2.0) ratings indicate PFs rated overall acceptability of the Data Comm for receiving Expected D-TAXI

and D-TAXI clearances statistically, but not operationally, significantly worse than PM ratings. 

However, the display conditions within the PF and PM ratings indicate no statistical significance. Pilots‟

ratings of Data Comm use in a busy terminal area were heavily skewed in the acceptable range of the 

scale. The few unacceptable ratings that occurred were predominately in the Data/Paper condition.

Figure 30. Data Comm acceptability rating by condition

Question 8: How much operational risk was introduced by the communication mode (Voice or Data 

Comm) used during this scenario? (1 – extremely low risk, 7 – extremely high risk)

Mean and Standard Deviation values are listed in Table 76, differences by crew position in Table 78, 

differences by Condition in Table 79, pairwise comparisons in Table 84, by altitude band in Table 86, and 

the Binomial Test in Table 88. Overall, the PFs rated operational risk higher than the PM, and display 

conditions affected both PF and PM ratings. For the PFs, ratings indicated that more operational risk was 

assumed when operating in the Data Comm/Paper condition than the Data Comm/MMD+Route

condition, however the difference was not considered operationally significant. PM ratings did not 

significantly differ by display condition. Although statistical differences were observed, the PF (mean 

2.8) and PM (mean 2.2) ratings indicate that operationally Data Comm is considered low risk by the crew. 

These post-scenario ratings were not correlated to post-experiment comments provided by the pilots, 

which could offer insight into why the distribution of responses varied based on condition (see chart #8 in 

Appendix O.3). However in the list of post-experiment comments from pilots (Section 4.6), there are 

several comments that indicate most pilots favored the use of Data Comm in general, with several specific 

instances where Data Comm should not be used. It is postulated (no analysis conducted) that many of the 

high operational risk ratings were due to one or two specific events within a scenario, and not meant to 

indicate the use of Data Comm in general. 
53

Question 3: Did you have confidence that the taxi route was accurately depicted based on the Data 

Comm ATC instruction? (1 – confident the route was accurate, 7 – not confident the route was accurate)

Question 9: Was there a point at which you did not feel that the transmitted taxi instructions were 

accurate? (1 – the message was accurate, 7 – did not feel the message was accurate)

Mean and Standard Deviation values are listed in Tables 71 and 77, differences by crew position in 

Table 78, differences by Condition in Table 79, pairwise comparisons in Table 85, by altitude band in 

Table 86, and the Binomial Test in Table 88. PFs and PMs did not differ significantly on rating their 

confidence that the taxi route was accurately depicted on the ND based on the Data Comm ATC 

instruction. They differ on their ratings as to whether at some point the transmitted taxi instructions were 

not accurate. PF ratings were on average higher than PM ratings. Condition (map or route) did not 

significantly affect either PF or PM ratings as to whether the taxi instructions may be inaccurately 

presented. Although statistical differences were observed, the PF (mean 1.7) and PM (mean 1.4) ratings

indicate that operationally the Data Comm messages were believed to be accurate by the crew.

In summary, the overall ratings indicated an acceptability of Data Comm by flight crews in all 

conditions. Where statistically significant differences for some of the acceptability questions were 

demonstrated across conditions, means for both PFs and PMs were well below the operationally 

acceptable rating on the scale (set as the mid-point of 4). PFs rated overall and several specific 

acceptability questions lower than PMs. In particular, PFs were less likely to indicate that there was 

sufficient time to respond to the message, that heads down time was appropriate, and that, overall, 

communication modality was acceptable and that operational risk was higher. Display conditions were 

distinguished only in ratings of time sufficiency, operational risk. Both PF and PM indicated that the 

Voice/Paper condition was most efficient in terms of time available to respond to messages, PFs found 

more operational risk in the Data Comm/Paper condition than the Data Comm / MMD+Route condition.

4.5 Post-Experiment Questionnaire Results

4.5.1 Post-Experiment Ratings on Workload Comparison

The first section of the post-experiment survey asked the subject pilots to compare the perceived 

workload of the four conditions to the other conditions. In terms of workload rating comparisons (Figure 

31), the results show a preference for DataComm/Paper compared to Voice/Paper, for DataComm/MMD 

compared to DataComm/Paper, and for DataComm/MMD+Route compared to any other display 

condition. The PF and PM ratings indicate that the Voice/Paper condition was the least preferred in terms 

of workload, effectively rating that condition as the highest workload. 
54

Figure 31. Display preference for lowest workload by condition

ANOVA showed no significant effect of crew role by condition interaction (F=0.379, p=0.768,) or 

main effect of crew role (F=0.030, p=0.862), but did indicate a significant effect of condition (F=272.309, 

p<0.001) (Table 90). Tukey HSD post-hoc comparisons among conditions revealed significant 

differences among all pairs (p≤0.001). 

4.5.2 Post-Experiment Ratings on Situation Awareness Comparison

The second section on the post-experiment survey asked subject pilots to compare their perceived SA 

between the four conditions (Appendix E.2, results in Appendix P.2). Figure 32 shows that the results for 

SA mirror those for workload. In terms of SA, both the PFs and PMs considered the Voice/Paper 

condition to be least preferred, DataComm/Paper more preferred, and preference increased with the 

addition of the MMD and again with the addition of the Route. 

Figure 32. Display preference for highest situation awareness by condition

ANOVA showed no significant effect of crew role*condition interaction (F=1.543, p=0.210) or main 

effect of seat (F=0.038, p=0.847), but did indicate a significant effect of condition (F=777.067, p<0.001)

(Table 96). Tukey HSD post-hoc comparisons among conditions revealed significant differences among 

all pairs (p≤0.001).
55

4.5.3 Post-Experiment Ratings on Acceptability of Expected D-TAXI Messages

The third section asked pilots to rate the acceptability of controllers sending Expected D-TAXI 

messages and the acceptability of flight crews responding to Expected D-TAXI messages by condition 

and by altitude (Appendix E.3, results in Appendix P.3). Responses to the acceptability of receiving 

Expected D-TAXI messages from controllers did not vary significantly by crew position (PF or PM) or 

by display methodology (paper, MMD, MMD+Route). On average, the 22 subject pilots responded that it 

was acceptable for controllers to send Data Comm messages to the flight crew in a busy terminal area

with the exception of the time between the Final Approach Fix (FAF) and above 80 knots during landing 

roll-out. A list of the number and percentage of crews who responded it is acceptable for a controller to 

send Data Comm messages is as follows:

When would it be acceptable for a controller to send an Expected Taxi clearance via Data Comm?

 above 10,000 feet MSL: 22 of 22 100%

 between 10,000 feet and Final Approach Fix: 18 of 22 82%

 between FAF and below 80 knots on roll-out: 3 of 22 14%

 during taxi or surface operations: 21 of 22 95%

A parallel question was asked regarding when the flight crew thought it would be acceptable for the 

crew to be expected to respond to an Expected D-TAXI message. Due to a paperwork error, the first twoperson crew was not asked this question. Responses to the acceptability of flight crew responding to Data 

Comm messages did not vary significantly by crew position or by display methodology. A list of the 

number and percentage of crews who responded they would respond within two minutes is as follows:

When would the flight crew respond to the Expected Taxi message within 2 minutes?

 above 10,000 feet MSL: 20 of 20 100%

 between 10,000 feet and Final Approach Fix: 12 of 20 60%

 between FAF and below 80 knots on roll-out: 1 of 20 5%

 during taxi or surface operations: 18 of 20 90%

4.5.4 Post-Experiment Ratings on Crew Coordination

The fifth section asked subject pilots to compare their perceived effective crew coordination between 

the four conditions (Appendix E.5, results in Appendix P.5). [Note: the fourth section pertained to Trust, 

and those questions and results are presented in Section 6 of this document.] Figure 33 shows preferences 

increase from Voice to Data Comm conditions, with increasing preference for additional display 

methodology (MMD and Route) with Data Comm conditions.
56

Figure 33. Display preference for effective crew coordination by condition

ANOVA showed no significant effect of crew role by condition interaction (F=0.562, p=0.642) or 

main effect of crew role (F=0.032, p=0.859), but did indicate a significant effect of condition 

(F=1915.420, p<0.001) (Table 104). Tukey HSD post-hoc comparisons revealed significant differences 

among all pairs (p≤0.001). 

4.5.5 Post-Experiment Summary Questions

The 22 subject pilots were given the opportunity during the post-experiment questionnaire to respond 

to nine open-ended, free-text questions. The subject pilots were not required to answer each question, 

therefore, the number of respondents varies for each question. During the post-experiment verbal debrief 

session, these particular questions generated a lot of discussion which clarified and sometimes changed 

the responses, which made tabulating responses difficult. A list of the responses is given below.

Question 1: To what degree did the scenarios in this experiment accurately simulate a complex, highworkload environment? If not, what was missing?

 17 of 21 respondents stated the experiment accurately simulated a complex, high-workload 

environment.

 Additional comments included the need to allow the use of the auto-pilot, and more radio 

communications during the airborne portion of the experiment scenarios.

Question 2: What is your overall assessment of the potential of communicating clearance updates or 

changes using Data Comm while an aircraft is taxiing or in busy terminal airspace?

 18 of 22 respondents stated it was realistic to use Data Comm to issue clearances or amended 

clearances in a busy terminal area, either airborne or on the surface.

 Additional comments included that clearances were given too close to the new taxiway 

intersection, and one respondent stated the use of Data Comm was not realistic.

Question 3: Should the dotted cyan lines for an “Expected Taxi” clearance include red hold short bars?
57

 17 of 20 respondents stated that a graphical display of an “Expected Taxi” clearance should 

include red hold short bars.

 3 responded that the red hold short bars were not needed for “Expected Taxi” clearances, but 

were necessary for “Taxi” and “Amended Taxi” clearances.

Question 4: Will the solid magenta line for a Taxi clearance on the Navigation Display encourage 

crew members to begin taxiing prior to receiving the Voice message from ATC?

 4 of 20 respondents thought the magenta line of acknowledged taxi clearance was compelling 

enough to cause flight crew to taxi without the required Voice instruction from ATC.

 11 of 20 thought it might cause the crews to taxi but would be offset by training and 

operational procedures.

 5 of 20 thought it would not cause the crews to taxi without the Voice instruction. This was 

supported by researchers who observed that many of crews initiated aircraft movement

without ATC Voice instruction.

 Several pilots suggested adding “Contact ATC on xxx.xx” to the end of the Data Comm 

instruction (undefined number).

Question 5: Was the simultaneous Voice and Data Comm instructions to cross an active runway 

clear? Was there a delay in the FO updating the graphical display on the ND? Was that delay important?

 16 of 19 respondents stated the use of Voice and Data Comm messages to cross an active 

runway while taxiing was clear.

 3 of 19 stated they did not like going head down prior to crossing a runway and that the Data 

Comm message was probably not necessary (however they acknowledged during the verbal 

debrief session that there was a need to have correct displays that matched Voice instructions, 

which was the rationale for both Voice and Data Comm messages).

 Almost all crews noted they did not like going head down prior to crossing the active runway

(undefined number).

Question 6: How would CDTI (Cockpit Display of Traffic Information) impact your workload, SA, 

and acceptability of using Data Comm messages in terminal airspace or surface operations?

 19 of 20 respondents stated the inclusion of CDTI in the cockpit would have a positive 

impact on their workload, SA, and acceptability of Data Comm in the terminal airspace and 

surface operations. Of those 19, several also noted that it might cause less head up time, 

however it would be very useful in low visibility conditions and reducing radio congestion, 

and in general the benefit would outweigh the potential cost.

 1 of 20 respondents stated it would slow operations if the information was too cluttered.

Question 7: Was the use of Voice by the controller for critical or time-sensitive information (such as 

crossing the runway) appropriate and necessary?

 20 of 20 respondents stated the use of Voice communication for critical or time sensitive 

information was appropriate and necessary.

Question 8: Were there any challenges with Data Comm unique to your flight duties as the PF or PM?

 4 of 8 respondents noted that the PM had a significant decrease in head up time and a 
58

significant increase in workload due to Data Comm.

 2 of 8 respondents stated it was difficult for the PF to stay in the information loop while using 

Data Comm, and that it was important to prioritize messages and tasks.

 2 of 8 respondents found it difficult to keep the CDU and ND aligned with the most current 

Data Comm message, and the Data Comm downlink response on Page 2 of the message led 

to less head up time and more errors in the use of Data Comm.

Question 9: Do you have any other comments? Include any unexpected events, operational issues, 

and any problems with the simulator that affected your performance.

 4 of 11 respondents stated they preferred Data Comm, especially when integrated with the 

MMD and route. Of those four, 1 of them stated Data Comm without the taxi route 

graphically displayed would be limited to providing a benefit only in situations where 

language was a barrier.

 4 of the 11 respondents stated Data Comm messages should not be used when time was 

critical, for any safety related information, or for crossing or entering a runway.

 2 of the 11 respondents stated Data Comm would greatly enhance the entire air transportation 

system, and the sooner it was implemented the better.

 1 of the 11 respondents stated there needed to be a way to visually determine the most recent 

clearance, for example using a different font or bold text.

4.6 Verbal debrief comments

Following the written post-experiment questionnaire, a verbal debrief session generally lasting 90 

minutes was conducted. Topics included questions about the concept, clarifications of the training 

program, explanation of the scenarios, a discussion of questionnaire items, and questions from the subject 

pilots. The content of the responses was recorded, but it was impractical to determine the number of 

respondents that concurred with a response other than a general description such as „a few‟, „some‟, 

„many‟, or „most.‟

Given the assumptions of the experiment and what the crews experienced for Data Comm in a 

Segment 2 (2017-2022) environment, the crews made the following comments:

 The flight crew should be able to respond within two minutes to ATC Data Comm Uplink 

messages in the Terminal Area. One minute is the absolute minimum time required.

 The use of Data Comm above 18,000 feet was generally considered okay, and by some (but 

not all) to be somewhat less desirable from 18,000 to 10,000 feet MSL.

 Many of the pilots (but not all) stated between 10,000 feet and the Final Approach Fix all 

Data Comm messages should be limited to only important messages.

 The vast majority of the pilots said Data Comm, and even Voice, should not be used for 

communication between FAF and clear of runway (but not all pilots agreed).

 Many pilots did consider it acceptable to send the taxi route as a Data Comm uplink “Taxi” 

message to aircraft between FAF and clear of the runway if no chime is used.

 Many of the pilots had issues with two messages in different modes to cross active runway 

(timing, priority, etc.). They agreed that an ATC Voice instruction was essential, but most 
59

did not think it was appropriate to go Head Down to acknowledge the Data Comm message. 

An option was to send the Data Comm message without a chime.

 Two different categories of Data Comm messages: important and informational. „Important‟

messages imply a change to the aircraft‟s route and require a timely response by the crew 

(taxi clearance, etc.), and may require the use of an audible chime. „Informational‟ messages 

may not meet both criteria (new altimeter setting, “Expected Taxi” clearance, etc.), and 

would probably not use an audible chime.

 Most crews thought “Expected Taxi” messages were useful.

 Most crews thought “Pushback” and “Start” messages should be combined.

 Some crews recommended that Data Comm “Taxi” message end with “Contact ATC on 

xxx.xx” since a Voice instruction is required to begin moving the aircraft. This would reduce 

the possibility of the crew beginning to taxi the aircraft without the Voice instruction.

 Many crews recommended that the ATC Voice instruction to begin taxiing be given by the 

controller without the need for a Voice request from the flight crew. It did not seem to be 

necessary for the flight crew to request a taxi clearance via Data Comm, then again via Voice.

 All crews except one stated the downlink message response should be on same page as uplink 

message to reduce Head Down time and potential for flight crew error or confusion while 

operating the Data Comm equipment.

 A need was identified for a method to handle outdated messages and displays.

 “Expected Taxi” message should include hold short instructions in the text message and red 

hold short bars when displayed graphically on the ND.

 One pilot suggested the words “Taxi Route” be used when issuing a taxi route message via 

Data Comm and “Cleared to taxi” when issuing a Voice instruction. The subtle wording 

difference may more accurately reflect NextGen Data Comm operations, and help prevent the 

flight crews from inadvertently beginning to taxi after receiving a Data Comm message. 

4.7 Operational errors

Several operational errors by the flight crew were observed, all but one of them occurred during the 

Trust scenarios described in Section 6. However, none of these errors can be solely attributed to the use 

of Data Comm in a terminal area. Further research will be needed to clarify the impact of Data Comm 

use to the frequency and magnitude of operational errors by flight crew.

The operational errors observed during the two Trust off-nominal scenarios are listed below. 

 6 of the 11 crews (55%) failed to correctly identify an incorrect D-TAXI clearance after 

clearing the runway and taxiing to the ramp during an arrival DataComm/MMD+Route 

scenario. The final clearance included a taxiway previously identified as closed for debris.

 7 of the 11 crews (64%) failed to correctly identify that the different runway given in their 

final D-TAXI clearance during a departure DataComm/MMD scenario was too short for 

takeoff. The runway had previously been identified as shortened due to construction.

The one operational error not during a Trust scenario occurred while taxiing for departure. The PF 

exceeded 35 knots and departed Taxiway Bravo while turning onto Taxiway Charlie. This error is most 

likely attributed to operation of a simulator versus an aircraft, and not use of Data Comm.
60

5 Synthesis of Results

Members of the FAA and NASA Data Comm Airside team developed the experiment hypotheses and 

design, and additional FAA requests for data analysis were received after NASA gave approval to 

proceed for this experiment. Therefore, the data and analysis in this section summarize the experiment 

results in a way to meet the direct requests of the FAA customer.

5.1 Impact of Communication Modality on Flight Crew in the Terminal Area

The first study (S1) investigated the effect of communications modality while using a paper airport 

diagram on the acceptability of Data Comm (Section 3.2 and 3.4). Hypothesis 1 (Section 3.1) was:

o Pilot workload and situation awareness will differ significantly between Voice and Data 

Comm communication mode.

Results and statistical comparisons between the Voice/Paper and DataComm/Paper conditions indicate 

the following:

 No statistically significant difference of NWS PSD (an indicator of PF physical workload) was 

observed across the Voice/Paper and DataComm/Paper conditions. (Section 4.2.1)

 Taxi speed, an indicator of PF situation awareness (and therefore related to acceptability), showed 

a statistically significant yet operationally slight (of the order of 2.0 knots) increase on arrivals, 

and a statistically significant yet operationally slight (again, of the order of 2.0 knots) decrease on 

departures in the presence of Data Comm. (Section 4.2.2)

 Statistically significant more head down time for both crew roles existed in all altitude bands, in 

Data Comm/Paper compared to Voice/Paper. This increase in head down time was not deemed 

operationally unacceptable by the crews, nor was it reflected in workload or SA preference 

ratings. (Section 4.3.2)

 There was no statistically significant difference in PF head up time while taxiing when comparing 

Voice/Paper condition and Data Comm/Paper condition, in either arrival or departure scenarios. 

The PM did spend statistically significantly less head up time in Data Comm/Paper compared to 

the Voice/Paper condition. (Section 4.3.3)

 Post-scenario workload and SA ratings remained generally favorable (low or adequate workload, 

upper third of SA scale) in both Voice and Data Comm modalities. For both PF and PM, both 

while in flight and in surface operations, the Voice/Paper and the DataComm/MMD conditions 

appeared associated with lower workload ratings. SA was statistically higher for Voice/Paper 

than any other condition. (Section 4.4.1 and 4.4.2)

 Statistically significant differences were reported by both PFs and PMs with respect to the 

introduction of operational risk of using Data Comm in the terminal area; however, the mean 

rating of all crew members found the risk to be operationally acceptable. (Section 4.4.3)

 Post-experiment workload comparison and SA comparison preference ratings improved in the 

presence of Data Comm. This is opposite of the post-scenario results. Analysis of results based 

on scenario run order show improved ratings with increased exposure to the operation. Other 

possibilities for the difference include “experimenter‟s bias” (where experiment subjects tend to 

rate new technology higher to validate the researcher‟s work) or the subjects believe there is 

potential for the technology and rate it higher during post-experiment questionnaires. (Section 
61

4.5.1 and 4.5.2)

 Post-experiment questionnaire results indicate acceptance of Data Comm in the terminal area; 

however, Voice should be used for time-critical or safety-related communication. (Section 4.5.3

and 4.6)

In summary, pilot workload and SA did differ significantly between Voice and Data Comm 

communication modes. Workload and SA improved in the presence of Data Comm in post-experiment 

questionnaire ratings, and was slightly reduced in post-scenario questionnaire ratings.

5.2 Impact of Display Methodology on Flight Crew in the Terminal Area

The second study (S2) investigated the effect of display methodology while using data 

communications on the acceptability of Data Comm (Section 3.2 and 3.4). Hypothesis 2 (Section 3.1) 

was:

o Pilot workload and situation awareness will differ significantly between display modes 

when using Data Comm. 

Results and statistical comparisons between the DataComm/Paper, DataComm/MMD, and 

DataComm/MMD+Route conditions indicate the following:

 No statistically significant difference existed in mean response times to Data Comm messages by 

display methodology. (Section 4.1.1)

 No effect on NWS PSD (an indicator of PF physical workload) was observed across experimental 

conditions on arrivals, but there was a significant increase in NWS PSD in departures with 

respect to display methodology. NWS PSD increased going from paper to MMD, and from 

MMD to loadable routes. (Section 4.2.1)

 Taxi speed showed a statistically significant (of the order of 4.0 knots) increase during arrival

scenarios, between the DataComm/MMD to DataComm/MMD+Route condition. There were no 

statistically significant differences during departure scenarios, regardless of display methodology. 

(Section 4.2.2)

 In general, there was no statistically significant difference for either PF or PM head up time 

across the three display conditions. The one exception was both PF and PM had statistically 

significant less head up time in the Data Comm/Paper condition compared to the other two 

display conditions. (Section 4.3.1)

 More head down time, in both crew roles, existed in the low (5K-7K ft) altitude band in the 

presence of Data Comm, proceeding from paper, to MMD, and MMD+Route display 

methodology. This increase in head down time was not deemed unacceptable by the crews, nor 

was it reflected in workload or SA preference ratings. (Section 4.3.2)

 Post-scenario workload ratings by PFs and PMs remained favorable (75% of responses scored 

“3” or less) in all Data Comm scenarios, regardless of display methodology. PF workload was 

statistically higher than PM workload during Data Comm (except for surface operations when in 

Data Comm/MMD+Route condition), though still in the adequate region. (Section 4.4.1)

 Post-scenario ratings by the PFs scored the flight operation workload significantly different

among the display conditions (worst to best of DataComm/Paper, DataComm/MMD, 

DataComm/MMD+Route). The PMs did not rate inflight operations significantly different. Both 

PF and PM rated surface taxi operations significantly different among the display conditions (PF 
62

worst to best of DataComm/Paper, DataComm/MMD, DataComm/MMD+Route, and PM of 

DataComm/Paper, DataComm/MMD+Route, and DataComm/MMD). (Section 4.4.1, Table 46, 

Table 47)

 Post-scenario SA ratings were statistically higher for Voice/Paper than any Data Comm 

condition. (Section 4.4.2)

 Post-scenario SA ratings remained favorable (upper third of SART scale) in all Data Comm 

scenarios, regardless of display methodology. Display condition did not have a statistically 

significant impact on SA ratings of either the PF or PM. (Section 4.4.2)

 Post-experiment workload comparison and SA comparison preference ratings, for both airborne 

and surface operations, improved when the MMD was available, and again, when implementing 

loadable routes on the MMD. (Section 4.4.4 and 4.4.5)

In summary, pilot workload and SA differed significantly by display methodology when using Data 

Comm. Display methodology did not have a statistically significant impact on SA ratings for either the 

PF or PM.

5.3 Acceptability of Data Comm Use to Flight Crew in the Terminal Area 

This section collates analyses that addresses whether the flight crew found the use of Data Comm in a 

busy terminal area to be acceptable. This was defined in Hypothesis 3 as:

o Pilots will rate the Data Comm used within this experiment as operationally acceptable.

Results and statistical comparisons indicate the following:

 Mean response time to Data Comm messages was 20.7 seconds, with over 95% of the responses 

occurring under one minute and 97% occurring under two minutes. A statistical difference of 

approximately six seconds was found between the two lowest Conditions (Arrival/MMD and 

Arrival/MMD+Route) and the highest condition (Departure/MMD+Route); however, the 

difference in response times is not considered operationally significant. (Section 4.1.1, 4.1.2)

 Approximately 3% of the Data Comm messages were not responded to within two minutes. 

Researcher observation and flight crew comments during the post-experiment debrief session 

indicated these late responses were due to the crew believing they had responded using Page 2 of 

the Data Comm message, or forgetting to acknowledge. In all cases, the message was read and 

briefed to the other crew member, and the crew had time available to respond. (Section 4.1.2)

 A statistically significant difference of the mean message response time of approximately four 

seconds was found between Data Comm arrival and departure scenarios, however, this is not 

considered operationally significant. (Section 4.1.1)

 A statistically significant difference between mean response time to a frequency change message 

compared to an Expected Taxi message of ten seconds was found, and this could be considered 

operationally significant. (Section 4.1.1 and 4.1.2)

 Post-scenario acceptability ratings remained high for all Data Comm scenarios. On a scale of 1 

(completely acceptable) to 7 (completely unacceptable), the mean rating for all Data Comm 

conditions by the PF was 2.7, and by the PM was 1.9. (Section 4.4.3, Table 79)

 During post-scenario questionnaires, crews indicated that Data Comm during approaches above 

10,000 feet MSL would be acceptable. 82% of the crews reported that Data Comm messages 
63

from 10,000 feet to the Final Approach Fix could be sent by controllers; however only 60% felt 

the crews would always be able to respond within two minutes. Post-experiment questionnaire 

responses indicated 82% of the crews felt that the use of Data Comm as the communication 

modality was acceptable as experienced in this high traffic density and high workload terminal 

area environment. (Section 4.4.6 and 4.4.8)

 Post-experiment workload comparison (Section 4.4.4) and SA comparison (Section 4.4.5) 

preference ratings were generally high, and improved in the presence of Data Comm.

 All crews indicated that Data Comm should not be used between the FAF and 80 knots during 

landing roll-out. However, many crews also stated Data Comm messages during that time would 

be acceptable if they were not accompanied by a chime and the flight crew was not expected to 

immediately respond to the message. (Section 4.4.6)

 Crews indicated the use of Voice communication to cross an active runway while taxiing was 

necessary and appropriate. It was also stated the simultaneous use of a Data Comm message was 

not appropriate since it caused crew members to go head down at a critical time. Use of a Data 

Comm message without a chime was considered an acceptable alternative. (Section 4.4.8 and 

4.4.9)

In summary, pilots rated Data Comm (as implemented within this experiment) as operationally 

acceptable in a complex and busy terminal area environment.
64

6 Exploratory Study: Rare Event Scenario and Trust Assessment

The purpose of the rare event scenarios was to test the impact of incorrect Data Comm messages on 

the flight crew‟s trust in the accuracy of the information. Errors in data transmission can undermine a 

crew‟s confidence in automation, as well as their faith in the integrity of the information. This section 

discusses the two Data Comm scenarios that captured the pilot‟s perceptions of trust (these two runs and 

the data collected are not part of the text and conclusions in the remainder of this text). The summary of 

data collected and analyzed during these two runs is presented here, with complete data in Appendix P.

6.1 Rare Event Scenarios

Research on trust in automation has typically been tailored to an experiment in which there is a 

contrived failure within a system which must be detected, diagnosed and resolved, either through the aid 

of automation or through human intervention. After the scenarios, subjects are queried on their faith in 

the system and whether or not they perceived that faith to present an operational risk. Additionally, 

subjects were asked questions pertaining to their perception of the reliability and dependability of the 

automation, personal attachment to the automation, and their confidence in the automation to perform 

routine tasks typically performed by the human operator(s). [37][38][39]

For this experiment, two scenarios were created with events that, in the course of flight operations, 

might cause pilots to lose confidence in the electronic delivery of a taxi instruction. Subjects were 

advised prior to the simulator portion of training, that during the course of the experiment, there would be 

a potential for human error, with respect to clearances provided (just as in real-world scenarios).

The first rare event occurred in the final (fourth) training scenario, and was a Norwich Three arrival 

operation with Data Comm and MMD+Route display. The ATIS noted that taxiway Alpha-1 (A1) was 

closed due to debris. Upon landing, the flight crew received a D-TAXI message to turn off the active 

runway and taxi to the terminal via N-B-L-A, a feasible taxi route. This clearance was amended to taxi 

via B-A1-A, an infeasible taxi route since A1 was closed for debris. As soon as the crews recognized the 

inappropriate clearance, the scenario was ended. If they proceeded despite the inappropriate clearance, 

they would be able to see that there was a baggage cart placed near the center of the taxiway, and the 

scenario was ended when they stopped the aircraft and queried ATC. Had the crew accepted the 

clearance and not seen the cart, a ground collision would have occurred between the aircraft and baggage 

cart. Of the eleven crews, six (55%) failed to process the inappropriate clearance (but stopped upon 

noticing the baggage cart). Of those six crews, one taxied past the cleared taxiway (primarily due to a

delay in responding to the Data Comm amended taxi clearance message).

The second scenario occurred after the sixteen data collection runs for each crew, and was a departure 

scenario with Data Comm and MMD display. This scenario was a departure scenario, initially to Runway 

15R at KBOS, then changing to Runway 27. ATIS messages delivered before and after the runway 

change indicated that the first 2,000 ft. of Runway 9 were closed due to maintenance (equivalent to the 

last 2,000 ft. of Runway 27). Two feasible Expected Taxi clearances were sent to the flight crew to taxi

to Runway 15R, an appropriate runway for takeoff. The final D-TAXI clearance was provided to Runway 

27, an inappropriate runway for takeoff due to insufficient length for the aircraft weight. The scenario 

was ended either when crew narrative comments indicated they recognized that the scenario presented a 

potentially unsafe takeoff situation, or when they taxied on to the runway for takeoff not having processed

the potentially unsafe situation of a shortened runway. Had the crew accepted the takeoff clearance for 

the shortened runway at that aircraft weight, a serious mishap would most likely have occurred since the 

takeoff distance required exceeded the 5000 feet available on Runway 27. Seven of the crews (64%) 
65

accepted the unsafe clearance on to the runway for takeoff. The remaining four crews correctly assessed 

this as a potentially unsafe situation and ended the scenario.

In all cases, post-experiment recorded comments indicated that the rare event scenarios were 

considered possible, and at least somewhat realistic. One crew member indicated he had been a member 

of a flight crew during a real world operation of essentially the departure scenario rare event.

6.2 Trust Questions and Results

Trust questions were incorporated into both the post-scenario and the post-experiment questionnaires. 

The post-scenario questions focused on accuracy of the loading of the route onto the moving map display, 

time constraints for responding to the taxi instructions regardless of the modality (Voice versus Data 

Comm), and whether or not the implementation of Data Comm posed an operational or safety risk. The 

post-experiment questionnaire targeted other constructs of confidence, verification, detection, integrity, 

reliability, and elements of uncertainty – ambiguity and completeness.

The subject pilots were asked to compare their perceived Trust among the four conditions (Appendix 

P.4), and a summary of the results are shown in Figure 34. In this comparison, both the PF and PM 

preferred the Data Comm / MMD+Route condition.

Figure 34. Display preference for highest trust by condition

Trust Question 1: Overall, how confident were you that the data linked message was properly loaded 

into the FMS and then graphically displayed on the ND? (1 – Complete Confidence; 7 – No Confidence)

 Summary of results from Appendix P.4, question #1, is shown in Table 33.

 Subject pilots generally had a high level of confidence in the loading of the Data Comm message 

so that it would create a graphic of either the ownship on the airport or the combination of the 

ownship and a route map. Median scores for all crews was 2 (N=22, SD=0.99).

Table 33. Confidence Data Comm message displayed properly on ND

N Minimum Maximum Median Standard Deviation

Trust 1 22 1 4 2 0.99
66

Trust Question 2: How often did you verify the accuracy of the data link taxi instructions? (1 – All 

The Time; 7 – None Of The Time)

 Summary of results from Appendix P.4, question #2, is shown in Table 34.

 The median rating for verifying the accuracy of Data Comm instructions among subjects was 1 

(N=22, SD=1.35). This suggests that pilots verified the Data Comm message content by analyzing 

the airport diagram or the map. It was not clear whether or not this was due to mistrust of the 

information, creation of a mental map, or due to company policy. Verification could be linked to 

lack of trust, but it could also be a policy of the flight crew and/or organization to confirm

information. 

 Comment: Some pilots indicated that they didn‟t verify the instructions because they presumed it 

was accurate or the other crew member had compared the instruction with the moving map. 

Table 34. Verification of D-TAXI instruction feasibility

N Minimum Maximum Median Standard Deviation

Trust 2 22 1 6 1 1.35

Trust Question 3: How often did you verify the taxi route displayed on the Navigation Display with 

the Data Comm message on the CDU? (1 – All The Time; 7 – None Of The Time)

 Summary of results from Appendix P.4, question #3, is shown in Table 35.

 The median score among subjects for verification was 1 (N=22, SD=0.95). This indicates that 

most of the time pilots verified the taxi route on the ND with the Data Comm message on the 

CDU. 

 Comments: The question may not have captured the intended purpose of this inquiry to determine 

if pilots trusted the automation. Research suggests that verification indicates a lack of trust.

However, when asked, two of the pilots stated it was company policy to always verify 

information. Other pilots indicated that they didn‟t verify the instructions against the moving map 

because they presumed it was accurate or they believed that the other crew member had compared 

the instruction with the moving map. 

Table 35. How often was D-TAXI instruction verified for correct display on MMD?

N Minimum Maximum Median Standard Deviation

Trust 3 22 1 5 1 0.95

Trust Question 4: How long did it take you to notice the data link message was incorrect? (1 – Did 

Not Notice It Was Incorrect; 7 – Noticed Immediately)

 Summary of results from Appendix P.4, question #4, is shown in Table 36.

 The median score was 5 (N=19,SD=2.07). This indicated that most pilots felt they correctly

noticed when the Data Comm message was incorrect (either the closed taxiway, or construction on 
67

a runway that made it too short for takeoff). This is inconsistent with their performance; given 

that most crews did not detect the inappropriate clearances (the reason for this discrepancy is not 

known). Three subjects did not respond to the question, with two indicating it was not applicable 

to any of the scenarios they experienced. Observations of flight crews demonstrated a range of 

behaviors and attention towards information provided. Two first officers turned the ATIS off after 

getting specific information about the wind, temperature, dew point, altimeter and active runway. 

These crews missed the rare event information that would affect their taxi route or takeoff.

Table 36. How long did it take to notice data link message was incorrect?

N Minimum Maximum Median Standard Deviation

Trust 4 19 1 7 5 2.07

Trust Question 5: The method for receiving, uploading, and carrying out air traffic taxi instructions 

via Data Comm has integrity, is reliable, is incomplete, and is ambiguous. (1 “No” to 7 “Yes”). 

 Summary of results from Appendix P.4, question #5, is shown in Table 37.

 During analysis, the scores were analyzed on a seven point scale to capture moderate perceptions. 

In addition, ratings for incompleteness and ambiguity were reverse coded to normalize values for 

data analysis. The median score for Integrity was 6 (N=22, SD=0.90) indicating a high sense of 

integrity for Data Comm in general, and pilots felt the system was generally reliable with a median 

score of 6 (N=22, SD=0.79). For incompleteness, pilots felt the system provided nearly complete 

messaging with a median score of 2 (N=22, SD=1.67). Pilots also felt that the system was not

ambiguous, with a median score of 1.5 (N=22, SD=1.19).

 Summarizing the results for Trust Question 5, the subject pilots‟ attitude toward trusting the Data 

Comm system was high considering the constructs of confidence, accuracy, risk, integrity, and 

reliability. Even when anomalies occurred, pilots indicated they would contact ATC for 

clarification. Incorrect information sent via Data Comm did not appear to impact the subjects‟ 

trust in the system.

Table 37. Integrity, reliability, incompleteness, and ambiguity of Data Comm

Trust 5 N Minimum Maximum Median Mean Standard Deviation

Integrity 22 4 7 6 6.05 0.90

Reliability 22 5 7 6 6.05 0.79

Incomplete 22 1 7 2 2.27 1.67

Ambiguity 22 1 5 1.5 1.91 1.19

Overall, this study indicated that Trust was not impacted by errors in the Data Comm message itself. 

Constructs of integrity, confidence, and reliability did not appear to be affected when pilots encountered 

contradictions in instructions caused by incompleteness or accuracy. However, pilots did not feel the 

need to verify the message, indicating a level of complacency with not only the automation but the 

reliance of one pilot on the other for processing information. 
68

7 Conclusion

The FAA worked with NASA Langley Research Center to study the impact caused by the use of Data 

Comm on flight crew during terminal area operations. Crews‟ qualitative comments indicated in general 

an acceptance of Data Comm use in the terminal area as experienced within this experiment, and 

favorable ratings of workload and SA. Qualitative data showed that crews found the decrease in head up

time associated with Data Comm use to be acceptable. There was also consensus in acceptability and 

desirability of employment of a Moving Map Display, particularly in conjunction with loadable routes, in 

the presence of Data Comm. 

In general, there was a desire to limit Data Comm during certain critical phases when it was important 

for the crew to be head up. However, the crews also stated that even in these phases, Data Comm could 

potentially be acceptable if there was an improvement to the ease of responding and a reduction in the 

intrusiveness of chimes and alerts. The two identifiable segments where the majority of crews found the 

use of Data Comm unacceptable were:

1) From the Final Approach Fix to approximately 80 knots during landing rollout 

2) While crossing an active runway during taxi operations

Quantitative results showed that within the scope of this experiment, the use of Data Comm in the 

terminal area was acceptable in terms of perceived workload, SA, and flight technical performance. 

Though statistical differences were identified that favor PF workload in Voice modality and SA in the 

Voice/Paper condition, all PF and PM workload and SA ratings using Data Comm remained acceptable.

Within the scope of this experiment, the use of two minutes as the expected time for Data Comm 

message downlink responses is consistent with the quantitative and qualitative data observed. The 

observed mean response time was 20 seconds, with over 95% of all messages responded to in less than 

one minute. However, debrief comments from all the crews indicated they felt one minute was not quite 

sufficient and two minutes would be significantly better.

The crews rated the Expected Taxi message as useful in both arrival and departure scenarios. Most 

crews also commented that the text instruction should include hold short instructions, and graphical 

displays should include red hold short bars.

Effects of Data Comm during arrival scenarios in the presence of paper airport diagram utilization 

included a no effect on NWS activity and statistically significant, but perhaps not operationally significant

decreases in head up time for the PMs (only), and minor increases in taxi speed. Effects of Data Comm 

employment during departure scenarios in the presence of paper airport diagram utilization included a

statistically significant, but perhaps not operationally significant decreases in head up time for both crew, 

and decreases in taxi speed.

The introduction of route symbology on the MMD while using Data Comm produced a minor increase 

in NWS Rate activity in departures only, and a minor decrease in head up time in the case of the PM for 

arrivals only. Crew preference for MMD employment during both arrivals and departures, particularly 

with loadable routes, in terms of workload and SA, was strong.

Crew head up scan time while inflight was approximately 10%, and during surface operations 

approximately 60% for the PF and 35% for the PM. Head up scan time was impacted by display 

condition.
69

8 Recommendations and Future Research

The following three sections are prioritized lists of recommendations from researcher observations and 

flight crews comments about procedures, avionics, and future research. This Section was created in 

response to discussions with the FAA Data Comm Program representatives to document information that 

was not directly derived from the experiment hypotheses and data collection, and to propose topics for 

future research of flight crew use of Data Comm in a terminal area environment.

8.1 Controller-Pilot Operational Procedure Recommendations

1. Either “Start” or “Pushback” Downlink request should be sent, but not both.

2. The “Taxi” uplink clearance should be sent automatically at some set time after the “Pushback” 

uplink was sent, it should not be required to have the flight crew send a separate downlink request.

3. The “Taxi” uplink clearance message should end with text that states: “Contact GRND on xxx.xx” if 

Voice communication from ATC is required to begin moving the aircraft. This applies when taxiing

from the terminal to the runway and taxiing from the runway to the terminal.

4. Two minutes should be allotted for the flight crew to read, brief, and respond to Data Comm 

messages or take action related to that message when in the terminal area. One minute is the absolute 

minimum.

5. Data Comm implementation strategies that require the crew to look down at a CDU to respond to a 

message when cleared to cross an active runway should be avoided.

6. Implementation for communicating significant airport surface information to flight crews should be 

structured so that the rare events are not likely to occur (such as this experiment‟s rare event scenario 

that closed taxiways and shortened runways). Examples of methodology to support this requirement 

would be linking D-ATIS content to MMD display graphics and requiring runway remaining markers 

to be covered or altered when they are inaccurate for existing conditions.

8.2 Aircraft Avionics Implementation Recommendations

1. Crew should have the ability to respond to a Data Comm message on the same page as the message 

itself. That would reduce workload and the probability of the crew thinking they had acknowledged 

the Data Comm message but had not (this was the root cause for a significant portion of the delayed 

crew responses). On the other hand, one pilot recommended retaining a two page set-up to prevent 

accidental responses, which he thinks are likely especially when the crew is busy and trying to move 

quickly. However, that error never occurred, while almost every crew at least once thought they had 

responded to the Data Comm message and were executing it, yet had not acknowledged it.

2. If one crewmember selects a new Data Comm message of the same type as the other crew member 

currently has displayed on their CDU, the other crew member‟s CDU should automatically display 

the new message as well. This was felt to be important by many subject pilots to reduce the 

possibility of one crew member reviewing and acting on an outdated Data Comm message. This is 

particularly important for Taxi messages, and several times during the experiment the FO 
70

acknowledged an AMENDED TAXI clearance while the Captain still had the original TAXI 

clearance displayed on his CDU. NOTE: Other options include erasing the old message, or including 

text in the page displaying the message saying OUTDATED.

3. The Moving Map Display (MMD) should automatically load the graphical representation of the route 

sent by Data Comm. This would reduce workload and the number of times the crew forgets it is

available. This option may require the option to erase the route.

4. The WILCO/ROGER letters should be removed from the acknowledgement page once the message 

has been acknowledged to make it clear the crew has already responded.

5. The Data Comm interface device would be separate from the interface device for the FMS since the 

CDU is needed for critical navigation tasks.

6. The Start/Pushback/Taxi downlink requests should each take only one button push on one page.

7. “Expected Taxi” message should include hold short instructions, and the display should include hold 

short bars.

8. The word DISPLAY should be used instead of LOAD to display the route on the ND. This will assist 

in differentiating from current Data Comm route clearances that are LOADED into the FMS, which 

changes the aircraft‟s flight path (D-TAXI does not).

9. The flight crew would benefit from the ability to delete or archive specific types of messages (ATIS, 

altimeter, “Expected Taxi”) while retaining others.

8.3 Future Research Issues

1. How can the flight crews‟ degradation of SA due to loss of information from “party-line” Voice

communication be offset by incorporating the use of CDTI (TCAS and ADS-B “In”)?

2. What is the acceptability and impact of conducting more complex NextGen type flight operations that 

require other CPDLC messages, such as trajectory reroutes and speed change messages?

3. What is the impact of allowing flight crews to use the auto-pilot for NextGen procedures?

4. What are other options to crossing an active runway than those explored in this experiment 

(simultaneous Voice and Data Comm messages from ATC)?

5. What procedures and mitigation strategies should exist when the flight crew receives, understands, 

and is executing Data Comm message, but forgets to respond to ATC about the message?

6. How can Data Comm be functionally integrated with D-ATIS to graphically display closed taxiways, 

construction areas, changes to runway length, etc.?

7. How can Data Comm be functionally integrated with other runway incursion prevention devices, such 

as red hold short bars that are controlled by ATC?
71

8. Are there operational issues that would be discovered by utilizing subject controllers and subject 

pilots in the same scenario?

9. When does the flight crew need to uplink both a “Start” and a “Taxi” Data Comm request? Are there

options to have the second message occur automatically after a pre-determined time interval?

10. Would the use of “Route” be more distinct and accurate than “Clearance”? (Example: the Data 

Comm downlink message from ATC would be “Expected Taxi Route”, “Taxi Route”, or “Amended 

Taxi Route”. This differentiates it from today‟s “Clearance” which gives the crew authority to begin 

moving the aircraft, whereas a Data Comm instruction does not.)

11. How important is it that the “Expected Taxi” route be close to the actual “Taxi” route? 

12. What happens if a crew does not respond to an “Amended Taxi” clearance while taxiing, or responds 

after they have passed the new route?

13. Is a Voice call to Ground necessary after the crew acknowledged the Data Comm “Taxi” clearance? 

For example, after an aircraft lands and has acknowledged a Data Comm “Taxi” clearance, does the 

flight crew also need to acknowledge the taxi clearance via voice communication?

14. If the Data Comm message is erased or archived, would/should that erase any taxi routing displayed 

on the ND? 

15. Should specific chime sounds (including no sound) be used to indicate the priority or urgency of the 

message?
72

References

[1] Data Communications National Airspace System Human-in-the-Loop Simulation, Airside Research 

Request, FAA ATO Ops Planning, Air Traffic Systems Concept Development and Validation 

Group, Final Draft Version 1.4, 30 Dec 2008

[2] Reimbursable Interagency Agreement IA1-973 Between FAA and NASA Langley Research Center 

for Enhancement of Aeronautical Research and Technology Development, 10 June 2009

[3] Concept Of Operations for the Next Generation Air Transportation System, Version 3.0, Joint 

Planning and Development Office (JPDO), Washington DC, 1 October 2009

[4] Interoperability Requirements Standard for Aeronautical Telecommunications Network Baseline 1, 

RTCA DO-280B, Vol 1 of 2, June 26, 2007

[5] Interoperability Requirements Standard for Aeronautical Telecommunications Network Baseline 1, 

RTCA DO-280B, Vol 2 of 2, June 26, 2007

[6] Data Communications Human Factors Discussion paper, Segment One, Aircraft/Flight Crew HF 

Challenges, FAA ATO Data Comm HFWG-08, March 28, 2008

[7] Herschler, D., NextGen Human Factors in Data Communications, FAA Human Factors Research 

and Engineering Group (AJP-61), June 1, 2009, Version 6.0

[8] LINK 2000+, Flight Crew Datalink Operational Guidance for LINK2000+, Ver 4.0, 30 Jun 09

[9] Pinska, E., Whiteley, M., LINK 2000+, France Real-Time Simulation Project, Eurocontrol 

Experimental Center, EEC Report No. 395, Sept 2004

[10] D-TAXI Trial Final Report, Version 2.0, EUROCONTROL CASCADE Programme, April 2007

[11] Roeder, M., EMMA Publishable Final Activity Report, Deutsches Zentrum fur Luft und Raumfahrt

(DLR), D013, 17 Sept 2007

[12] Jakobi, J., EMMA Recommendations Report, Deutsches Zentrum fur Luft und Raumfahrt (DLR), 

Document D6.8.1, 2007

[13] Teutsch, J., EMMA2 Validation Comparative Analysis Report (European Airport Movement 

Management by A-SMGS, Part 2), National Aerospace Laboratory of the Netherlands (NLR), 

Document No. 2-D6.7.1, Version 1.0, 2009

[14] Jakobi, J., EMMA2 Recommendations Report (European Airport Movement Management by ASMGS, Part 2), DLR, Document No. 2-D6.7.2, Version 1.0, 2009

[15] Jakobi, J., Teotino, D., Montebello, P., Higher-Level Services of an Advanced Surface Movement 

Guidance and Control System (A-SMGCS), ATC Quarterly, in press

[16] Volker, Stuhlsatz, Air/Ground Data Link Procedures For Flights Within The Area of Responsibility 

of Maastricht-UAC, Eurocontrol, Edition 2.5, 31 July 2008

[17] Waller, M., Lohr, G., A Piloted Simulation Study of Data Link ATC Message Exchange, NASA TP2859, Feb 1989

[18] Harvey, C., Reynolds, M., Pacley, A., Koubek, R., Rehmann, A., Effects of the Controller-To-Pilot 

Data Link on Crew Communication, Wright State University, FAA WJHTC, 46th Human Factors 

and Ergonomics Society Annual Meeting, pages 61 – 65, 2002

[19] Scanlon, C., Know, C., Flight Test Show Potential Benefits of Datalink as Primary Communication 

Medium, SAE Aerospace and Technology Conference, 1990
73

[20] Knox, C., Scanlon, C., Flight Tests With a Data Link Used for Air Traffic Control Information 

Exchange, NASA TP-3135, Sept 1991

[21] van Gent, R., Human Factors Issues with Airborne Data Link, National Aerospace Laboratory of 

the Netherlands (NLR), AIAA Guidance, Navigation, and Control Conference, San Diego, AIAA1996-3855

[22] Rehmann, A., HF Recommendations for Airborne CPDLC Systems: A Synthesis of Research 

Results and Literature, ACT-350, DOT/FAA/CT-TN97/6, June 1997

[23] Lee, K., Sanford, B., Slatterly, R., The Human Factors of FMS Usage in The Terminal Area, NASA 

Ames, AIAA Modeling and Simulation Technologies Conference, New Orleans, AIAA-1997-3804

[24] Prinzo, O., United States Airline Transport Pilot International Flight Language Experiences Report 

2: Word Meaning and Pronunciation, DOT/FAA/AM-10/7, April 2010

[25] Hooey, B., Foyle, D., Andre, A., Parke, B., Integrating Datalink and Cockpit Display Technologies 

into Current and Future Taxi Operations, Proceedings of the AIAA/IEEE 19th Digital Avionics 

System Conference, Philadelphia, PA, Oct 2000

[26] Hellenberg, J., Wickens, C., Pilot Expectancy and Attentional Effects for Hazard Awareness: 

Effects of Data Link Modality on Pilot Attention and Communication Effectiveness, University of 

Illinois, Tech Report ARL-00-7/FAA-00-4, Aug 2000

[27] Mueller, E., Lozito, S., Flight Deck Procedural Guidelines for Datalink Trajectory Negotiation, 

NASA, AIAA ATIO Conference, AIAA 2008-8901, Sept 2008

[28] Rehmann, A., Flight Simulator Evaluation of Baseline Crew Performance with Three Data Link 

Interfaces, FAA Technical Center, DOT/FAA/CT-TN95/19, Sept 1995

[29] Parke, B., Renfroe, D., Kanki, B., Hooey, B., Munro, P., Patankar, K., Foyle, D., The Effects of 

Advanced Navigation Aids and Different ATC Environments on Task-Management and 

Communication in Low Visibility Landing and Taxi, 11th International Symposium on Aviation 

Psychology, Ohio State University, March 2001

[30] Navarro, C., Sikorski, S., Datalink Communication in Flight Deck Operations: A Synthesis of 

Recent Studies, University of Toulouse, The International Journal of Aviation Psychology, pg 361-

376, 1999

[31] Lozito, S., Verma, S., Martin, L., Dunbar, M., McGann, A., The Impact of Voice, Data Link, and 

Mixed Air Traffic Control Environments on Flight Deck Procedures, FAA/Eurocontrol ATM 

Conference, 2003

[32] Prinzel, L., Shelton, K., Jones, D., Allamandola, A., Arthur III, J., Bailey, R., Evaluation of MixedMode Data-Link Communications for NextGen 4DT and Equivalent Visual Surface operations, Air 

Traffic Control Quarterly, Vol 18, Issue 2, 2010

[33] Data Communications Safety and Performance Requirements, RTCA SC-214/EUROCAE WG-78, 

Draft E, 30 April 2009 [Note: this version of the SPR document complies with guidance in RTCA 

DO-264/EUROCAE ED-78A, and used to provide input to RTCA DO-305A published in 2012.]

[34] Roscoe, A. H., Ellis, G. A., A Subjective Rating Scale For Assessing Pilot Workload In Flight: A 

Decade Of Practical Use, Royal Aerospace Establishment, Bedford, UK, 1990

[35] Taylor, R. M., Situational Awareness Rating Technique (SART): The development of a tool for 

aircrew systems design. Proceedings of the AGARD AMP Symposium on Situational Awareness in 

Aerospace Operations. Seuilly-sur Seine:: CP478. NATO AGARD, 1989
74

[36] FAR|AIM 2010, Federal Aviation Regulations and Aeronautical Information Manual, ASA Inc., 

Newcastle, WA, 2010

[37] Lewandowski, S., Mundy, M., Tan, G.P.A., The Dynamics of Trust: Comparing Humans to 

Automation, Journal of Experimental Psychology: Applied, Vol 6(2), June 2000, pg 104-123

[38] Lee, J., Moray, N., Trust, Control Strategies and Allocation of Function in Human-Machine 

Systems, Vol 35 (10), Ergonomics, Oct 1992, pg 1243-1270

[39] Madhavan, P, Wiegmann, D.A. & Lacson, F.C., Occasional Automation Failures on Easy Tasks 

Undermines Trust in Automation, Human Factors: The Journal of the Human Factors and 

Ergonomics Society, 2006

[40] MINITAB Inc., (2005), MINITAB Help v 14.20. Hampton, VA.
Appendix A: Interagency Agreement 75

Appendix A: FAA/NASA Interagency Agreement

A.1 FAA/NASA Interagency Agreement

Once the FAA and NASA leadership agreed in principal to the collaboration, the agreement was 

codified in an Interagency Agreement. The following paragraphs of this section are excerpts from 

FAA/NASA Reimbursable Interagency Agreement IA1-973 [1] and the Airside Research Request [2]. 

NASA Langley Research Center (NASA LaRC) and the Department of 

Transportation/Federal Aviation Administration (DOT/FAA) enter into this Technical 

Direction for the purpose of collaborative research activities to ensure effective development 

and implementation of data communications in the future Air Traffic Management 

environment. The focus of this agreement is on (1) the development of a Test Plan for the 

conduct of a Human-in-the-Loop Simulation designed to address specific key issues in the 

successful development and implementation of data linked communications, (2) the 

implementation of the Test Plan to conduct a Human-in-the-Loop Simulation at the NASA 

Langley Research Center, and (3) the analysis and reporting of results obtained from the 

simulation testing. [1]

High-level guidance comes from the FAA Data Communications National Airspace 

System Human-in-the-Loop Simulation, Airside Research Request, December 30, 2008, 

Version 1.4, and any subsequent updates or revisions as mutually agreed to during the 

execution of this Agreement by the FAA and NASA. [2]

When using Data Comm, taxi instructions are delivered through the data link system to the airplane's 

cockpit systems/avionics and then displayed as text to the pilot, rather than delivering them through the 

radio. Key research issues used as dependent variables include pilot performance, pilot errors, head-down 

time, workload, and message dialog and reply times.

A.2 Assumptions Contained in Interagency Agreement and Addendum

The following assumptions are specified in the 30 July 2009 Addendum to the Agreement, and 

subsequently expanded upon in the final FAA/NASA Data Comm Test Plan: [1] [2]

 NASA and FAA shall make an effort to incorporate scenarios that maximize similarities between 

the GENERA airport diagram (used FAA Technical Center‟s Research Development and Human 

Factors Laboratory, or RDHFL), and the airport selected for this study.

 Consistent with the expected capabilities and functions in Segment 2 (2017-2022), the test plan 

will focus on the terminal domain and shall use realistic traffic levels. 

 The study shall be conducted using instrument flight rules in day visual meteorological conditions.

 Real-world arrival and taxi routes, procedures, and operations will be simulated to the maximum 

extent practicable.

 Checklists and performance data will be provided to the crews. Communications and navigational 

facilities and procedures will be simulated to the maximum fidelity feasible.

 The departure phase of operations shall consist of taxiing from the gate onto position-and-hold of 

the departure runway. Two “Expected Taxi” messages will be given while the aircraft is at the 
Appendix A: Interagency Agreement 76

gate and prior to push back, one “Taxi” message will be given after pushback and with the engines 

running, one “Amended Taxi” message will be given to change the taxi route prior to reaching the 

intersection of an active runway, and an “Amended Taxi” message will be given to cross the 

active runway.

 The arrival phase of operations shall consist of flight from 18,000 feet MSL and terminate at the 

gate. Two “Expected Taxi” messages will be given while airborne, one Taxi message will be 

given during landing roll-out, and one “Amended Taxi” to change the taxi route will be given after 

clear of the runway.

 Ten 2-pilot crews shall be utilized, each for one full test day. NASA will recruit Airline Transport 

Pilots with Boeing 757 or 767 type-rating with current or recent flight experience.

 Datalink and Voice communications shall be utilized.

 Voice synthesis (real-time text to speech generation) shall not be used.

 Pre-recorded air traffic control communications (both directive and party-line) shall be used with 

researcher intervention when necessary.

 The Class D-level Integration Flight Deck (IFD) simulator in a fixed-base platform configuration 

will be used.

 The content, size, and location of the ND shall not constitute a Data Comm-research question but 

may be used to display graphical data link taxi (D-TAXI) clearances (graphical displays shall 

include a moving map, with own ship, and taxi route). [Amplifying comment: ND location will 

not be varied, displayed route size and color will not be varied.]

 The location of the Data Comm display will be kept constant.

 The control display unit (CDU) will be the main display to emulate Data Comm messages.

 NASA will emulate a Future Air Navigation System-1/A (FANS-1/A) capable flight management 

system (FMS) CDU, and subsets of the FANS-1/A message sets.

 The following documents are references for Data Communications messages and FANS-1/A: DO219, -256, -269, -287, -305 and -306, as well as current SC 214 documents (Safety and 

Performance Requirements).

 D-TAXI clearances shall be delivered via controller-pilot data link communications (CPDLC), 

other clearances and communications shall be delivered via Voice.

 Free text capability will not be available (need for specific free text scenarios may be assessed in 

post-experiment questionnaire).

 The time it takes a Data Comm message to travel from ATC to the CDU will be held constant.

 Rare event trials shall be part of the experiment and may include (a) non-D-TAXI tactical CPDLC 

clearances, (b) obstacles, and/or (c) errors.

 An audible chime shall be incorporated to indicate the reception of CPDLC messages. The same 

chime shall be used for all CPDLC messages .

 Each crew shall experience D-TAXI out, D-TAXI Expected, and D-TAXI-in, as well as one rare 

event.

 No Data Comm errors will be modeled. If resources permit, NASA will use procedural errors.
Appendix A: Interagency Agreement 77

 NASA will emulate ATC instructions. Clearances and background chatter will be pre-recorded.

 NASA will emulate full D-TAXI air traffic control capabilities.

 NASA will emulate realistic gate-to-gate batch mode (i.e., non-real time) traffic levels.

 For technical reasons, the departures and arrivals are to be considered at the same airport.

 Pilot error and read back-error concepts will be clearly defined during the test plan.

 NASA plans to use real-world “best-practices” flight deck roles and responsibilities.
Appendix B: Scenario Descriptions 78

Appendix B: Scenario Descriptions

Section B.1 of this Appendix defines the scenario by Display Type, Communication Modality, and 

Phase of Flight. Section B.2 contains a description and the taxi route for the arrival scenarios, and 

Section B.3 a description and the taxi route for the departure scenarios.

B.1 Scenario Case Number by Display Type, Communication, and Flight Phase

Table 38. Scenario Case Number by Display Type and Communication Modality

Display Type Comm Phase Rep Scenario ATIS Phase Case Fig

Paper Voice Arrival 1 NW3A A Data 101 32

Paper Voice Arrival 2 NW3A A Data 102 32

Paper Voice Departure 1 RW27A J Data 141 33

Paper Voice Departure 2 RW27A J Data 142 33

Paper Voice Departure 1 RW27C J Trng 181 33

Paper Data Comm Arrival 1 SC4A D/E Data 211 30

Paper Data Comm Arrival 2 SC4A D/E Data 212 30

Paper Data Comm Departure 1 RW33LA H Data 251 35

Paper Data Comm Departure 2 RW33LA H Data 252 35

Paper Data Comm Departure 1 RW33LC H Trng 281 35

MMD Data Comm Arrival 1 NW3B B/C Data 321 32

MMD Data Comm Arrival 2 NW3B B/C Data 322 32

MMD Data Comm Departure 1 RW27B I Data 361 34

MMD Data Comm Departure 2 RW27B I Data 362 34

MMD Data Comm Arrival 1 SC4C F/G Trng 381 33

MMD + Route Data Comm Arrival 1 SC4B F/G Data 431 33

MMD + Route Data Comm Arrival 2 SC4B F/G Data 432 33

MMD + Route Data Comm Departure 1 RW33LB K Data 471 36

MMD + Route Data Comm Departure 2 RW33LB K Data 472 36

MMD + Route Data Comm Arrival 1 NW3C B/M Trng (Last) 581 34

MMD Data Comm Departure 1 RW27T N/O Data (Last) 561 35

Scenario Legend:

NW3A NORWICH THREE Arrival A, ILS Runway 33L Approach

NW3B NORWICH THREE Arrival B, ILS Runway 33L Approach

NW3C NORWICH THREE Arrival C, ILS Runway 33L Approach (Training)

SC4A SCUPP FOUR Arrival A, ILS Runway 27 Approach

SC4B SCUPP FOUR Arrival B, ILS Runway 27 Approach

SC4C SCUPP FOUR Arrival C, ILS Runway 27 Approach (Training)

RW27A Runway 27 Departure A

RW27B Runway 27 Departure B

RW27C Runway 27 Departure C (Training)

RW33LA Runway 33L Departure A

RW33LB Runway 33L Departure B

RW33LC Runway 33L Departure C (Training)
Appendix B: Scenario Descriptions 79

Case Legend:

1st Digit – Cell

1: Paper / Voice

2: Paper / Data Comm

3: MMD / Data Comm

4: MMD+Route / Data Comm

5: Trust (was an arrival MMD / Data Comm scenario)

2nd Digit – Type

0: Arrival, Paper / Voice (S1 Baseline)

1: Arrival, Paper / Data Comm (S1, S2 Baseline)

2: Arrival, MMD / Data Comm (S2)

3: Arrival, MMD+Route / Data Comm (S2)

4: Departure, Paper / Voice (S1 Baseline)

5: Departure, Paper / Data Comm (S1, S2 Baseline)

6: Departure, MMD / Data Comm (S2)

7: Departure, MMD+Route / Data Comm (S2)

8: Training

3rd Digit – Replication Number (1 or 2)

Table 39. Scenario run order by crew

Crew →

Run # ↓

1 2 3 4 5 6 7 8 9 10 11

1 101 431 H 141 471 251 361 211 M 321 M 251 101 251

2 251 361 211 H 321 H 431 L 101 471 141 321 L 431 M 431 L

3 321 H 211 L 361 251 141 471 101 431 H 471 361 141

4 471 141 431 M 101 321 M 211 M 361 251 211 H 141 321 M

5 141 471 101 431 M 211 H 321 L 251 361 141 211 L 211 H

6 211 M 321 M 251 361 471 141 431 L 101 361 471 471

7 361 251 321 L 211 L 101 431 H 141 471 431 M 321 H 101

8 431 L 101 471 141 361 251 321 H 211 L 101 251 361

9 102 432 H 142 472 252 362 212 M 322 M 252 102 252

10 252 362 212 H 322 H 432 L 102 472 142 322 L 432 M 432 L

11 322 H 212 L 362 252 142 472 102 432 H 472 362 142

12 472 142 432 M 102 322 M 212 M 362 252 212 H 142 322 M

13 142 472 102 432 M 212 H 322 L 252 362 142 212 L 212 H

14 212 M 322 M 252 362 472 142 432 L 102 362 472 472

15 362 252 322 L 212 L 102 432 H 142 472 432 M 322 H 102

16 432 L 102 472 142 362 252 322 H 212 L 102 252 362

NOTE: Suffix indicates altitude range that the Data Comm messages were given at.

 H: 16,000 – 14,000 feet MSL

 M: 10,000 – 8,000 feet MSL

 L: 7,000 – 5,000 feet MSL
Appendix B: Scenario Descriptions 80

B.2 Taxi Routes for Arrival Scenarios

SCUPP 4 Arrival to Runway 27, Arrival A (Case #211, 212)

Description: Landing Runway 27 (aircraft can exit at taxiways E, K, & M)

Data Comm message: Taxi Terminal B via M.K.E.

Figure 35. Runway 27 Arrival A
Appendix B: Scenario Descriptions 81

SCUPP4 Arrival to Runway 27, Arrival B and Arrival C (Case #431, 432, and #381)

Description: Landing Runway 27 (aircraft can exit at taxiways E, K, & M)

Data Comm message: Taxi Terminal B via K.A.

Figure 36. Runway 27 Arrival B and C
Appendix B: Scenario Descriptions 82

NORWICH3 Arrival to Runway 33L Arrival A, B, and C (Case #101, 102, 321, 322, and #581)

Description: Landing Runway 33L (aircraft can exit at taxiways Q, N, Z, X, & L)

Data Comm message: Taxi Terminal E via B.A1.

Figure 37. Runway 33L Arrival A, B, and C
Appendix B: Scenario Descriptions 83

B.3 Taxi Routes for Departure Scenarios

Runway 27 Departure A, C, and T (Case #141, 142, 181, and 561)

Description: From Terminal E to Runway 27

Data Comm message: Taxi via A.C.D; Hold short 33L

Figure 38. Runway 27 Departure A, C, and T
Appendix B: Scenario Descriptions 84

Runway 27 Departure B (Case #361, 362)

Description: From Terminal E to Runway 27

Data Comm message: Taxi via A.F.H.RW22L.C.D; Hold short 33L

Figure 39. Runway 27 Departure B
Appendix B: Scenario Descriptions 85

Runway 33L Departure A and C (Case #251, 252, and 281)

Description: From Terminal E to Runway 33L: 

Data Comm message: Taxi via A.C; Hold short 27

Figure 40. Runway 33L Departure A and C
Appendix B: Scenario Descriptions 86

Runway 33L Departure B (Case #471, 472)

Description: From Terminal E to Runway 33L

Data Comm message: Taxi via A.F.H.RW22L.C; Hold short 27.

Figure 41. Runway 33L Departure B
Appendix C: Bio Questionnaire 87

Appendix C: Biographical Questionnaire

Appendix C is an exact copy of the Biographical Questionnaire completed by the subject pilots.

This questionnaire requests the most up to date information about the Subject Pilot. This data may be 

used during data analysis, however, no personal information will be connected to any of the data recorded 

in this simulation. 

Age ____________

Gender (please circle) MALE FEMALE

Commercial aircraft type / hours ____________ ____________

Military aircraft type / hours ____________ ____________

Total flight hours / total simulator hours ____________ ____________

Date of last flight (airline transport) ____________

Will you wear glasses during this experiment? YES NO

Have you had eye surgery? (Please describe your surgery below) YES NO

Do you have any known eye or eyelid abnormalities (astigmatism, etc)? (Describe) YES NO

Are your eyes corrected to different distances? (Describe) YES NO

Do you have experience using Data Comm equipment and procedures? (Describe) YES NO

How often have you flown into and out of Boston Logan airport in the past five years? ______
Appendix D: Post-Scenario Questionnaire 88

Appendix D: Post-Scenario Questionnaire

Appendix D contains all the questions in the Post-Scenario Questionnaire completed by the subject 

pilots on a Tablet PC (personal computer) after the last training run, and after every data collection run.

D.1 Workload During Scenario by Phase of Flight

Using the chart below, read the descriptions that define a particular workload level during a particular 

phase of flight or during ground operations. Move vertically up the scale until you find a description that 

accurately portrays the level of workload based on the scenario you have just flown. Move to the right 

and read the choices. Below the chart, record the appropriate ratings associated with receiving messages 

on the specified phase of flight from 1 to 10, 1 being lowest and 10 being the highest workload. If the 

scenario is a departure there will only be one question to rate. (NOTE: the entire scale was visible to the 

subject pilot while answering the workload rating questions, and the training package as well as 

instructions throughout the experiment specified the workload pertained to all normal PF or PM duties 

and functions, not just those related to Data Comm or flying the aircraft.)

Workload by Phase of Flight

1) Your workload in flight __________ (1-10)

2) Your workload during surface / taxi operations __________ (1-10)

Figure 42. Bedford work scale
Appendix D: Post-Scenario Questionnaire 89

D.2 Situation Awareness by Phase of Flight

Please answer the questions below with respect to the impact of Voice or Data Communications 

between the controller and pilot during the scenario. Select the rating that reflects your understanding of 

the dimensions described at the left for the appropriate phase of flight (all phases for the arrival scenarios, 

and surface operations only for the departure scenarios). 

DEMAND ON ATTENTIONAL RESOURCES: 

Rate your overall impression of the scenario in terms of how much 

attention and effort was required to successfully perform the tasks. 

Items to consider include: the likelihood of the situation changing 

suddenly, the degree of complexity associated with this scenario; and 

the number of variables changing during the scenario.

 (1) High Low (7)

2A) 1 2 3 4 5 6 7 during flight

2B) 1 2 3 4 5 6 7 surface ops

SUPPLY OF ATTENTIONAL RESOURCES: 

Rate the degree of spare attention that you had available to perform 

tasks other than your primary task of piloting the aircraft was 

performed. Items to consider include: how much focus and 

concentration was necessary and how you divided your attention 

between the flying task and other tasks. High = plenty of spare 

capacity; Low = little spare capacity..

 (1) High Low (7)

2C) 1 2 3 4 5 6 7 during flight

2D) 1 2 3 4 5 6 7 surface ops

UNDERSTANDING OF THE SITUATION: 

Rate your overall understanding of what was happening with the 

aircraft during this scenario. Items to consider include: the quantity 

of information received and understood; the quality of the 

information; and the familiarity you may have had with what was 

taking place during the scenario.

 (1) High Low (7)

2E) 1 2 3 4 5 6 7 during flight

2F) 1 2 3 4 5 6 7 surface ops
Appendix D: Post-Scenario Questionnaire 90

D.3 Sources of Information

Please rate the following with “1” as Very Important, and “7” as Not Important, areas that contributed 

to your Situation Awareness given all available resources in the flight test scenario. Place an “X” by 

those areas that did not contribute to your SA.

1. Visual information on the Primary Flight Display

2. Visual perception on the NAV Display

3. Visual information on the charts

4. Visual information available out the window

5. Visual information on the CDU pages

6. Visual information that your crew member directed your attention to

7. Auditory information conveyed by ATC

8. Auditory information conveyed by your crew member

9. Your perception of your crew member‟s actions

D.4 Crew Interaction

1) Your performance was proficient in this scenario. (1) Strongly Agree, (7) Strongly Disagree

|____|____|____|____|____|____|

1 7

2) My crewmember‟s performance was proficient in this scenario. (1) Strongly Agree, (7) Strongly Disagree

|____|____|____|____|____|____|

1 7

3) Your awareness of operational plans, decisions, and had appropriate 

SA throughout the flight. 

(1) Strongly Agree, (7) Strongly Disagree

|____|____|____|____|____|____|

1 7

4) The other pilot was aware of operational plans, decisions, and had 

appropriate SA throughout the flight. 

(1) Strongly Agree, (7) Strongly Disagree

|____|____|____|____|____|____|

1 7

5) There was adequate communication. (1) Strongly Agree, (7) Strongly Disagree

|____|____|____|____|____|____|

1 7

6) The Captain and FO maintained their roles throughout the scenario. (1) Strongly Agree, (7) Strongly Disagree

|____|____|____|____|____|____|

1 7
Appendix D: Post-Scenario Questionnaire 91

D.5 Acceptability of “Expected Taxi” and “Taxi” Clearances

1) Did the display of the OWNSHIP POSITION on the navigation display make the taxi clearance easier 

to understand and to carry out? [NA for runs without ownship displayed]

| | | | | | |

instructions were sometimes easier did not make easier

easier to understand to understand to understand

N/A

____

2) Did the display of the ROUTE on the navigation display make the taxi clearance easier to understand 

and to carry out? [NA for runs without route displayed]

| | | | | | |

instructions were sometimes easier did not make easier

easier to understand to understand to understand

N/A

____

3) Did you have confidence that the taxi route was accurately depicted based on the Data Comm ATC instruction?

| | | | | | |

confident the taxi route confident route accurate not confident taxi route

was accurate & followed the route but verified the route displayed accurately

4) Did you have a sufficient amount of time to respond to the Voice or Data Comm transmitted messages?

| | | | | | |

I had more than just about the right I did not have enough

enough time to respond amount of time time to respond

5) Was the amount of Head Down time required to receive and respond to just the “Expected Taxi” Data Comm 

messages acceptable in this scenario?

| | | | | | |

Minimal increase in Acceptable amount Too much head

Head Down time of Head Down time down time

6) Was the amount of heads-down time required to receive and respond to other non-time critical Data Comm 

messages acceptable in this scenario? (e.g., frequency changes, new altimeter setting, etc)

| | | | | | |

Minimal increase in Acceptable amount Too much heads

Head Down time of Head Down time down time

7) Overall, was the communication mode (Voice or Data Comm) for receiving Expected Taxi and Taxi clearances 

acceptable during this scenario? (Include consideration of message intrusiveness, amount of heads-down time 

required, effect of party line information, expected response and timing of the response, ease of use, etc.) 

| | | | | | |

Completely Neither unacceptable Completely

acceptable nor acceptable unacceptable

8) How much operational risk was introduced by the communication mode (Voice or Data Comm) used during this 

scenario?

| | | | | | |

extremely low risk neither high or low risk extremely high risk

9) Was there a point at which you did not feel that the transmitted taxi instructions were accurate?

| | | | | | |

the message some aspects were I did not feel the

was accurate inaccurate or in question message was accurate
Appendix E: Post-Experiment Questionnaire 92

Appendix E: Post-Experiment Questionnaire 

E.1 Workload Comparison

Considering all the scenarios in this simulation, compare the perceived workload of the scenario type on the left 

side of the scale to that of the other scenario type at the right end of the scale. Please circle a tick mark at the level of

workload considering the impact the communication mode and display had on your task execution and completion 

(consider time to write down or read the clearance, understand the clearance, upload the clearance if applicable, 

brief the other crewmember, and then respond to ATC). 

Voice with paper displays Data Comm with paper displays

 | | | | | | |

 Least workload Equal workload Least workload

Voice with paper displays Data Comm with Moving Map Display

 | | | | | | |

 Least workload Equal workload Least workload

Voice with paper displays Data Comm with MMD and route

 | | | | | | |

 Least workload Equal workload Least workload

Data Comm with paper displays Data Comm with Moving Map Display

 | | | | | | |

 Least workload Equal workload Least workload

Data Comm with paper displays Data Comm with MMD and route

 | | | | | | |

 Least workload Equal workload Least workload

Data Comm with MMD Data Comm with MMD and route

 | | | | | | |

 Least workload Equal workload Least workload

E.2 Situation Awareness Comparison

Considering all the scenarios in this simulation, compare the perceived SA of the scenario type on the left side of 

the scale to that of the other scenario type at the right end of the scale. Please circle a tick mark at the level of SA 

considering the impact the communication mode and display had on your task execution and completion (consider 

time to write down or read the clearance, understand the clearance, upload the clearance if applicable, brief the other 

crewmember, and then respond to ATC). 

Voice with paper displays Data Comm with paper displays

 | | | | | | |

 High SA Equal SA High SA 

Voice with paper displays Data Comm with Moving Map Display

 | | | | | | |

 High SA Equal SA High SA 

Voice with paper displays Data Comm with MMD and route

 | | | | | | |

 High SA Equal SA High SA 

Data Comm with paper displays Data Comm with Moving Map Display

 | | | | | | |

 High SA Equal SA High SA 

Data Comm with paper displays Data Comm with MMD and route

 | | | | | | |

 High SA Equal SA High SA 

Data Comm with MMD Data Comm with MMD and route

 | | | | | | |

 High SA Equal SA High SA 
Appendix E: Post-Experiment Questionnaire 93

E.3 Acceptability of “Expected Taxi” Data Comm message

Please mark the appropriate boxes to indicate:

 when it would be acceptable for a controller to send an Expected Taxi clearance via Data 

Comm (for planning purposes, an immediate response is not required, etc)

 when the flight crew would respond to the Expected Taxi message (within 2 minutes):

Controller to send 

Expected Taxi msg

Flight crew to 

respond to message

YES NO YES NO

Condition: Data Comm with paper

Above 10,000 feet MSL

Below 10,000 feet MSL

Final Approach Fix through roll-out

Taxiing Surface Operations

Condition: Data Comm with Moving Map

Above 10,000 feet MSL

Below 10,000 feet MSL

Final Approach Fix through roll-out

Taxiing Surface Operations

Condition: Data Comm with MMD and route

Above 10,000 feet MSL

Below 10,000 feet MSL

Final Approach Fix through roll-out

Taxiing Surface Operations

E.4 Trust in the System

Considering all the scenarios in this simulation, compare your perceived trust of the system as it pertains to the 

communication modality or display configuration on the left side of the scale to that of the other communication 

modality or display configuration at the right end of the scale. Please circle a tick mark at the level of trust in the 

system considering the impact the communication mode and/or display had on your task execution and completion 

(consider time to write down or read the clearance, understand the clearance, upload the clearance if applicable, 

brief the other crewmember, and then respond to ATC). 

Voice with paper displays Data Comm with paper displays

 | | | | | | |

 High Trust High Trust 
Appendix E: Post-Experiment Questionnaire 94

Voice with paper displays Data Comm with Moving Map Display

 | | | | | | |

 High Trust High Trust 

Voice with paper displays Data Comm with MMD and route

 | | | | | | |

 High Trust High Trust 

Data Comm with paper displays Data Comm with Moving Map Display

 | | | | | | |

 High Trust High Trust 

Data Comm with paper displays Data Comm with MMD and route

 | | | | | | |

 High Trust High Trust 

Data Comm with MMD Data Comm with MMD and route

 | | | | | | |

 High Trust High Trust 

On a scale of 1 to 7 by circling a mark along the scale, with 1 being the highest and 7 being the lowest, please rate 

the next five questions based on your experience here in the experiment.

1. Overall, how confident were you that the data linked message was properly loaded into the FMS and then 

correctly displayed graphically on the Navigation Display? 1 (Complete confidence) 7 (No Confidence)

2. How often did you verify the accuracy of the data link taxi instructions? 1 (All the time) 7(None of the 

time)

 If you didn‟t verify the accuracy, why not (please check those that apply): 

___ Not enough time 

___ presumed accurate

___ presumed other crew member verified

___ Other, please explain. 

3. How often did you verify the taxi route displayed on the Navigation Display with the Data Comm message 

on the CDU? 1 (All the time) 7 (None of the time)

 If you didn‟t verify the route described by the instructions , why not (check those that apply): 

___ Not enough time 

___ presumed accurate

___ presumed other crew member verified

___ Other, please explain. 

4. How long did it take you to notice the data link message was incorrect? 1 (Immediately) 7 (Did not notice)

5. The method for receiving, uploading, and carrying out the air traffic taxi instructions via Data Comm: 

Has Integrity, Is Reliable, Is incomplete, Is ambiguous ? 1 (No) 7 (Yes)

E.5 Crew Coordination Support

Considering all the scenarios in this simulation, compare the perceived support for effective Crew 

Resource Management and Crew Coordination of the scenario type on the left side of the scale to that of 

the other scenario type at the right end of the scale. Please circle a tick mark on the scale index that 

reflects the effect the communication mode and display had on your ability to effectively coordinate as a 

team, distribute your attentional resources effectively, and ensure common situation awareness. 
Appendix E: Post-Experiment Questionnaire 95

Voice with paper displays Data Comm with paper displays

 | | | | | | |

 High SA Equal SA High SA 

Voice with paper displays Data Comm with Moving Map Display

 | | | | | | |

 High SA Equal SA High SA 

Voice with paper displays Data Comm with MMD and route

 | | | | | | |

 High SA Equal SA High SA 

Data Comm with paper displays Data Comm with Moving Map Display

 | | | | | | |

 High SA Equal SA High SA 

Data Comm with paper displays Data Comm with MMD and route

 | | | | | | |

 High SA Equal SA High SA 

Data Comm with MMD Data Comm with MMD and route

 | | | | | | |

 High SA Equal SA High SA 

E.6 Summary

1. To what degree did the scenarios in this experiment accurately simulate a complex, high-workload 

environment? If not, what was missing? 1 (realistic) 7 (unrealistic)

2. What is your overall assessment of the potential of communicating clearance updates or changes 

using datalink while an aircraft is taxiing or in busy terminal airspace? 1 (realistic) 7 (unrealistic)

3. Should the dotted cyan lines for an “Expected Taxi” clearance include red hold short bars?

4. Will the solid magenta line for a Taxi clearance on the Navigation Display encourage crew members 

to begin taxiing prior to receiving the Voice message from ATC?

5. Was the simultaneous Voice and Data Comm instructions to cross an active runway clear? Was there 

a delay in the FO updating the graphical display on the ND? Was that delay important?

6. How would CDTI (Cockpit Display of Traffic Information) impact your workload, SA, and 

acceptability of using Data Comm messages in terminal airspace or surface operations?

7. Was the use of Voice by the controller for critical or time-sensitive information (such as crossing the 

runway) appropriate and necessary?

8. Were there any challenges with Data Comm unique to your flight duties as the PF or PM?

9. Do you have any other comments? Include any unexpected events, operational issues, and any 

problems with the simulator that affected your performance.
Appendix F: Oculometer Apparatus 96

Appendix F: Oculometer Apparatus

A ten-camera oculometer system was installed in the IFD to 

support unobtrusive collection of eye tracking and head 

position data for both flight crew subjects. The Smart Eye Inc. 

eye tracker used in this experiment (Figure 43) was a remote 

eye tracking system that used facial recognition to calculate the 

position of defined points on a subjects head relative to the 

calibrated position of two or more cameras. The cameras used

the facial features to locate the corners of each of the subject‟s 

eyes and digitally zoomed to enhance the image of the eye.

Figure 43. Oculometer and IR Flasher

To calculate eye gaze vectors from the head origin, 

infrared light emitting diodes projected infrared light to 

illuminate the pilots face and to create two ocular 

reflections; a static corneal reflection and a pupil 

reflection that moves in conjunction with eye movements. 

Triangulating the angular difference between the corneal 

reflection and pupil reflection, the Smart Eye eye tracking 

system creates a vector between the two points, which 

creates an eye gaze vector originating from the corneal 

reflection at the center of the pilot‟s eyes (Figure 44). 

 Figure 44. Eye Gaze Vector

Ten cameras in total were utilized, with one eye tracking system for the PF and one for the PM, each 

with five cameras to capture the gaze vectors of both pilots simultaneously (installation shown in Figure 

45). To synchronize the systems, Smart Eye Inc. created a modified eye tracking system network, 

tethering two systems together using a master-slave relationship. Each system is time stamped 

synchronously with global positioning system time so eye gaze vector data from both pilots can be 

compared. 

In order to achieve robust eye tracking data over the span of coverage required for normal cockpit 

operations, the system had to be capable of covering +/-45 degrees of center, and +10 degrees from 

horizon and to the base of the CDU for each pilot. This requirement had to be met while still maintaining 

a high level of simulator fidelity by making the cameras as inconspicuous as possible on the flight deck. 

Camera placement was optimized for coverage within constraints imposed by limited available real estate.

To test which available locations for installation on the flight deck provided the greatest coverage 

capability, a mockup of the IFD was created. Test results concluded with five locations per side being 

chosen (mirrored locations between left and right seat) that yielded sufficient coverage to perform flight 

testing while remaining minimally obtrusive in the flight deck. System spatial accuracy was tested to be 

no greater than 2 degrees gaze angle for any calibration point on the display panels. 
Appendix F: Oculometer Apparatus 97

 The oculometer provided the following raw data in real-time:

 Gaze vectors for each eye of both crew members (raw)

 Head and eye position (each eye) for each crew

 Eyelid closure distance for each eye for each crew

 Pupil size for each eye for each crew

Figure 45. Location of Oculometers and IR Flashers in IFD Simulator
Appendix G: Data Comm Message Format 98

Appendix G: Data Comm Message Format

Appendix G lists all the Data Comm Uplink and Downlink message IDs and formats used in this 

research, and were based on the proposed revision to the Data Comm standards (Reference 33), or 

developed specifically for this experiment (marked as “New”).

NOTE 1: No standard yet for taxi messages, therefore followed NASA Langley and EMMA2 

operational evaluation with each taxiway defined by a single letter, e.g., “A” and not “ALPHA”.

NOTE 2: All datalink taxi messages provide the route only, and do not constitute direction to begin 

taxiing, nor permission to cross any active or inactive runway (movement instructions given via voice).

NOTE 3: No yet defined if Taxi Clearance is from current position to takeoff runway or parking 

location, to include segments after crossing a runway.

NOTE 4: Data Comm uplink CDU displayable characters need to be restricted to uppercase alpha 

characters A - Z; numerical numbers 0 - 9; space ( ); and symbols (,) (.) (/) (+) (-).

Table 40. Data Comm uplink messages (UM) and downlink messages (DM) 

UM

0, 1, 3, 4, 5

General Responses. UNABLE, STANDBY, ROGER, 

AFFIRM, NEGATIVE 

UM DT01

(New)

Instruction that engine start up is approved at 

the specified time.

START UP APPROVED [assigned 

time]

UM DT03

(New)

Instruction that push back is approved at 

specified location, direction, and time

PUSH BACK APPROVED [pushback 

information] [assigned time]

UM DT05

(New)

Notification that taxi clearance may be issued 

on the specified taxi route

EXPECT TAXI [taxi route]

UM DT09 

(New)

Instruction to taxi to the specified location 

without a hold short instruction.

TAXI [taxi route]

UM DT10 

(New)

Instruction to taxi to the specified location 

with a hold short position.

RUNWAY [runway] TAXI [taxi route]

New UM 

DT12

Instruction to hold the current position. HOLD POSITION

UM DT73 

(New)

Notification to the aircraft of the instructions 

to be followed from departure until the 

specified clearance limit.

[departure clearance routing]

UM47 Instruction that the specific position is to be 

crossed at or above the specified level.

CROSS [position] AT OR ABOVE 

[level]

UM117 Instruction that the ATS unit with specified 

ATS name is to be contacted on the specified 

frequency.

CONTACT [unit name] [frequency]

UM212 ATS advisory that the specified ATIS 

information at the specified airport is current

[facility designation] ATIS [atis code] 

CURRENT

UM 213

(New)

ATS advisory that the specified altimeter 

setting relates to the specified facility.

[facility designation] ALTIMETER 

[altimeter] [timesec]

DM0, 1, 2, 3,

4, 5, 6

General Responses: WILCO, UNABLE, STANDBY, ROGER, 

AFFIRM, NEGATIVE, REQUEST
Appendix H: Data Comm Messages 99

Appendix H: Data Comm Uplink Messages

Table 41. Data Comm uplink messages by scenario

Case # Arrival, Data Comm with Paper

211 CROSS SCUPP AT 11,000 FT 230 KIAS

211 EXPECT TAXI TO TERMINAL B VIA E.M.C.A

211 KBOS ALTIMETER 30.02

211 EXPECT TAXI TO TERMINAL B VIA E

211 KBOS ATIS ECHO CURRENT

211 CONTACT BOS TOWER 132.22

211 TAXI TO TERMINAL B VIA K.E-1

211 AMENDED CLEARANCE TAXI TO TERMINAL B VIA K.B.E

212 CROSS SCUPP AT 11,000 FT 230 KIAS

212 EXPECT TAXI TO TERMINAL B VIA E.M.C.A

212 KBOS ALTIMETER 30.02

212 EXPECT TAXI TO TERMINAL B VIA E

212 KBOS ATIS ECHO CURRENT

212 CONTACT BOS TOWER 132.22

212 TAXI TO TERMINAL B VIA K.E-1

212 AMENDED CLEARANCE TAXI TO TERMINAL B VIA K.B.E

Case # Departure, Data Comm with Paper

251 CLEARED TO START

251 KBOS ATIS HOTEL CURRENT

251 PUSHBACK AT 1931Z

251 KBOS ALTIMETER 29.96

251 EXPECT TAXI TO RW 33L VIA A.Z.B.F.M.C

251 EXPECT TAXI TO RW 33L VIA A.A-1.B.Q.M.F.H.RW22L.C

251 TAXI TO RW 33L VIA A.C HOLD SHORT RW 27

251 AMENDED CLEARANCE TAXI TO RW 33L VIA A.F.M.C HOLD SHORT RW 27

251 AMENDED CLEARANCE TAXI TO RW 33L VIA C HOLD SHORT RW 33L

252 KBOS ATIS HOTEL CURRENT

252 KBOS ALTIMETER 29.96

252 EXPECT TAXI TO RW 33L VIA A.Z.B.F.M.C

252 EXPECT TAXI TO RW 33L VIA A.A-1.B.Q.M.F.H.RW22L.C

252 PUSHBACK AT 1434Z

252 CLEARED TO START

252 TAXI TO RW 33L VIA A.C HOLD SHORT RW 27

252 AMENDED CLEARANCE TAXI TO RW 33L VIA A.F.M.C HOLD SHORT RW 27

252 AMENDED CLEARANCE TAXI TO RW 33L VIA C HOLD SHORT RW 33L
Appendix H: Data Comm Messages 100

Case # Arrival, Data Comm with Moving Map Display (MMD)

321 CROSS PVD AT 11000 FT 250 KIAS

321 KBOS ALTIMETER 30.02

321 EXPECT TAXI TO TERMINAL E VIA L.B.A-1

321 KBOS ATIS CHARLIE CURRENT

321 EXPECT TAXI TO TERMINAL E VIA L.B.Z

321 CONTACT BOS TOWER 128.8

321 TAXI TO TERMINAL E VIA N.B.Z

321 AMENDED CLEARANCE TAXI TO TERMINAL E VIA B.L.A

322 CROSS PVD AT 11000 FT 250 KIAS

322 KBOS ALTIMETER 30.02

322 EXPECT TAXI TO TERMINAL E VIA L.B.A-1

322 KBOS ATIS CHARLIE CURRENT

322 EXPECT TAXI TO TERMINAL E VIA L.B.Z

322 CONTACT BOS TOWER 128.8

322 TAXI TO TERMINAL E VIA N.B.Z

322 AMENDED CLEARANCE TAXI TO TERMINAL E VIA B.L.A

Case # Departure, Data Comm with Moving Map Display (MMD)

361 KBOS ATIS INDIA CURRENT

361 KBOS ALTIMETER 29.90

361 EXPECT TAXI TO RW 27 VIA A.C.D

361 EXPECT TAXI TO RW 27 VIA A.Q.M.C.D

361 PUSHBACK AT 2158Z

361 CLEARED TO START

361 TAXI TO RW 27 VIA A.F.M.C.D HOLD SHORT RW 33L

361 AMENDED CLEARANCE TAXI TO RW 27 VIA F.H.RW22L.C.D HOLD SHORT RW 

33L

361 AMENDED CLEARANCE TAXI TO RW 27 VIA D HOLD SHORT RW 27

362 KBOS ATIS INDIA CURRENT

362 KBOS ALTIMETER 29.90

362 EXPECT TAXI TO RW 27 VIA A.C.D

362 PUSHBACK AT 1648Z

362 CLEARED TO START

362 EXPECT TAXI TO RW 27 VIA A.Q.M.C.D

362 TAXI TO RW 27 VIA A.F.M.C.D HOLD SHORT RW 33L

362 AMENDED CLEARANCE TAXI TO RW 27 VIA F.H.RW22L.C.D HOLD SHORT RW

33L

362 AMENDED CLEARANCE TAXI TO RW 27 VIA D HOLD SHORT RW 27
Appendix H: Data Comm Messages 101

Case # Arrival, Data Comm with Moving Map Display and Route

431 CROSS SCUPP AT 11,000 FT 230 KIAS

431 EXPECT TAXI TO TERMINAL B VIA E

431 KBOS ALTIMETER 30.02

431 EXPECT TAXI TO TERMINAL B VIA K.B.E

431 KBOS ATIS GOLF CURRENT

431 CONTACT BOS TOWER 132.22

431 TAXI TO TERMINAL B VIA K.B.A-2

431 AMENDED CLEARANCE TAXI TO TERMINAL B VIA K.E-1

432 CROSS SCUPP AT 11,000 FT 230 KIAS

432 EXPECT TAXI TO TERMINAL B VIA E

432 KBOS ALTIMETER 30.02

432 EXPECT TAXI TO TERMINAL B VIA K.B.E

432 KBOS ATIS GOLF CURRENT

432 CONTACT BOS TOWER 132.22

432 TAXI TO TERMINAL B VIA K.B.A-2

432 AMENDED CLEARANCE TAXI TO TERMINAL B VIA K.E-1

Case # Departure, Data Comm with Moving Map Display and Route

471 KBOS ATIS KILO CURRENT

471 KBOS ALTIMETER 30.04

471 EXPECT TAXI TO RW 33L VIA A.F.M.C

471 EXPECT TAXI TO RW 33L VIA A.Q.M.F.H.RW22L.C

471 PUSHBACK AT 2033Z

471 CLEARED TO START

471 TAXI TO RW 33L VIA A.C HOLD SHORT RW 27

471 AMENDED CLEARANCE TAXI TO RW 33L VIA M.E.P.D.C HOLD SHORT RW 27

471 AMENDED CLEARANCE TAXI TO RW 33L VIA C HOLD SHORT RW 33L

472 KBOS ATIS KILO CURRENT

472 KBOS ALTIMETER 30.04

472 EXPECT TAXI TO RW 33L VIA A.F.M.C

472 PUSHBACK AT 1544Z

472 CLEARED TO START

472 EXPECT TAXI TO RW 33L VIA A.Q.M.F.H.RW22L.C

472 TAXI TO RW 33L VIA A.C HOLD SHORT RW 27

472 AMENDED CLEARANCE TAXI TO RW 33L VIA M.E.P.D.C HOLD SHORT RW 27

472 AMENDED CLEARANCE TAXI TO RW 33L VIA C HOLD SHORT RW 33L
Appendix I: Training Program 102

Appendix I: Flight Crew Training Program

Appendix I contains the slides given to the flight crew during training prior to proceeding to the 

simulator. The two-hour training program was structured to provide the subject pilots an overview of the 

NextGen environment by 2017, to include new technologies, new flight procedures, and the challenges in 

implementing this concept of operations. An experiment hypothesis and test plan was described, and then

an in-depth discussion was held on Data Comm messages and required crew interaction, as well as the 

associated displays. At that point in the training program, tablet PCs were given to both crew members 

for them to practice the Data Comm messages and responses, as well as to see what the graphical display 

looked like.

Once all the individual messages were understood and replied to properly, the training shifted to 

describing each of the eight scenarios in detail, and ensuring that the crews understood what to expect and

what was expected of them. The training program finished with practicing how to answer the electronic 

questionnaires on the tablet PC, and a short description of the oculometer system and how the calibration 

process worked.

Following the academic portion of the training program, the two crew members were brought to the 

IFD, where they started with building facial profiles for the oculometer system, then began part-task and 

differences training in the IFD. This was followed by four training runs which consisted of departure, 

arrival, departure, and arrival scenarios. For training purposes, and to provide data for one of the two 

Trust scenarios, the electronic post-scenario questionnaire was completed after the final training run.
Appendix I: Training Program 103
Appendix I: Training Program 104
Appendix I: Training Program 105
Appendix I: Training Program 106
Appendix I: Training Program 107
Appendix I: Training Program 108
Appendix I: Training Program 109
Appendix I: Training Program 110
Appendix I: Training Program 111
Appendix I: Training Program 112
Appendix I: Training Program 113
Appendix I: Training Program 114
Appendix I: Training Program 115
Appendix I: Training Program 116
Appendix I: Training Program 117
Appendix I: Training Program 118
Appendix I: Training Program 119
Appendix I: Training Program 120
Appendix I: Training Program 121
Appendix I: Training Program 122
Appendix I: Training Program 123
Appendix I: Training Program 124
Appendix I: Training Program 125
Appendix I: Training Program 126
Appendix I: Training Program 127
Appendix J: Scenario Briefings 128

Appendix J: Scenario Briefings

This Appendix provides the scenario briefings given to subject pilots prior to starting the run. After 

acknowledging the briefing, the flight crew finished configuring the simulator, accomplished the Descent 

Checklist, then notified the researcher that they were ready to begin the run. Section J.1 contains the 

briefings for the NORWICH3 arrival to Runway 33L, Section J.2 the SCUPP4 arrival to Runway 27, 

Section J.3 the departures to Runway 27, and Section J.4 the departures to Runway 33L.

J.1 NORWICH3 (Arrival to Runway 33L)

NW3A (101, 102): This is a Voice scenario, with paper airport diagram only. Your Callsign is 

NASA 557. You are on a flight from KSFO to KBOS, and this scenario starts overhead NORWICH, 

established on the NORWICH THREE Arrival. You have previously been cleared for the NORWICH 

THREE Arrival, and down to 11,000 ft. You have previously been told to cross Providence at 11,000 feet 

and 250 KIAS. The FMS has been programmed for the NORWICH THREE Arrival, and the ILS 

Runway 33L Approach. You are in a descent, passing 18,000 ft at 300 KIAS, with Speedbrakes retracted. 

Altimeter setting is 30.00. You are established on Boston Approach Frequency 120.6. The Descent 

Checklist and Approach Brief have not been accomplished yet. You have been assigned Gate E2, which 

is at the Northeastern edge of Terminal E. The Autopilot, TCAS, and ACARS are inoperative. Moving 

Map Displays are not available. Data Comm is not in use.

NW3B/C (321, 322, 581): This is a Data Comm scenario, with Moving Map Displays, and no routes. 

Your Callsign is NASA 557. You are on a flight from KSFO to KBOS, and this scenario starts overhead 

NORWICH, established on the NORWICH THREE Arrival. You have previously been cleared for the 

NORWICH THREE Arrival, and down to 11,000 ft. The FMS has been programmed for the NORWICH 

THREE Arrival, and the ILS Runway 33L Approach. You are in a descent, passing 18,000 ft at 300 

KIAS, with Speedbrakes retracted. Altimeter setting is 30.02. You are established on Boston Approach 

Frequency 120.6. The Descent Checklist and Approach Brief have not been accomplished yet. You have

been assigned Gate E2, which is at the Northeastern edge of Terminal E. The Autopilot, TCAS, and 

ACARS are inoperative. Moving Map Displays are available, depicting Ownship only (no route). Data 

Comm is in use. You have previously received Data Comm messages from Boston Center (KZBW) and 

Boston Approach (KBOS) which may be reviewed prior to starting the run.

J.2 SCUPP4 (Arrival to Runway 27)

SC4A (211, 212): This is a Data Comm scenario, with paper airport diagram only. Your Callsign is 

NASA 557. You are on a flight from KDEN to KBOS, and this scenario starts overhead ARMUN, 

established on the SCUPP4 Arrival. You have previously been cleared for the SCUPP4 Arrival, and 

down to 11,000 ft. The FMS has been programmed for the SCUPP4 Arrival, and the ILS Runway 27 

Approach. You are in a descent, passing 18,000 ft at 270 KIAS, with Speedbrakes retracted. Altimeter 

setting is 29.98. You are on Boston Approach Frequency 120.6. The Descent Checklist and Approach 

Brief have not been accomplished yet. You have been assigned Gate B20, which is at the Southern edge 

of Terminal B. The Autopilot, TCAS, and ACARS are inoperative. Moving Map Displays are not 

available. Data Comm is in use. You have previously received Data Comm messages from Boston Center 

(KZBW) and Boston Approach (KBOS) which may be reviewed prior to starting the run.

SC4B/C (431, 432, 381): This is a Data Comm scenario, with Moving Map Displays, and no routes. 

Your Callsign is NASA 557. You are on a flight from KDEN to KBOS, and this scenario starts overhead 

ARMUN, established on the SCUPP4 Arrival. You have previously been cleared for the SCUPP4 
Appendix J: Scenario Briefings 129

Arrival, and down to 11,000 ft. The FMS has been programmed for the SCUPP4 Arrival, and the ILS 

Runway 27 Approach. You are in a descent, passing 18,000 ft at 270 KIAS, with speed brakes retracted. 

Altimeter setting is 30.04. You are established on Boston Approach Frequency 120.6. The Descent 

Checklist and Approach Brief have not been accomplished yet. You have been assigned Gate B20, which 

is at the Southern edge of Terminal B. The Autopilot, TCAS, and ACARS are inoperative. Moving Map 

Displays are available, with route depictions. Data Comm is in use. You have previously received Data 

Comm messages from Boston Center (KZBW) and Boston Approach (KBOS) which may be reviewed 

prior to starting the run.

J.3 Runway 27 (Departure to Runway 27)

RWY27A/C (141, 142, 181): This is a Voice scenario, with paper airport diagram only. Your 

Callsign is NASA 557. This scenario starts parked at Boston Logan Terminal E, Gate E-8A, which is at 

the North West corner of the terminal. You are on Auxiliary Power Unit (APU) power, with the engines 

shut down. You have previously received your clearance to KDEN, as per your Dispatch paperwork. 

The FMS has been programmed for a LOGAN FOUR Departure. You are on Boston Ground Frequency 

121.9, and have not asked for pushback yet. Moving Map Displays are not available. Data Comm is not 

in use. Your planned Gross Weight is 200,000 pounds. You will receive your final fuel, weight, and 

takeoff power settings during taxi out.

RWY27B/T (361, 362, 571): This a Data Comm scenario, with Moving Map Displays, and no routes. 

Your Callsign is NASA 557. This scenario starts parked at Boston Logan Terminal E, Gate E-8A, which 

is at the North West corner of the terminal. You are on APU power, with the engines shut down. You 

have previously received your clearance to KDEN, as per your Dispatch paperwork. The FMS has been 

programmed for a LOGAN FOUR Departure. You are on Boston Ground Frequency 121.9, and have not 

asked for pushback yet. Moving Map Displays are available, without routes. Data Comm is in use for DTAXI only. Your planned gross weight is 200,000 pounds. You will receive your final fuel, weight, and 

takeoff power settings during taxi out.

J.4 Runway 33L (Departure to Runway 33L)

RWY33LA/C (251, 252, 281): This is a Data Comm scenario, with paper airport diagram only. Your 

Callsign is NASA 557. This scenario starts parked at Boston Logan Terminal E, Gate E-8A, which is at 

the North West corner of the terminal. You are on APU power, with the engines shut down. You have 

previously received your clearance to KORD, as per your Dispatch paperwork. The FMS has been 

programmed for a LOGAN FOUR Departure. You are on Boston Ground Frequency 121.9, and have not 

asked for pushback yet. Moving Map Displays are not available. Data Comm is in use for D-TAXI only. 

Your planned Gross Weight is 200,000 pounds. You will receive your final fuel, weight, and takeoff

power settings during taxi out.

RWY33LB (471, 472): This is a Data Comm scenario, with Moving Map Displays and displayed 

routes. Your Callsign is NASA 557. This scenario starts parked at Boston Logan Terminal E, Gate E8A, which is at the North West corner of the terminal. You are on APU power, with the engines shut 

down. You have previously received your clearance to KORD, as per your Dispatch paperwork. The 

FMS has been programmed for a LOGAN FOUR Departure. You are on Boston Ground Frequency 

121.9, and have not asked for pushback yet. Moving Map Displays are available, with routes loadable. 

Data Comm is in use for D-TAXI only. Your planned Gross Weight is 200,000 pounds. You will receive 

your final fuel, weight, and takeoff power settings during taxi out.
Appendix K: Message Response Time 130

Appendix K: Response Time, Technical Performance, and Raw Data

Section K.1 contains all flight crew response time to Data Comm uplink messages, Section K.2 the 

technical performance data, and Section K.3 taxi speed data and raw data. All Data Comm response 

times, including responses longer than two minutes or no response at all, are included in analysis in this 

Appendix. 

K.1 Message Response Time by Altitude

Figure 46. Mean response time to “Expected Taxi” message

Figure 47. Mean response time to other Data Comm messages
Appendix K: Message Response Time 131

K.2 Technical Performance

Figure 48. Flight director error by condition

Figure 49. Flight director error by altitude
Appendix K: Message Response Time 132

K.3 Raw Data by Flight Crew

K.3.1 Crew #1
Appendix K: Message Response Time 133
Appendix K: Message Response Time 134
Appendix K: Message Response Time 135

K.3.2 Crew #2
Appendix K: Message Response Time 136
Appendix K: Message Response Time 137
Appendix K: Message Response Time 138

K.3.3 Crew #3
Appendix K: Message Response Time 139
Appendix K: Message Response Time 140
Appendix K: Message Response Time 141

K.3.4 Crew #4
Appendix K: Message Response Time 142
Appendix K: Message Response Time 143
Appendix K: Message Response Time 144

K.3.5 Crew #5
Appendix K: Message Response Time 145
Appendix K: Message Response Time 146
Appendix K: Message Response Time 147

K.3.6 Crew #6
Appendix K: Message Response Time 148
Appendix K: Message Response Time 149
Appendix K: Message Response Time 150

K.3.7 Crew #7
Appendix K: Message Response Time 151
Appendix K: Message Response Time 152
Appendix K: Message Response Time 153

K.3.8 Crew #8
Appendix K: Message Response Time 154
Appendix K: Message Response Time 155
Appendix K: Message Response Time 156

K.3.9 Crew #
Appendix K: Message Response Time 157
Appendix K: Message Response Time 158
Appendix K: Message Response Time 159

K.3.10 Crew #10
Appendix K: Message Response Time 160
Appendix K: Message Response Time 161
Appendix K: Message Response Time 162

K.3.11 Crew #11
Appendix K: Message Response Time 163
Appendix K: Message Response Time 164
Appendix L: Response Time Distribution 165

Appendix L: Data Comm Response Time Distributions

Unlike Appendix K, Appendix L contains analysis results that removed Data Comm response times 

caused by the pilot forgetting to acknowledge a message after reading it and briefing it to the other crew 

member. A very conservative limit of 120 seconds was used, which resulted in 39 of the 1056 Data 

Comm Uplink response times being removed (approximately 4%). Of these, 34 were responded to at a 

time greater than 120 seconds and 5 messages were not responded to at all. The break-down of these 

messages (i.e., those that are not included in the statistical analysis of this Appendix) was as follows:

Data Comm message type Percent Response > 2 min N

Pushback and Start: 4% 5 of 132 127

Expected Taxi-Out (ground): 6% 8 of 132 124

Taxi-Out (ground): 0% 0 of 66 66

Amended Taxi-In & Out (ground): 4% 8 of 198 190

Expected Taxi-In (airborne): 2% 3 of 132 129

Taxi-In: 0% 0 of 66 66

Frequency change: 2% 1 of 66 65

ATIS (ground and airborne): 6% 8 of 132 124

Altimeter (ground and airborne): 5% 6 of 132 126

TOTAL 39 of 1056 1017

NOTE 1: Of the eight Amended Taxi uplink messages that were removed from data analysis in this 

Appendix, four were Amended Taxi-Out messages that were responded to but at a time greater 

than 120 seconds, and four were Amended Taxi-In messages not responded to at all. It is 

postulated the four Amended Taxi-In messages not responded may be due to the scenario being 

terminated prior to the flight crew responding to the message.

NOTE 2: The fifth Data Comm message not responded to at all was an altimeter change uplink 

message during an arrival. It is not known why the crew did not respond.

The root cause for pilots not acknowledging or not acknowledging the Uplink messages in a timely 

fashion was likely the intentional selection of the FANS-1/A interface, creating a non-optimized Data 

Comm solution for terminal area operations. 

Figure 47 to 54 present the response time distribution by Data Comm message type. The following 

labels define terms unique to these data.

Skewness: values closer to 0 indicate symmetric data, negative values indicate left skew, positive 

values indicate right skew. Skew direction is the direction of the tail. Right skew means tail points right 

as we see below. 

Kurtosis: values closer to 0 indicate normally peaked data (relative to all data points), negative values 

indicate a distribution that is flatter than normal, positive values indicate a distribution that is sharper than 

normal.

The following graphs show response time distribution by Data Comm message type.
Appendix L: Response Time Distribution 166

Figure 50. Response time to Pushback message

Figure 51. Response time to Start message

Figure 52. Response time to Expected Taxi message
Appendix L: Response Time Distribution 167

Figure 53. Response time to Taxi message

Figure 54. Response time to Amended Taxi

Figure 55. Response time to Frequency change message
Appendix L: Response Time Distribution 168

Figure 56. Response time to ATIS message

Figure 57. Response time to Altimeter change
Appendix M: SC-214 Time Tables 169

Appendix M: Data Comm Response Time Tables for SC-214

In Table 42 through Table 45 of this appendix the flight crew response time to Data Comm uplink 

messages are tabulated according to the classification table used by the RTCA SC-214 work group. This 

was a special request by the FAA after the experiment had been conducted, but prior to publishing the 

original NASA report. Data analysis of pilot response time indicated that there was no statistically 

significant difference in message response time across display methodology (F=1.51, p=0.222); however,

there was a statistically significant difference in message response across message type (F=13.06, 

p<0.001). (Data used by FAA and RTCA to inform development of Reference 33.)

Table 42. Data Comm response time for all display conditions

Table 43. Data Comm response time by Paper display condition

Service ACM ACL

VCM FI

Frequency 

Change

Flight Info

RCP 102

UM117

(7) Freq

RCP 181

UMDT05 

(5) Expect 

Taxi-In

RCP 102

UM212 & 

UM213

(8) ATIS & Alt

average 16.18 17.61 19.32

95% 29.00 32.66 31.98

99.90% 29.00 32.66 68.46

RCP 47

UMDT01 & 

UMDT02 

(1) 

Push/Start

RCP 181

UMDT05 

(2) Expect 

Taxi-Out

RCP 181

UMDT10

(3) TaxiOut

RCP 181

UMDT15

(4) Amend 

TaxiIn&Out

RCP 181

UMDT10

(6) Taxi-In

RCP 102

UM212 & 

UM213

(9) ATIS & Alt

average 20.72 37.82 13.89 17.61 25.37 19.44

95% 46.54 49.08 20.92 29.86 43.00 50.77

99.90% 52.50 112.52 20.92 46.10 61.68 71.06

Airport

TMA

Domain RP

Route planning

DTAXI Service ACM ACL

VCM FI

Frequency 

Change

Flight Info

RCP 102

UM117

(7) Freq

RCP 181

UMDT05 

(5) Expect 

Taxi-In

RCP 102

UM212 & 

UM213

(8) ATIS & 

average 15.97 20.70 14.97 Alt

95% 35.46 40.78 31.98

99.90% 51.24 63.00 48.76

RCP 47

UMDT01 & 

UMDT02 

(1) 

Push/Start

RCP 181

UMDT05 

(2) Expect 

Taxi-Out

RCP 181

UMDT10

(3) TaxiOut

RCP 181

UMDT15

(4) Amend 

TaxiIn&Out

RCP 181

UMDT10

(6) Taxi-In

RCP 102

UM212 & 

UM213

(9) ATIS & 

Alt average 19.52 33.12 14.58 18.57 25.09 23.03

95% 46.54 72.44 23.94 38.00 58.12 53.86

99.90% 55.16 104.62 33.68 46.10 61.68 87.34

Airport

TMA

Domain RP

Route planning

DTAXI
Appendix M: SC-214 Time Tables 170

Table 44. Data Comm response time by MMD display condition

NOTE: Row shaded in blue is from original FAA table, and is not pertinent to this experiment.

Table 45. Data Comm response time by MMD+Route display condition

Service ACM ACL

VCM FI

Frequency 

Change

Flight Info

RCP 102

UM117

(7) Freq

RCP 181

UMDT05 

(5) Expect 

Taxi-In

RCP 102

UM212 & UM213

(8) ATIS & Alt

average 18.23 23.50 10.91

95% 36.78 40.78 18.02

99.90% 51.24 53.24 20.16

Table G-2

timers

RCP 47

UMDT01 & 

UMDT02 

(1) 

Push/Start

RCP 181

UMDT05 

(2) Expect 

Taxi-Out

RCP 181

UMDT10

(3) TaxiOut

RCP 181

UMDT15

(4) Amend 

TaxiIn&Out

RCP 181

UMDT10

(6) Taxi-In

RCP 102

UM212 & UM213

(9) ATIS & Alt

average 17.36 28.23 16.48 17.59 22.77 21.40

95% 35.52 59.70 23.86 37.00 33.48 52.68

99.90% 42.20 83.28 33.68 43.12 116.64 63.80

Note 11

Airport 100s 45s/100s/180s

TMA

Route planning

DTAXI

Domain RP

Service ACM ACL

VCM FI

Frequency 

Change

Flight Info

RCP 102

UM117

(7) Freq

RCP 181

UMDT05 

(5) Expect 

Taxi-In

RCP 102

UM212 & 

UM213

(8) ATIS & Alt

average 13.39 18.96 14.86

95% 19.22 38.82 30.00

99.90% 19.22 58.18 36.00

RCP 47

UMDT01 & 

UMDT02 

(1) 

Push/Start

RCP 181

UMDT05 

(2) Expect 

Taxi-Out

RCP 181

UMDT10

(3) TaxiOut

RCP 181

UMDT15

(4) Amend 

TaxiIn&Out

RCP 181

UMDT10

(6) Taxi-In

RCP 102

UM212 & 

UM213

(9) ATIS & Alt

average 20.54 33.80 13.36 20.44 27.14 28.82

95% 55.16 65.72 19.06 40.08 46.48 62.90

99.90% 80.02 88.14 30.70 58.52 60.00 119.74

Airport

TMA

Route planning

DTAXI

Domain RP
Appendix N: Oculometer Results 171

Appendix N: Oculometer Results

N.1 General Information

N.1.1 Interpreting the ANOVA

Analysis of Variance (ANOVA) identifies statistically significant variance across groups of data. 

ANOVA performs a statistical test to determine if the means of various groups are equal, generalizing a 

two-sample t-test to two or more groups. An adjusted P-value of 0.05 or less is considered significant, 

and is indicated by yellow highlighting in this appendix. The ANOVAs shown below are General Linear 

Model (GLM) ANOVAs, with the model: Condition, PF-PM, and Condition*PF-PM (interaction term). 

This produces results indicating variance across Condition, variance across PF-PM, and the interaction 

term of Condition crossed with PF-PM. The interaction term identifies if there is variance between the 

variance across condition within the PF group and the variance across condition in the PM group (i.e., if 

the observed variance under varying conditions followed a similar trend for each pilot or not).

N.1.2 Interpreting the Graphical Outputs

(1) Residual Plots

a) Histogram of residuals. An exploratory tool to show general characteristics of the data, 

including: 

 Typical values, spread or variation, and shape

 Unusual values in the data

Long tails in the plot may indicate skewness in the data. If one or two bars are far from the others, 

those points may be outliers. Because the appearance of the histogram changes depending on the number 

of intervals used to group the data, use the normal probability plot and goodness-of-fit tests to assess the 

normality of the residuals.

b) Normal plot of residuals. The points in this plot should generally form a straight line if the 

residuals are normally distributed. If the points on the plot depart from a straight line, the normality 

assumption may be invalid. If your data have fewer than 50 observations, the plot may display curvature 

in the tails even if the residuals are normally distributed. As the number of observations decreases, the 

probability plot may show substantial variation and nonlinearity even if the residuals are normally 

distributed. 

c) Residuals versus fits. This plot should show a random pattern of residuals on both sides of 0. If a 

point lies far from the majority of points, it may be an outlier. Also, there should not be any recognizable 

patterns in the residual plot. The following may indicate error that is not random:

 a series of increasing or decreasing points

 a predominance of positive residuals, or a predominance of negative residuals

 patterns, such as increasing residuals with increasing fits

d) Residuals versus order. This is a plot of all residuals in the order that the data was collected and 

can be used to find non-random error, especially of time-related effects. A positive correlation is 
Appendix N: Oculometer Results 172

indicated by a clustering of residuals with the same sign. A negative correlation is indicated by rapid 

changes in the signs of consecutive residuals. [40]

(2) Main Effects Plot

The main effects plot shows the average value for each main effect of the ANOVA model and draws a 

connecting line to emphasize the relative comparisons independently for PF /PM and Conditions. 

(3) Interaction Plot

The interaction plot shows the average values of the combined effects in the ANOVA model, helping 

to identify variance across Conditions grouped by PF and PM. 

N.2 Arrival: High altitude messages

General Linear Model: Percent head up versus Condition, PF - PM 

Factor Type Levels Values

Condition fixed 4 1 Voice/Paper, 2 Data/Paper, 3 Data/MMD, 4

 Data/MMD+Route

PF - PM fixed 2 PF, PM

Analysis of Variance for Percent Head Up (High Band), using Adjusted SS for

 Tests

Source DF Seq SS Adj SS Adj MS F P

Condition 3 4070.0 4018.3 1339.4 8.10 0.000

PF - PM 1 392.6 367.7 367.7 2.22 0.138

Condition*PF - PM 3 454.0 454.0 151.3 0.92 0.435

Error 151 24970.4 24970.4 165.4

Total 158 29887.0

S = 12.8595 R-Sq = 16.45% R-Sq(adj) = 12.58%

Unusual Observations for Percent Head Up (High Band)

Obs Percent Fit SE Fit Residual St Resid

23 59.9911 11.9312 2.8755 48.0599 3.83 R

50 52.6620 15.7882 2.8755 36.8738 2.94 R

58 58.5643 23.0469 2.8062 35.5174 2.83 R

91 48.5083 12.2729 2.8062 36.2354 2.89 R

92 48.5083 12.2729 2.8062 36.2354 2.89 R

93 37.1604 5.6591 2.8755 31.5013 2.51 R

112 54.9374 11.2222 2.8755 43.7152 3.49 R

R denotes an observation with a large standardized residual.

Tukey 95.0% Simultaneous Confidence Intervals
Appendix N: Oculometer Results 173

Response Variable Percent Head Up (High Band)

All Pairwise Comparisons among Levels of Condition

Condition = 1 Voice/Paper subtracted from:

Condition Lower Center Upper

2 Data/Paper -17.23 -9.86 -2.484

3 Data/MMD -21.45 -13.87 -6.285

4 Data/MMD+Route -15.26 -7.84 -0.423

Condition -+---------+---------+---------+-----

2 Data/Paper (------*-------)

3 Data/MMD (------*-------)

4 Data/MMD+Route (------*-------)

 -+---------+---------+---------+-----

 -20 -10 0 10

Condition = 2 Data/Paper subtracted from:

Condition Lower Center Upper -+---------+---------+---------+-----

3 Data/MMD -11.59 -4.009 3.572 (-------*-------)

4 Data/MMD+Route -5.40 2.016 9.434 (------*------)

 -+---------+---------+---------+-----

 -20 -10 0 10

Condition = 3 Data/MMD subtracted from:

Condition Lower Center Upper -+---------+---------+---------+-----

4 Data/MMD+Route -1.599 6.025 13.65 (-------*-------)

 -+---------+---------+---------+-----

 -20 -10 0 10

Tukey Simultaneous Tests

Response Variable Percent Head Up (High Band)

All Pairwise Comparisons among Levels of Condition

Condition = 1 Voice/Paper subtracted from:

 Difference SE of Adjusted

Condition of Means Difference T-Value P-Value

2 Data/Paper -9.86 2.841 -3.469 0.0038

3 Data/MMD -13.87 2.921 -4.746 0.0000

4 Data/MMD+Route -7.84 2.858 -2.743 0.0341

Condition = 2 Data/Paper subtracted from:

 Difference SE of Adjusted

Condition of Means Difference T-Value P-Value

3 Data/MMD -4.009 2.921 -1.372 0.5186

4 Data/MMD+Route 2.016 2.858 0.705 0.8949

Condition = 3 Data/MMD subtracted from:

 Difference SE of Adjusted

Condition of Means Difference T-Value P-Value

4 Data/MMD+Route 6.025 2.938 2.051 0.1743
Appendix N: Oculometer Results 174

Residual

Percent

-50 -25 0 25 50

99.9

99

90

50

10

1

0.1

Fitted Value

Residual

5 10 15 20 25

40

20

0

-20

Residual

Frequency

-20 -10 0 10 20 30 40 50

40

30

20

10

0

Observation Order

Residual

1 20 40 60 80 100 120 140 160

40

20

0

-20

Normal Probability Plot of the Residuals Residuals Versus the Fitted Values

Histogram of the Residuals Residuals Versus the Order of the Data

Residual Plots for Percent Head Out (Hi Band)

Figure 58. Residual plots for percent head up (16 – 14,000 feet MSL)

Mean of Percent Head Out (Hi Band)

4 Data/MMD+Route

3 Data/MMD

2 Data/Paper

1 Voice/Paper

20.0

17.5

15.0

12.5

10.0

7.5

5.0

PF

PM

Condition PF - PM

Main Effects Plot (fitted means) for Percent Head Out (Hi Band)

Figure 59. Main effects plot (fitted means) for percent head up (16 – 14,000 feet MSL)
Appendix N: Oculometer Results 175

PF - PM

Mean

PF PM

25

20

15

10

5

1 Voice/Paper

2 Data/Paper

3 Data/MMD

4 Data/MMD+Route

Condition

Interaction Plot (fitted means) for Percent Head Out (Hi Band)

Figure 60. Interaction plot (fitted means) for percent head up (16 – 14,000 feet MSL)

N.3 Arrival: Medium altitude messages

General Linear Model: Percent Head Up versus Condition, PF - PM 

Factor Type Levels Values

Condition fixed 4 1 Voice/Paper, 2 Data/Paper, 3 Data/MMD, 4

 Data/MMD+Route

PF - PM fixed 2 PF, PM

Analysis of Variance for Percent Head Up (Med Band), using Adjusted SS for

 Tests

Source DF Seq SS Adj SS Adj MS F P

Condition 3 2600.4 2579.9 860.0 3.96 0.009

PF - PM 1 481.4 486.0 486.0 2.24 0.137

Condition*PF - PM 3 578.8 578.8 192.9 0.89 0.449

Error 151 32826.6 32826.6 217.4

Total 158 36487.2

S = 14.7443 R-Sq = 10.03% R-Sq(adj) = 5.86%

Unusual Observations for Percent Head Up (Med Band)

Obs Percent Fit SE Fit Residual St Resid

58 56.6589 22.8489 3.2175 33.8100 2.35 R
Appendix N: Oculometer Results 176

80 62.8501 12.4426 3.2969 50.4075 3.51 R

89 61.0188 22.8489 3.2175 38.1699 2.65 R

113 57.8865 16.1795 3.2969 41.7070 2.90 R

116 45.9061 13.7628 3.2969 32.1433 2.24 R

124 41.2925 11.0928 3.2175 30.1997 2.10 R

141 60.6329 12.1394 3.2969 48.4935 3.37 R

145 61.7805 16.1795 3.2969 45.6010 3.17 R

146 50.8180 16.1795 3.2969 34.6385 2.41 R

147 43.0785 13.7628 3.2969 29.3157 2.04 R

149 56.2240 8.5562 3.5760 47.6678 3.33 R

R denotes an observation with a large standardized residual.

Tukey 95.0% Simultaneous Confidence Intervals

Response Variable Percent Head Up (Med Band)

All Pairwise Comparisons among Levels of Condition

Condition = 1 Voice/Paper subtracted from:

Condition Lower Center Upper

2 Data/Paper -15.54 -7.09 1.367

3 Data/MMD -17.86 -9.17 -0.474

4 Data/MMD+Route -18.79 -10.29 -1.782

Condition ---+---------+---------+---------+---

2 Data/Paper (---------*----------)

3 Data/MMD (----------*---------)

4 Data/MMD+Route (---------*----------)

 ---+---------+---------+---------+---

 -16.0 -8.0 0.0 8.0

Condition = 2 Data/Paper subtracted from:

Condition Lower Center Upper ---+---------+---------+---------+---

3 Data/MMD -10.77 -2.080 6.612 (---------*----------)

4 Data/MMD+Route -11.71 -3.201 5.304 (----------*----------)

 ---+---------+---------+---------+---

 -16.0 -8.0 0.0 8.0

Condition = 3 Data/MMD subtracted from:

Condition Lower Center Upper ---+---------+---------+---------+---

4 Data/MMD+Route -9.863 -1.121 7.622 (----------*----------)

 ---+---------+---------+---------+---

 -16.0 -8.0 0.0 8.0

Tukey Simultaneous Tests

Response Variable Percent Head Up (Med Band)

All Pairwise Comparisons among Levels of Condition

Condition = 1 Voice/Paper subtracted from:

 Difference SE of Adjusted

Condition of Means Difference T-Value P-Value

2 Data/Paper -7.09 3.257 -2.175 0.1348

3 Data/MMD -9.17 3.350 -2.737 0.0347

4 Data/MMD+Route -10.29 3.277 -3.139 0.0109

Condition = 2 Data/Paper subtracted from:
Appendix N: Oculometer Results 177

 Difference SE of Adjusted

Condition of Means Difference T-Value P-Value

3 Data/MMD -2.080 3.350 -0.6210 0.9252

4 Data/MMD+Route -3.201 3.277 -0.9766 0.7630

Condition = 3 Data/MMD subtracted from:

 Difference SE of Adjusted

Condition of Means Difference T-Value P-Value

4 Data/MMD+Route -1.121 3.369 -0.3326 0.9873

Residual

Percent

-50 -25 0 25 50

99.9

99

90

50

10

1

0.1

Fitted Value

Residual

5 10 15 20 25

60

40

20

0

-20

Residual

Frequency

-20 -10 0 10 20 30 40 50

40

30

20

10

0

Observation Order

Residual

1 20 40 60 80 100 120 140 160

60

40

20

0

-20

Normal Probability Plot of the Residuals Residuals Versus the Fitted Values

Histogram of the Residuals Residuals Versus the Order of the Data

Residual Plots for Percent Head Out (Med Band)

Figure 61. Residual plots for percent head up (10 – 8,000 feet MSL)
Appendix N: Oculometer Results 178

Mean of Percent Head Out (Med Band)

4 Data/MMD+Route

3 Data/MMD

2 Data/Paper

1 Voice/Paper

20

18

16

14

12

10

PF

PM

Condition PF - PM

Main Effects Plot (fitted means) for Percent Head Out (Med Band)

Figure 62. Main effects plot (fitted means) for percent head up (10 – 8,000 feet MSL)

PF - PM

Mean

PF PM

25

20

15

10

5

1 Voice/Paper

2 Data/Paper

3 Data/MMD

4 Data/MMD+Route

Condition

Interaction Plot (fitted means) for Percent Head Out (Med Band)

Figure 63. Interaction plot (fitted means) for percent head up (10 – 8,000 feet MSL)
Appendix N: Oculometer Results 179

N.4 Arrival: Low altitude messages

General Linear Model: Percent Head Up versus Condition, PF - PM 

Factor Type Levels Values

Condition fixed 4 1 Voice/Paper, 2 Data/Paper, 3 Data/MMD, 4

 Data/MMD+Route

PF - PM fixed 2 PF, PM

Analysis of Variance for Percent Head Up (Low Band), using Adjusted SS for

 Tests

Source DF Seq SS Adj SS Adj MS F P

Condition 3 2326.5 2288.0 762.7 4.46 0.005

PF - PM 1 702.9 699.6 699.6 4.09 0.045

Condition*PF - PM 3 476.8 476.8 158.9 0.93 0.428

Error 151 25805.2 25805.2 170.9

Total 158 29311.5

S = 13.0727 R-Sq = 11.96% R-Sq(adj) = 7.88%

Unusual Observations for Percent Head Up (Low Band)

Obs Percent Fit SE Fit Residual St Resid

 1 43.8347 10.5682 2.9231 33.2665 2.61 R

55 38.9768 13.4396 2.9231 25.5372 2.00 R

58 50.9821 19.3774 2.8527 31.6047 2.48 R

61 47.8629 16.1893 2.9231 31.6736 2.49 R

95 47.8224 12.6623 2.9231 35.1601 2.76 R

118 39.7289 11.0439 3.1706 28.6850 2.26 R

138 61.4298 19.3774 2.8527 42.0524 3.30 R

150 58.0565 11.0439 3.1706 47.0126 3.71 R

172 43.4291 7.1894 2.8527 36.2397 2.84 R

174 49.3665 16.1893 2.9231 33.1772 2.60 R

175 44.7406 12.6623 2.9231 32.0783 2.52 R

R denotes an observation with a large standardized residual.

Tukey 95.0% Simultaneous Confidence Intervals

Response Variable Percent Head Up (Low Band)

All Pairwise Comparisons among Levels of Condition

Condition = 1 Voice/Paper subtracted from:

Condition Lower Center Upper -------+---------+---------+---------

2 Data/Paper -17.10 -9.601 -2.106 (------*-------)

3 Data/MMD -9.06 -1.356 6.351 (-------*------)

4 Data/MMD+Route -9.46 -1.922 5.619 (------*-------)

 -------+---------+---------+---------

 -10 0 10

Condition = 2 Data/Paper subtracted from:

Condition Lower Center Upper -------+---------+---------+---------

3 Data/MMD 0.5381 8.245 15.95 (------*-------)

4 Data/MMD+Route 0.1389 7.679 15.22 (-------*------)

 -------+---------+---------+---------

 -10 0 10

Condition = 3 Data/MMD subtracted from:
Appendix N: Oculometer Results 180

Condition Lower Center Upper -------+---------+---------+---------

4 Data/MMD+Route -8.317 -0.5657 7.186 (------*-------)

 -------+---------+---------+---------

 -10 0 10

Tukey Simultaneous Tests

Response Variable Percent Head Up (Low Band)

All Pairwise Comparisons among Levels of Condition

Condition = 1 Voice/Paper subtracted from:

 Difference SE of Adjusted

Condition of Means Difference T-Value P-Value

2 Data/Paper -9.601 2.888 -3.324 0.0061

3 Data/MMD -1.356 2.970 -0.457 0.9682

4 Data/MMD+Route -1.922 2.906 -0.661 0.9114

Condition = 2 Data/Paper subtracted from:

 Difference SE of Adjusted

Condition of Means Difference T-Value P-Value

3 Data/MMD 8.245 2.970 2.776 0.0312

4 Data/MMD+Route 7.679 2.906 2.643 0.0445

Condition = 3 Data/MMD subtracted from:

 Difference SE of Adjusted

Condition of Means Difference T-Value P-Value

4 Data/MMD+Route -0.5657 2.987 -0.1894 0.9976

Residual

Percent

-50 -25 0 25 50

99.9

99

90

50

10

1

0.1

Fitted Value

Residual

5 10 15 20

50

25

0

Residual

Frequency

-20 -10 0 10 20 30 40

40

30

20

10

0

Observation Order

Residual

1 20 40 60 80 100 120 140 160

50

25

0

Normal Probability Plot of the Residuals Residuals Versus the Fitted Values

Histogram of the Residuals Residuals Versus the Order of the Data

Residual Plots for Percent Head Out (Low Band)

Figure 64. Residual plots for percent head up (7 – 5,000 feet MSL)
Appendix N: Oculometer Results 181

Mean of Percent Head Out (Low Band)

4 Data/MMD+Route

3 Data/MMD

2 Data/Paper

1 Voice/Paper

15.0

12.5

10.0

7.5

5.0

PF

PM

Condition PF - PM

Main Effects Plot (fitted means) for Percent Head Out (Low Band)

Figure 65. Main effects plot (fitted means) for percent head up (7 – 5,000 feet MSL)

PF - PM

Mean

PF PM

20.0

17.5

15.0

12.5

10.0

7.5

5.0

1 Voice/Paper

2 Data/Paper

3 Data/MMD

4 Data/MMD+Route

Condition

Interaction Plot (fitted means) for Percent Head Out (Low Band)

Figure 66. Interaction plot (fitted means) for percent head up (7 – 5,000 feet MSL)
Appendix N: Oculometer Results 182

N.5 Arrival: Taxi operations

Taxi operations during an arrival scenario began once the aircraft slowed below 80 KIAS during 

landing roll-out for oculometer data analysis.

General Linear Model: Percent head up versus Condition, PF - PM 

Factor Type Levels Values

Condition fixed 4 1 Voice/Paper, 2 Data/Paper, 3 Data/MMD, 4

 Data/MMD+Route

PF - PM fixed 2 PF, PM

Analysis of Variance for Percent Head Up (below 80 knots), using Adjusted SS for

 Tests

Source DF Seq SS Adj SS Adj MS F P

Condition 3 3155.6 2997.1 999.0 4.47 0.005

PF - PM 1 19915.9 20100.5 20100.5 89.89 0.000

Condition*PF - PM 3 2790.3 2790.3 930.1 4.16 0.007

Error 147 32869.5 32869.5 223.6

Total 154 58731.3

S = 14.9533 R-Sq = 44.03% R-Sq(adj) = 41.37%

Unusual Observations for Percent Head Up (below 80 knots)

Obs Percent Fit SE Fit Residual St Resid

17 83.4763 53.7174 3.3437 29.7589 2.04 R

34 23.8551 53.7174 3.3437 -29.8623 -2.05 R

50 89.2779 53.7174 3.3437 35.5605 2.44 R

72 87.2655 53.8645 3.4305 33.4010 2.29 R

99 28.0447 59.5175 3.3437 -31.4728 -2.16 R

135 22.5296 53.8645 3.4305 -31.3349 -2.15 R

145 17.6622 53.7174 3.3437 -36.0552 -2.47 R

154 13.0155 44.2720 3.2631 -31.2565 -2.14 R

176 53.6511 22.4910 3.4305 31.1601 2.14 R

R denotes an observation with a large standardized residual.

Tukey 95.0% Simultaneous Confidence Intervals

Response Variable Percent Head Up (below 80 knots)

All Pairwise Comparisons among Levels of Condition

Condition = 1 Voice/Paper subtracted from:

Condition Lower Center Upper ----+---------+---------+---------+--

2 Data/Paper -12.02 -3.42 5.174 (----------*---------)

3 Data/MMD -18.31 -9.33 -0.359 (----------*-----------)

4 Data/MMD+Route -19.58 -10.82 -2.054 (---------*----------)

 ----+---------+---------+---------+--

 -16.0 -8.0 0.0 8.0

Condition = 2 Data/Paper subtracted from:

Condition Lower Center Upper ----+---------+---------+---------+--

3 Data/MMD -14.88 -5.910 3.063 (-----------*----------)

4 Data/MMD+Route -16.16 -7.394 1.369 (----------*----------)
Appendix N: Oculometer Results 183

 ----+---------+---------+---------+--

 -16.0 -8.0 0.0 8.0

Condition = 3 Data/MMD subtracted from:

Condition Lower Center Upper ----+---------+---------+---------+--

4 Data/MMD+Route -10.62 -1.484 7.650 (----------*-----------)

 ----+---------+---------+---------+--

 -16.0 -8.0 0.0 8.0

Tukey Simultaneous Tests

Response Variable Percent Head Up (below 80 knots)

All Pairwise Comparisons among Levels of Condition

Condition = 1 Voice/Paper subtracted from:

 Difference SE of Adjusted

Condition of Means Difference T-Value P-Value

2 Data/Paper -3.42 3.304 -1.036 0.7286

3 Data/MMD -9.33 3.449 -2.706 0.0378

4 Data/MMD+Route -10.82 3.368 -3.212 0.0087

Condition = 2 Data/Paper subtracted from:

 Difference SE of Adjusted

Condition of Means Difference T-Value P-Value

3 Data/MMD -5.910 3.449 -1.714 0.3202

4 Data/MMD+Route -7.394 3.368 -2.196 0.1292

Condition = 3 Data/MMD subtracted from:

 Difference SE of Adjusted

Condition of Means Difference T-Value P-Value

4 Data/MMD+Route -1.484 3.510 -0.4228 0.9745

Tukey 95.0% Simultaneous Confidence Intervals

Response Variable Percent Head Up (below 80 knots)

All Pairwise Comparisons among Levels of PF - PM

PF - PM = PF subtracted from:

PF

-

PM Lower Center Upper -----+---------+---------+---------+-

PM -27.61 -22.85 -18.09 (-----*-----)

 -----+---------+---------+---------+-

 -24.0 -16.0 -8.0 0.0

Tukey Simultaneous Tests

Response Variable Percent Head Up (below 80 knots)

All Pairwise Comparisons among Levels of PF - PM

PF - PM = PF subtracted from:

PF

- Difference SE of Adjusted

PM of Means Difference T-Value P-Value

PM -22.85 2.410 -9.481 0.0000

Tukey 95.0% Simultaneous Confidence Intervals

Response Variable Percent Head Up (below 80 knots)
Appendix N: Oculometer Results 184

All Pairwise Comparisons among Levels of Condition*PF - PM

Condition = 1 Voice/Paper

PF - PM = PF subtracted from:

 PF

-

Condition PM Lower Center Upper

1 Voice/Paper PM -23.82 -9.45 4.93

2 Data/Paper PF -8.74 5.80 20.34

2 Data/Paper PM -36.46 -22.09 -7.72

3 Data/MMD PF -18.14 -2.71 12.72

3 Data/MMD PM -40.14 -25.40 -10.67

4 Data/MMD+Route PF -14.59 0.15 14.88

4 Data/MMD+Route PM -45.96 -31.23 -16.49

 PF

-

Condition PM -+---------+---------+---------+-----

1 Voice/Paper PM (-----*-----)

2 Data/Paper PF (----*-----)

2 Data/Paper PM (-----*-----)

3 Data/MMD PF (-----*-----)

3 Data/MMD PM (-----*-----)

4 Data/MMD+Route PF (-----*-----)

4 Data/MMD+Route PM (-----*----)

 -+---------+---------+---------+-----

 -50 -25 0 25

Condition = 1 Voice/Paper

PF - PM = PM subtracted from:

 PF

-

Condition PM Lower Center Upper

2 Data/Paper PF 0.87 15.25 29.616

2 Data/Paper PM -26.84 -12.65 1.549

3 Data/MMD PF -8.53 6.74 21.999

3 Data/MMD PM -30.52 -15.96 -1.394

4 Data/MMD+Route PF -4.97 9.59 24.156

4 Data/MMD+Route PM -36.34 -21.78 -7.218

 PF

 -

Condition PM -+---------+---------+---------+-----

2 Data/Paper PF (-----*-----)

2 Data/Paper PM (-----*-----)

3 Data/MMD PF (-----*-----)

3 Data/MMD PM (-----*----)

4 Data/MMD+Route PF (-----*-----)

4 Data/MMD+Route PM (-----*-----)

 -+---------+---------+---------+-----

 -50 -25 0 25

Condition = 2 Data/Paper

PF - PM = PF subtracted from:

 PF

-

Condition PM Lower Center Upper

2 Data/Paper PM -42.26 -27.89 -13.52

3 Data/MMD PF -23.94 -8.51 6.92
Appendix N: Oculometer Results 185

3 Data/MMD PM -45.94 -31.20 -16.47

4 Data/MMD+Route PF -20.39 -5.65 9.08

4 Data/MMD+Route PM -51.76 -37.03 -22.29

 PF

-

Condition PM -+---------+---------+---------+-----

2 Data/Paper PM (-----*-----)

3 Data/MMD PF (------*-----)

3 Data/MMD PM (-----*----)

4 Data/MMD+Route PF (-----*-----)

4 Data/MMD+Route PM (-----*-----)

 -+---------+---------+---------+-----

 -50 -25 0 25

Condition = 2 Data/Paper

PF - PM = PM subtracted from:

 PF

-

Condition PM Lower Center Upper

3 Data/MMD PF 4.12 19.381 34.645

3 Data/MMD PM -17.87 -3.312 11.252

4 Data/MMD+Route PF 7.67 22.238 36.801

4 Data/MMD+Route PM -23.70 -9.136 5.427

 PF

-

Condition PM -+---------+---------+---------+-----

3 Data/MMD PF (-----*-----)

3 Data/MMD PM (-----*-----)

4 Data/MMD+Route PF (-----*-----)

4 Data/MMD+Route PM (----*-----)

 -+---------+---------+---------+-----

 -50 -25 0 25

Condition = 3 Data/MMD

PF - PM = PF subtracted from:

 PF

-

Condition PM Lower Center Upper

3 Data/MMD PM -38.30 -22.69 -7.09

4 Data/MMD+Route PF -12.75 2.86 18.46

4 Data/MMD+Route PM -44.12 -28.52 -12.91

 PF

-

Condition PM -+---------+---------+---------+-----

3 Data/MMD PM (-----*-----)

4 Data/MMD+Route PF (-----*-----)

4 Data/MMD+Route PM (------*-----)

 -+---------+---------+---------+-----

 -50 -25 0 25

Condition = 3 Data/MMD

PF - PM = PM subtracted from:

 PF

-
Appendix N: Oculometer Results 186

Condition PM Lower Center Upper

4 Data/MMD+Route PF 10.63 25.549 40.472

4 Data/MMD+Route PM -20.75 -5.824 9.099

 PF

-

Condition PM -+---------+---------+---------+-----

4 Data/MMD+Route PF (-----*-----)

4 Data/MMD+Route PM (-----*-----)

 -+---------+---------+---------+-----

 -50 -25 0 25

Condition = 4 Data/MMD+Route

PF - PM = PF subtracted from:

 PF

-

Condition PM Lower Center Upper

4 Data/MMD+Route PM -46.30 -31.37 -16.45

 PF

-

Condition PM -+---------+---------+---------+-----

4 Data/MMD+Route PM (-----*-----)

 -+---------+---------+---------+-----

 -50 -25 0 25

Tukey Simultaneous Tests – (Including Interaction Comparison)

Response Variable Percent Head Up (below 80 knots)

All Pairwise Comparisons among Levels of Condition*PF - PM

Condition = 1 Voice/Paper

PF - PM = PF subtracted from:

 PF

- Difference SE of Adjusted

Condition PM of Means Difference T-Value P-Value

1 Voice/Paper PM -9.45 4.672 -2.022 0.4710

2 Data/Paper PF 5.80 4.729 1.227 0.9228

2 Data/Paper PM -22.09 4.672 -4.728 0.0002

3 Data/MMD PF -2.71 5.016 -0.540 0.9994

3 Data/MMD PM -25.40 4.790 -5.303 0.0000

4 Data/MMD+Route PF 0.15 4.790 0.031 1.0000

4 Data/MMD+Route PM -31.23 4.790 -6.518 0.0000

Condition = 1 Voice/Paper

PF - PM = PM subtracted from:

 PF

- Difference SE of Adjusted

Condition PM of Means Difference T-Value P-Value

2 Data/Paper PF 15.25 4.672 3.263 0.0291

2 Data/Paper PM -12.65 4.615 -2.740 0.1190

3 Data/MMD PF 6.74 4.962 1.358 0.8747

3 Data/MMD PM -15.96 4.735 -3.370 0.0210

4 Data/MMD+Route PF 9.59 4.735 2.026 0.4681

4 Data/MMD+Route PM -21.78 4.735 -4.600 0.0003

Condition = 2 Data/Paper

PF - PM = PF subtracted from:
Appendix N: Oculometer Results 187

 PF

- Difference SE of Adjusted

Condition PM of Means Difference T-Value P-Value

2 Data/Paper PM -27.89 4.672 -5.970 0.0000

3 Data/MMD PF -8.51 5.016 -1.697 0.6895

3 Data/MMD PM -31.20 4.790 -6.513 0.0000

4 Data/MMD+Route PF -5.65 4.790 -1.180 0.9364

4 Data/MMD+Route PM -37.03 4.790 -7.729 0.0000

Condition = 2 Data/Paper

PF - PM = PM subtracted from:

 PF

- Difference SE of Adjusted

Condition PM of Means Difference T-Value P-Value

3 Data/MMD PF 19.381 4.962 3.906 0.0035

3 Data/MMD PM -3.312 4.735 -0.699 0.9969

4 Data/MMD+Route PF 22.238 4.735 4.697 0.0002

4 Data/MMD+Route PM -9.136 4.735 -1.930 0.5330

Condition = 3 Data/MMD

PF - PM = PF subtracted from:

 PF

- Difference SE of Adjusted

Condition PM of Means Difference T-Value P-Value

3 Data/MMD PM -22.69 5.074 -4.473 0.0004

4 Data/MMD+Route PF 2.86 5.074 0.563 0.9992

4 Data/MMD+Route PM -28.52 5.074 -5.620 0.0000

Condition = 3 Data/MMD

PF - PM = PM subtracted from:

 PF

- Difference SE of Adjusted

Condition PM of Means Difference T-Value P-Value

4 Data/MMD+Route PF 25.549 4.852 5.266 0.0000

4 Data/MMD+Route PM -5.824 4.852 -1.200 0.9307

Condition = 4 Data/MMD+Route

PF - PM = PF subtracted from:

 PF

- Difference SE of Adjusted

Condition PM of Means Difference T-Value P-Value

4 Data/MMD+Route PM -31.37 4.852 -6.467 0.0000
Appendix N: Oculometer Results 188

Residual

Percent

-50 -25 0 25 50

99.9

99

90

50

10

1

0.1

Fitted Value

Residual

20 30 40 50 60

40

20

0

-20

-40

Residual

Frequency

-30 -20 -10 0 10 20 30

20

15

10

5

0

Observation Order

Residual

1 20 40 60 80 100 120 140 160

40

20

0

-20

-40

Normal Probability Plot of the Residuals Residuals Versus the Fitted Values

Histogram of the Residuals Residuals Versus the Order of the Data

Residual Plots for Percent Head Out (below 80 kts)

Figure 67. Residual plots for percent head up (below 80 knots)

Mean of Percent Head Out (below 80 kts)

4 Data/MMD+Route

3 Data/MMD

2 Data/Paper

1 Voice/Paper

55

50

45

40

35

30

PF

PM

Condition PF - PM

Main Effects Plot (fitted means) for Percent Head Out (below 80 kts)

Figure 68. Main effects plot (fitted means) for percent head up (below 80 knots)
Appendix N: Oculometer Results 189

PF - PM

Mean

PF PM

60

50

40

30

20

1 Voice/Paper

2 Data/Paper

3 Data/MMD

4 Data/MMD+Route

Condition

Interaction Plot (fitted means) for Percent Head Out (below 80 kts)

Figure 69. Interaction plot (fitted means) for percent head up (below 80 knots)
Appendix N: Oculometer Results 190

N.6 Departure: entire scenario

General Linear Model: Percent head up versus Condition, PF - PM 

Factor Type Levels Values

Condition fixed 4 1 Voice/Paper, 2 Data/Paper, 3 Data/MMD, 4

 Data/MMD+Route

PF - PM fixed 2 PF, PM

Analysis of Variance for Percent Head Up (Entire Run), using Adjusted SS for

 Tests

Source DF Seq SS Adj SS Adj MS F P

Condition 3 2732.9 2659.7 886.6 11.08 0.000

PF - PM 1 15732.4 15660.6 15660.6 195.70 0.000

Condition*PF - PM 3 488.0 488.0 162.7 2.03 0.112

Error 152 12163.2 12163.2 80.0

Total 159 31116.5

S = 8.94547 R-Sq = 60.91% R-Sq(adj) = 59.11%

Unusual Observations for Percent Head Up (Entire Run)

Obs Percent Fit SE Fit Residual St Resid

 2 53.2186 71.2814 2.0003 -18.0628 -2.07 R

10 31.9115 53.1752 1.9521 -21.2637 -2.44 R

11 24.9345 42.5773 1.9521 -17.6428 -2.02 R

50 90.6775 71.2814 2.0003 19.3961 2.22 R

58 71.6278 53.1752 1.9521 18.4526 2.11 R

60 61.6453 42.5773 1.9521 19.0680 2.18 R

124 22.7567 42.5773 1.9521 -19.8206 -2.27 R

153 28.0458 53.1752 1.9521 -25.1294 -2.88 R

170 71.3411 53.1752 1.9521 18.1659 2.08 R

174 60.4640 39.8905 2.0003 20.5735 2.36 R

176 66.1148 46.7586 2.0522 19.3562 2.22 R

R denotes an observation with a large standardized residual.

Tukey 95.0% Simultaneous Confidence Intervals

Response Variable Percent Head Up (Entire Run)

All Pairwise Comparisons among Levels of Condition

Condition = 1 Voice/Paper subtracted from:

Condition Lower Center Upper

2 Data/Paper -12.99 -7.89 -2.795

3 Data/MMD -16.21 -10.98 -5.747

4 Data/MMD+Route -13.21 -8.01 -2.821

Condition ---+---------+---------+---------+---

2 Data/Paper (-------*------)

3 Data/MMD (------*-------)

4 Data/MMD+Route (-------*------)

 ---+---------+---------+---------+---

 -14.0 -7.0 0.0 7.0

Condition = 2 Data/Paper subtracted from:
Appendix N: Oculometer Results 191

Condition Lower Center Upper ---+---------+---------+---------+---

3 Data/MMD -8.288 -3.087 2.114 (-------*------)

4 Data/MMD+Route -5.285 -0.122 5.041 (-------*------)

 ---+---------+---------+---------+---

 -14.0 -7.0 0.0 7.0

Condition = 3 Data/MMD subtracted from:

Condition Lower Center Upper ---+---------+---------+---------+---

4 Data/MMD+Route -2.331 2.965 8.261 (------*-------)

 ---+---------+---------+---------+---

 -14.0 -7.0 0.0 7.0

Tukey Simultaneous Tests

Response Variable Percent Head Up (Entire Run)

All Pairwise Comparisons among Levels of Condition

Condition = 1 Voice/Paper subtracted from:

 Difference SE of Adjusted

Condition of Means Difference T-Value P-Value

2 Data/Paper -7.89 1.964 -4.018 0.0005

3 Data/MMD -10.98 2.016 -5.446 0.0000

4 Data/MMD+Route -8.01 2.002 -4.004 0.0006

Condition = 2 Data/Paper subtracted from:

 Difference SE of Adjusted

Condition of Means Difference T-Value P-Value

3 Data/MMD -3.087 2.004 -1.540 0.4162

4 Data/MMD+Route -0.122 1.990 -0.061 0.9999

Condition = 3 Data/MMD subtracted from:

 Difference SE of Adjusted

Condition of Means Difference T-Value P-Value

4 Data/MMD+Route 2.965 2.041 1.453 0.4688
Appendix N: Oculometer Results 192

Residual

Percent

-30 -15 0 15 30

99.9

99

90

50

10

1

0.1

Fitted Value

Residual

40 50 60 70

20

10

0

-10

-20

Residual

Frequency

-22.5 -15.0 -7.5 0.0 7.5 15.0

20

15

10

5

0

Observation Order

Residual

1 20 40 60 80 100 120 140 160

20

10

0

-10

-20

Normal Probability Plot of the Residuals Residuals Versus the Fitted Values

Histogram of the Residuals Residuals Versus the Order of the Data

Residual Plots for Percent Head Out (Entire Run)

Figure 70. Residual plots for percent head up (entire run)

Mean of Percent Head Out (Entire Run)

4 Data/MMD+Route

3 Data/MMD

2 Data/Paper

1 Voice/Paper

65

60

55

50

45

PF

PM

Condition PF - PM

Main Effects Plot (fitted means) for Percent Head Out (Entire Run)

Figure 71. Main effects plot (fitted means) for percent head up (entire run)
Appendix N: Oculometer Results 193

PF - PM

Mean

PF PM

75

70

65

60

55

50

45

40

1 Voice/Paper

2 Data/Paper

3 Data/MMD

4 Data/MMD+Route

Condition

Interaction Plot (fitted means) for Percent Head Out (Entire Run)

Figure 72. Interaction plot (fitted means) for percent head up (entire run)

N.7 Arrival versus Departure ANOVA

General Linear Model: Percent head up versus PF - PM, Phase, Condition 

Factor Type Levels Values

PF - PM fixed 2 PF, PM

Phase fixed 2 Arr, Dep

Condition fixed 4 1Voice/Paper, 2Data/Paper, 4DCom/MMD, 4DCom/Rte

Analysis of Variance for Percent Head Up (Entire Run), using Adjusted SS for

 Tests

Source DF Seq SS Adj SS Adj MS F P

PF - PM 1 36887.0 36472.4 36472.4 213.77 0.000

Phase 1 10259.1 11372.9 11372.9 66.66 0.000

Condition 3 6543.2 6127.7 2042.6 11.97 0.000

PF - PM*Phase 1 218.1 244.9 244.9 1.44 0.232

PF - PM*Condition 3 1611.4 1615.7 538.6 3.16 0.025

Phase*Condition 3 684.7 684.7 228.2 1.34 0.262

Error 320 54596.3 54596.3 170.6

Total 332 110799.9

S = 13.0619 R-Sq = 50.73% R-Sq(adj) = 48.88%

Unusual Observations for Percent Head Up (Entire Run)

Obs Percent Fit SE Fit Residual St Resid
Appendix N: Oculometer Results 194

63 11.5143 58.9509 2.3223 -47.4366 -3.69 R

99 14.2824 58.9509 2.3223 -44.6685 -3.48 R

162 4.0331 39.1154 2.2540 -35.0823 -2.73 R

181 28.0458 56.1312 2.5765 -28.0854 -2.19 R

208 83.4763 56.8212 2.6254 26.6551 2.08 R

218 23.8551 56.8212 2.6254 -32.9661 -2.58 R

227 89.2779 56.8212 2.6254 32.4567 2.54 R

242 87.2655 50.5721 2.6908 36.6934 2.87 R

253 31.0100 56.8212 2.6254 -25.8112 -2.02 R

255 28.0447 59.2803 2.6228 -31.2356 -2.44 R

277 22.5296 50.5721 2.6908 -28.0425 -2.19 R

280 17.6622 56.8212 2.6254 -39.1590 -3.06 R

327 58.1727 31.8527 2.5742 26.3200 2.06 R

380 13.0155 41.3160 2.5765 -28.3005 -2.21 R

395 53.6511 25.7834 2.6908 27.8677 2.18 R

R denotes an observation with a large standardized residual.

Tukey 95.0% Simultaneous Confidence Intervals

Response Variable Percent Head Up (Entire Run)

All Pairwise Comparisons among Levels of PF - PM

PF - PM = PF subtracted from:

PF

-

PM Lower Center Upper ----+---------+---------+---------+--

PM -23.85 -21.02 -18.20 (---*---)

 ----+---------+---------+---------+--

 -21.0 -14.0 -7.0 0.0

Tukey Simultaneous Tests

Response Variable Percent Head Up (Entire Run)

All Pairwise Comparisons among Levels of PF - PM

PF - PM = PF subtracted from:

PF

- Difference SE of Adjusted

PM of Means Difference T-Value P-Value

PM -21.02 1.438 -14.62 0.0000

Tukey 95.0% Simultaneous Confidence Intervals

Response Variable Percent Head Up (Entire Run)

All Pairwise Comparisons among Levels of Phase

Phase = Arr subtracted from:

Phase Lower Center Upper ----+---------+---------+---------+--

Dep 8.948 11.79 14.63 (-----------------*----------------)

 ----+---------+---------+---------+--

 9.6 11.2 12.8 14.4

Tukey Simultaneous Tests

Response Variable Percent Head Up (Entire Run)

All Pairwise Comparisons among Levels of Phase

Phase = Arr subtracted from:

 Difference SE of Adjusted

Phase of Means Difference T-Value P-Value

Dep 11.79 1.444 8.165 0.0000
Appendix N: Oculometer Results 195

Tukey 95.0% Simultaneous Confidence Intervals

Response Variable Percent Head Up (Entire Run)

All Pairwise Comparisons among Levels of Condition

Condition = 1Voice/Paper subtracted from:

Condition Lower Center Upper ---+---------+---------+---------+---

2Data/Paper -10.88 -5.66 -0.439 (-------*------)

4DCom/MMD -16.43 -11.25 -6.073 (------*------)

4DCom/Rte -14.78 -9.46 -4.135 (------*-------)

 ---+---------+---------+---------+---

 -14.0 -7.0 0.0 7.0

Condition = 2Data/Paper subtracted from:

Condition Lower Center Upper ---+---------+---------+---------+---

4DCom/MMD -10.75 -5.591 -0.4292 (------*------)

4DCom/Rte -9.10 -3.797 1.5090 (-------*------)

 ---+---------+---------+---------+---

 -14.0 -7.0 0.0 7.0

Condition = 4DCom/MMD subtracted from:

Condition Lower Center Upper ---+---------+---------+---------+---

4DCom/Rte -3.469 1.794 7.057 (-------*------)

 ---+---------+---------+---------+---

 -14.0 -7.0 0.0 7.0

Tukey Simultaneous Tests

Response Variable Percent Head Up (Entire Run)

All Pairwise Comparisons among Levels of Condition

Condition = 1Voice/Paper subtracted from:

 Difference SE of Adjusted

Condition of Means Difference T-Value P-Value

2Data/Paper -5.66 2.034 -2.783 0.0277

4DCom/MMD -11.25 2.017 -5.577 0.0000

4DCom/Rte -9.46 2.073 -4.561 0.0000

Condition = 2Data/Paper subtracted from:

 Difference SE of Adjusted

Condition of Means Difference T-Value P-Value

4DCom/MMD -5.591 2.011 -2.780 0.0278

4DCom/Rte -3.797 2.067 -1.837 0.2560

Condition = 4DCom/MMD subtracted from:

 Difference SE of Adjusted

Condition of Means Difference T-Value P-Value

4DCom/Rte 1.794 2.050 0.8750 0.8178
Appendix N: Oculometer Results 196

Residual

Percent

-50 -25 0 25 50

99.9

99

90

50

10

1

0.1

Fitted Value

Residual

30 40 50 60 70

40

20

0

-20

-40

Residual

Frequency

-45 -30 -15 0 15 30

60

45

30

15

0

Observation Order

Residual

1 50 100 150 200 250 300 350

40

20

0

-20

-40

Normal Probability Plot of the Residuals Residuals Versus the Fitted Values

Histogram of the Residuals Residuals Versus the Order of the Data

Residual Plots for Percent Head Out (Entire Run)

Figure 73. Residual plots for percent head up (entire run)

Mean of Percent Head Out (Entire Run)

PF PM

60

55

50

45

40

Arr Dep

1Voice/Paper 2Data/Paper 4DCom/MMD 4DCom/Rte

60

55

50

45

40

PF - PM Phase

Condition

Main Effects Plot (fitted means) for Percent Head Out (Entire Run)

Figure 74. Main effects plot (fitted means) for percent head up (entire run)
Appendix N: Oculometer Results 197

PF - PM

Phase

Condition

DepA r4DC om/Rte4DC om/MMD2Data/Paper1Voice/Paper

60

45

30

60

45

30

PF

PM

PF - PM

Arr

Dep

Phase

Interaction Plot (fitted means) for Percent Head Out (Entire Run)

Figure 75. Interaction plot (fitted means) for percent head up (entire run)
Appendix O: Post-Scenario Results 198

Appendix O: Post-Scenario Questionnaire Results

This Appendix presents results from the Post-Scenario Questionnaire (Appendix D). Data collected 

and analyzed for in flight operations occurred only in arrival scenarios, and surface operations occurred in

both arrival and departure scenarios (departure scenarios terminated prior to takeoff). Therefore, for this 

experiment, “surface operations” and “taxi operations” are synonymous.

O.1 Workload (Bedford) rating

1) Your workload in-flight during arrivals 2) Your workload during surface operations

 1 is “workload insignificant”, 2 is “workload low”, 3 is “enough spare capacity for all 

desirable additional tasks”, and 10 is “task abandoned, pilot unable to apply sufficient effort”.

Table 46. Workload ratings: Inflight operations during arrivals

Role Conditions Median Mean Std. Deviation N

PF

Voice/Paper 2.00 2.2273 1.63100 22

DataComm/Paper 2.00 3.0909 1.79706 22

DataComm/MMD 2.00 2.4545 1.79224 22

DataComm/Route 2.00 2.3182 1.04135 22

PM

Voice/Paper 1.00 1.5909 .85407 22

DataComm/Paper 2.00 2.2273 1.41192 22

DataComm/MMD 2.00 1.6364 .65795 22

DataComm/Route 2.00 2.0909 .97145 22

1) Your workload in flight during arrivals 2) Your workload during surface / taxi operations
Appendix O: Post-Scenario Results 199

Table 47. Workload ratings: Surface operations during arrivals and departures

Role Conditions Median Mean Std. Deviation N

PF

Voice/Paper 2.00 2.3409 1.23784 44

DataComm/Paper 3.00 3.5227 1.75855 44

DataComm/MMD 3.00 2.9318 1.64808 44

DataComm/Route 2.00 2.4318 1.40427 44

PM

Voice/Paper 2.00 1.9773 1.10997 44

DataComm/Paper 2.50 2.9318 1.60519 44

DataComm/MMD 2.00 2.3409 .98697 44

DataComm/Route 2.50 2.7727 1.64053 44

Table 48. Workload ratings: PF and PM mean Ranks

Seat

Inflight 

Workload

N

Inflight Workload

Mean Rank

Surface 

Workload

N

Surface Workload

Mean Rank

PF 88 99.34 176 184.45

PM 88 77.66 176 168.55

Total 176 352

Table 49. Workload ratings: Binomial test of scale use

Role Category N

Observed 

Prop. Test Prop.

Asymp. Sig. 

(1-tailed)

PF Inflight Workload. Group 1 ≤ 3 74 .84 .75 .028(a)

Group 2 > 3 14 .16

Total 88 1.00

Surface Workload Group 1 ≤ 3 138 .78 .75 .009(a)

Group 2 > 3 38 .22

Total 176 1.00

PM Inflight Workload. Group 1 ≤ 3 83 .94 .75 .000(a)

Group 2 > 3 5 .06

Total 88 1.00

Surface Workload Group 1 ≤ 3 150 .85 .75 .000(a)

Group 2 > 3 26 .15

Total 176 1.00

a Based on Z Approximation.
Appendix O: Post-Scenario Results 200

Table 50. Workload ratings: Kruskal Wallis difference test for PF and PM 

Inflight 

Workload.

Surface 

Workload

Chi-Square 9.094 2.339

df 1 1

Asymp. Sig. .003 .126

Table 51. Workload ratings: PF and PM Friedman Ranks difference by condition

Seat Inflight_Wkld Mean Rank Surface_Wkld Mean Rank

PF

Inflight_Wkld_C1 1.93 Surface_Wkld_C1 1.98

Inflight_Wkld_C2 3.39 Surface_Wkld_C2 3.34

Inflight_Wkld_C3 2.34 Surface_Wkld_C3 2.58

Inflight_Wkld_C4 2.34 Surface_Wkld_C4 2.10

PM

Inflight_Wkld_C1 1.95 Surface_Wkld_C1 1.86

Inflight_Wkld_C2 3.00 Surface_Wkld_C2 3.01

Inflight_Wkld_C3 2.09 Surface_Wkld_C3 2.38

Inflight_Wkld_C4 2.95 Surface_Wkld_C4 2.75

Table 52. Workload ratings: PF and PM test statistics

Inflight_Wkld Surface_Wkld

PF

N 22 44

Chi-Square 28.525 43.603

df 3 3

Asymp. Sig. .000 .000

PM

N 22 44

Chi-Square 25.245 34.875

df 3 3

Asymp. Sig. .000 .000

Table 53. Legend for superscript in workload pairwise comparisons tables

a. Inflight_Wkld_C2 < Inflight_Wkld_C1 j. Inflight_Wkld_C3 < Inflight_Wkld_C2

b. Inflight_Wkld_C2 > Inflight_Wkld_C1 k. Inflight_Wkld_C3 > Inflight_Wkld_C2

c. Inflight_Wkld_C2 = Inflight_Wkld_C1 l. Inflight_Wkld_C3 = Inflight_Wkld_C2

d. Inflight_Wkld_C3 < Inflight_Wkld_C1 m. Inflight_Wkld_C4 < Inflight_Wkld_C2

e. Inflight_Wkld_C3 > Inflight_Wkld_C1 n. Inflight_Wkld_C4 > Inflight_Wkld_C2

f. Inflight_Wkld_C3 = Inflight_Wkld_C1 o. Inflight_Wkld_C4 = Inflight_Wkld_C2

g. Inflight_Wkld_C4 < Inflight_Wkld_C1 p. Inflight_Wkld_C4 < Inflight_Wkld_C3

h. Inflight_Wkld_C4 > Inflight_Wkld_C1 q. Inflight_Wkld_C4 > Inflight_Wkld_C3

i. Inflight_Wkld_C4 = Inflight_Wkld_C1 r. Inflight_Wkld_C4 = Inflight_Wkld_C3
Appendix O: Post-Scenario Results 201

Table 54. Workload ratings: Pairwise comparisons Ranks of inflight operations during arrivals

Seat N Mean Rank Sum of Ranks

PF

Inflight_Wkld_C2 -

Inflight_Wkld_C1

Negative Ranks 0

a

.00 .00

Positive Ranks 16b8.50 136.00

Ties 6

c

Total 22

Inflight_Wkld_C3 -

Inflight_Wkld_C1

Negative Ranks 1

d

4.00 4.00

Positive Ranks 6

e

4.00 24.00

Ties 15f

Total 22

Inflight_Wkld_C4 -

Inflight_Wkld_C1

Negative Ranks 3

g

7.83 23.50

Positive Ranks 7

h

4.50 31.50

Ties 12i

Total 22

Inflight_Wkld_C3 -

Inflight_Wkld_C2

Negative Ranks 12j6.50 78.00

Positive Ranks 0

k

.00 .00

Ties 10l

Total 22

Inflight_Wkld_C4 -

Inflight_Wkld_C2

Negative Ranks 11m6.00 66.00

Positive Ranks 0

n

.00 .00

Ties 11o

Total 22

Inflight_Wkld_C4 -

Inflight_Wkld_C3

Negative Ranks 3

p

4.50 13.50

Positive Ranks 3

q

2.50 7.50

Ties 16r

Total 22

PM

Inflight_Wkld_C2 -

Inflight_Wkld_C1

Negative Ranks 0

a

.00 .00

Positive Ranks 12b6.50 78.00

Ties 10c

Total 22

Inflight_Wkld_C3 -

Inflight_Wkld_C1

Negative Ranks 2

d

3.00 6.00

Positive Ranks 3

e

3.00 9.00

Ties 17f

Total 22

Inflight_Wkld_C4 -

Inflight_Wkld_C1

Negative Ranks 0

g

.00 .00

Positive Ranks 11h6.00 66.00

Ties 11i

Total 22

Inflight_Wkld_C3 -

Inflight_Wkld_C2

Negative Ranks 9

j

5.00 45.00

Positive Ranks 0

k

.00 .00

Ties 13l

Total 22

Inflight_Wkld_C4 -

Inflight_Wkld_C2

Negative Ranks 4

m

4.38 17.50

Positive Ranks 3

n

3.50 10.50

Ties 15o

Total 22

Inflight_Wkld_C4 -

Inflight_Wkld_C3

Negative Ranks 0

p

.00 .00

Positive Ranks 10q5.50 55.00

Ties 12r

Total 22
Appendix O: Post-Scenario Results 202

Table 55. Workload ratings: Pairwise comparisons Ranks for surface operations

Seat N Mean Rank Sum of Ranks

PF

Surface_Wkld_C2 -

Surface_Wkld_C1

Negative Ranks 5

a

15.70 78.50

Positive Ranks 32b19.52 624.50

Ties 7

c

Total 44

Surface_Wkld_C3 -

Surface_Wkld_C1

Negative Ranks 3

d

17.17 51.50

Positive Ranks 19e10.61 201.50

Ties 22f

Total 44

Surface_Wkld_C4 -

Surface_Wkld_C1

Negative Ranks 8

g

9.94 79.50

Positive Ranks 11h10.05 110.50

Ties 25i

Total 44

Surface_Wkld_C3 -

Surface_Wkld_C2

Negative Ranks 26j16.60 431.50

Positive Ranks 7

k

18.50 129.50

Ties 11l

Total 44

Surface_Wkld_C4 -

Surface_Wkld_C2

Negative Ranks 29m15.17 440.00

Positive Ranks 1

n

25.00 25.00

Ties 14o

Total 44

Surface_Wkld_C4 -

Surface_Wkld_C3

Negative Ranks 16p12.06 193.00

Positive Ranks 6

q

10.00 60.00

Ties 22r

Total 44

PM

Surface_Wkld_C2 -

Surface_Wkld_C1

Negative Ranks 1

a

8.00 8.00

Positive Ranks 25b13.72 343.00

Ties 18c

Total 44

Surface_Wkld_C3 -

Surface_Wkld_C1

Negative Ranks 5

d

11.30 56.50

Positive Ranks 16e10.91 174.50

Ties 23f

Total 44

Surface_Wkld_C4 -

Surface_Wkld_C1

Negative Ranks 0

g

.00 .00

Positive Ranks 21h11.00 231.00

Ties 23i

Total 44

Surface_Wkld_C3 -

Surface_Wkld_C2

Negative Ranks 20j13.35 267.00

Positive Ranks 6

k

14.00 84.00

Ties 18l

Total 44

Surface_Wkld_C4 -

Surface_Wkld_C2

Negative Ranks 14m11.07 155.00

Positive Ranks 7

n

10.86 76.00

Ties 23o

Total 44

Surface_Wkld_C4 -

Surface_Wkld_C3

Negative Ranks 5

p

11.00 55.00

Positive Ranks 13q8.92 116.00

Ties 26r

Total 44
Appendix O: Post-Scenario Results 203

Table 56. Workload ratings: Pairwise comparisons test statistics (a) for inflight operations

Seat

Inflight_Wkld_

C2 -

Inflight_Wkld_

C1 (b,b)

Inflight_Wkld_

C3 -

Inflight_Wkld_

C1 (b,b)

Inflight_Wkld_

C4 -

Inflight_Wkld_

C1 (b,b)

Inflight_Wkld_

C3 -

Inflight_Wkld_

C2 (c,c)

Inflight_Wkld_

C4 -

Inflight_Wkld_

C2 (c,c)

Inflight_Wkld_

C4 -

Inflight_Wkld_

C3 (c,b)

PF **-3.755b-1.890b-.432b**-3.276c**-3.022c-.647c

PM **-3.357b-.447b**-3.317b**-2.807c-.632c**-3.162b

a. Wilcoxon Signed Ranks Test (Z), ** p<0.008 (alpha=0.05 Bonferroni adjusted)

b. Based on negative ranks, c. Based on positive ranks.

Table 57. Workload ratings: Pairwise comparisons test statistics (a) for surface operations

Seat

Surface_Wkld

_C2 -

Surface_Wkld

_C1 (b,b)

Surface_Wkld_

C3 -

Surface_Wkld_

C1 (b,b)

Surface_Wkld_

C4 -

Surface_Wkld_

C1 (b,b)

Surface_Wkld_

C3 -

Surface_Wkld_

C2 (c,c)

Surface_Wkld

_C4 -

Surface_Wkld

_C2 (c,c)

Surface_Wkld_

C4 -

Surface_Wkld_

C3 (c,b)

PF **-4.245b-2.499b-.655b**-2.789c**-4.371c-2.342c

PM **-4.365b-2.128b**-4.200b**-2.374c-1.413c**-1.345b

 a. Wilcoxon Signed Ranks Test (Z), ** p<0.008 (alpha=0.05 Bonferroni adjusted)

b. Based on negative ranks, c. Based on positive ranks.

Table 58. Workload ratings: By message altitude band during arrivals

Table 59. Workload ratings: Difference by condition within each altitude band

MsgAltitude Surface 

Workload

Inflight 

Workload

Low

Chi-Square 1.602 1.157

df 2 2

Asymp. Sig. .449 .561

Medium

Chi-Square 6.405 4.569

df 2 2

Asymp. Sig. .041 .102

High

Chi-Square .636 .614

df 2 2

Asymp. Sig. .728 .736

Descriptive Statistics

43 2.1628 1.11120 1.00 7.00

44 2.9318 1.43701 1.00 6.00

44 3.0000 .86266 2.00 4.00

43 2.3256 1.10671 1.00 6.00

44 3.2727 1.77008 1.00 8.00

44 3.0909 .80169 2.00 4.00

44 2.1591 1.39673 1.00 7.00

44 2.8864 1.40126 1.00 7.00

44 2.9091 .80169 2.00 4.00

Inf light Workload

Surf ace Workload

Condition

Inf light Workload

Surf ace Workload

Condition

Inf light Workload

Surf ace Workload

Condition

MsgAltitude

Low

Medium

High

N Mean Std. Dev iation Minimum Maximum
Appendix O: Post-Scenario Results 204

O.2 Situation Awareness

The left column are SA ratings in flight, the right column for surface. The top row is DEMAND ON 

ATTENTIONAL RESOURCES, the middle is SUPPLY OF ATTENTIONAL RESOURCES, and the 

bottom row is UNDERSTANDING OF THE SITUATION. Ratings were 1 = High and 7 = Low.

2A) Demand on Attentional Resource, In Flight 2B) Demand on Attentional Resource, Surface 

Ops

2C) Supply of Attentional Resource, In Flight

2E) Understanding the Situation, In Flight

2D) Supply of Attentional Resource, Surface 

Ops

2F) Understanding the Situation, Surface Ops
Appendix O: Post-Scenario Results 205

Table 60. SART ratings: Inflight operations during arrivals

Role Conditions Mean Std. Deviation N

PF Voice/Paper 8.6818 2.80113 22

DataComm/Paper 7.5455 2.36497 22

DataComm/MMD 8.3182 2.51446 22

DataComm/Route 8.4091 2.06234 22

Total 8.2386 2.44477 88

PM Voice/Paper 10.6364 2.59203 22

DataComm/Paper 9.4545 2.44418 22

DataComm/MMD 9.6818 2.31735 22

DataComm/Route 9.2727 2.47236 22

Total 9.7614 2.47281 88

Total Voice/Paper 9.6591 2.84436 44

DataComm/Paper 8.5000 2.56542 44

DataComm/MMD 9.0000 2.48718 44

DataComm/Route 8.8409 2.29198 44

Total 9.0000 2.56793 176

Table 61. SART ratings: Surface operations during arrivals and departures

Role Conditions Mean Std. Deviation N

PF Voice/Paper 8.2273 2.45768 44

DataComm/Paper 6.3864 2.72128 44

DataComm/MMD 7.4318 2.46272 44

DataComm/Route 7.9545 2.73610 44

Total 7.5000 2.67047 176

PM Voice/Paper 9.3636 2.91021 44

DataComm/Paper 7.5909 3.01406 44

DataComm/MMD 7.6136 2.69553 44

DataComm/Route 7.6591 2.65841 44

Total 8.0568 2.89969 176

Total Voice/Paper 8.7955 2.73823 88

DataComm/Paper 6.9886 2.91841 88

DataComm/MMD 7.5227 2.56850 88

DataComm/Route 7.8068 2.68610 88

Total 7.7784 2.79739 352
Appendix O: Post-Scenario Results 206

Table 62. SART ratings: Surface operations during departures only

Role Conditions Mean Std. Deviation N

PF Voice/Paper 8.5455 2.57695 22

DataComm/Paper 7.2727 2.71121 22

DataComm/MMD 7.6364 2.23704 22

DataComm/Route 8.5909 2.95456 22

Total 8.0114 2.65007 88

PM Voice/Paper 8.9091 3.06919 22

DataComm/Paper 8.2727 2.91436 22

DataComm/MMD 7.8182 2.77122 22

DataComm/Route 8.3182 2.95016 22

Total 8.3295 2.90351 88

Total Voice/Paper 8.7273 2.80667 44

DataComm/Paper 7.7727 2.82731 44

DataComm/MMD 7.7273 2.49057 44

DataComm/Route 8.4545 2.92109 44

Total 8.1705 2.77631 176

Table 63. SART ratings: Surface operations during arrivals only

Role Conditions Mean Std. Deviation N

PF Voice/Paper 7.9091 2.34844 22

DataComm/Paper 5.5000 2.48328 22

DataComm/MMD 7.2273 2.70681 22

DataComm/Route 7.3182 2.39814 22

Total 6.9886 2.60633 88

PM Voice/Paper 9.8182 2.73664 22

DataComm/Paper 6.9091 3.02228 22

DataComm/MMD 7.4091 2.66653 22

DataComm/Route 7.0000 2.20389 22

Total 7.7841 2.88655 88

Total Voice/Paper 8.8636 2.69876 44

DataComm/Paper 6.2045 2.82497 44

DataComm/MMD 7.3182 2.65691 44

DataComm/Route 7.1591 2.28181 44

Total 7.3864 2.77100 176
Appendix O: Post-Scenario Results 207

Table 64. SART ratings: PF and PM difference test

SART_DF SART_SO

SART_Surface

Departure

SART_Surface

Arrival

Chi-Square 16.341 4.533 .872 4.450

df 1 1 1 1

Asymp. Sig. .000 .033 .351 .035

Table 65. SART ratings: PF and PM difference by condition

Role newSART_DF newSART_SO

newSART_Surf

aceDeparture

newSART_Surf

aceArrival

PF Chi-Square 2.723 10.649 2.982 10.342

df 3 3 3 3

Asymp. Sig. .436 .014 .394 .016

PM Chi-Square 5.205 12.332 1.875 15.459

df 3 3 3 3

Asymp. Sig. .157 .006 .599 .001

Table 66. SART ratings: Pairwise comparisons for inflight operations

Role (I) Conditions (J) Conditions

Mean 

Difference 

(I-J) Std. Error

95% Confidence Interval

Upper Bound Lower Bound

PF Voice/Paper DataComm/Paper 1.1364 .78159 -1.0422 3.3149

DataComm/MMD .3636 .80252 -1.8732 2.6005

DataComm/Route .2727 .74161 -1.7944 2.3398

DataComm/Paper Voice/Paper -1.1364 .78159 -3.3149 1.0422

DataComm/MMD -.7727 .73595 -2.8241 1.2786

DataComm/Route -.8636 .66900 -2.7284 1.0011

DataComm/MMD Voice/Paper -.3636 .80252 -2.6005 1.8732

DataComm/Paper .7727 .73595 -1.2786 2.8241

DataComm/Route -.0909 .69334 -2.0235 1.8416

DataComm/Route Voice/Paper -.2727 .74161 -2.3398 1.7944

DataComm/Paper .8636 .66900 -1.0011 2.7284

DataComm/MMD .0909 .69334 -1.8416 2.0235

PM Voice/Paper DataComm/Paper 1.1818 .75956 -.9353 3.2990

DataComm/MMD .9545 .74127 -1.1116 3.0207

DataComm/Route 1.3636 .76370 -.7650 3.4923

DataComm/Paper Voice/Paper -1.1818 .75956 -3.2990 .9353

DataComm/MMD -.2273 .71808 -2.2288 1.7743

DataComm/Route .1818 .74121 -1.8842 2.2478

DataComm/MMD Voice/Paper -.9545 .74127 -3.0207 1.1116

DataComm/Paper .2273 .71808 -1.7743 2.2288

DataComm/Route .4091 .72245 -1.6046 2.4228

DataComm/Route Voice/Paper -1.3636 .76370 -3.4923 .7650

DataComm/Paper -.1818 .74121 -2.2478 1.8842

DataComm/MMD -.4091 .72245 -2.4228 1.6046

Dunnett C

Based on observed means.
Appendix O: Post-Scenario Results 208

Table 67. SART ratings: Pairwise comparisons for all surface operations

Role (I) Conditions (J) Conditions

Mean 

Difference 

(I-J) Std. Error

95% Confidence Interval

Upper Bound Lower Bound

PF Voice/Paper DataComm/Paper 1.8409(*) .55279 .3636 3.3182

DataComm/MMD .7955 .52452 -.6063 2.1972

DataComm/Route .2727 .55445 -1.2090 1.7545

DataComm/Paper Voice/Paper -1.8409(*) .55279 -3.3182 -.3636

DataComm/MMD -1.0455 .55330 -2.5241 .4332

DataComm/Route -1.5682(*) .58176 -3.1229 -.0135

DataComm/MMD Voice/Paper -.7955 .52452 -2.1972 .6063

DataComm/Paper 1.0455 .55330 -.4332 2.5241

DataComm/Route -.5227 .55496 -2.0058 .9604

DataComm/Route Voice/Paper -.2727 .55445 -1.7545 1.2090

DataComm/Paper 1.5682(*) .58176 .0135 3.1229

DataComm/MMD .5227 .55496 -.9604 2.0058

PM Voice/Paper DataComm/Paper 1.7727(*) .63163 .0848 3.4607

DataComm/MMD 1.7500(*) .59801 .1519 3.3481

DataComm/Route 1.7045(*) .59422 .1165 3.2926

DataComm/Paper Voice/Paper -1.7727(*) .63163 -3.4607 -.0848

DataComm/MMD -.0227 .60959 -1.6518 1.6064

DataComm/Route -.0682 .60587 -1.6873 1.5510

DataComm/MMD Voice/Paper -1.7500(*) .59801 -3.3481 -.1519

DataComm/Paper .0227 .60959 -1.6064 1.6518

DataComm/Route -.0455 .57074 -1.5707 1.4798

DataComm/Route Voice/Paper -1.7045(*) .59422 -3.2926 -.1165

DataComm/Paper .0682 .60587 -1.5510 1.6873

DataComm/MMD .0455 .57074 -1.4798 1.5707

Dunnett C

Based on observed means.

* The mean difference is significant at the .05 level.
Appendix O: Post-Scenario Results 209

Table 68. SART ratings: Pairwise comparisons for surface departure operations

Role (I) Conditions (J) Conditions

Mean 

Difference 

(I-J) Std. Error

95% Confidence Interval

Upper Bound Lower Bound

PF Voice/Paper DataComm/Paper 1.2727 .79748 -.9501 3.4956

DataComm/MMD .9091 .72754 -1.1188 2.9370

DataComm/Route -.0455 .83585 -2.3752 2.2843

DataComm/Paper Voice/Paper -1.2727 .79748 -3.4956 .9501

DataComm/MMD -.3636 .74939 -2.4524 1.7252

DataComm/Route -1.3182 .85493 -3.7012 1.0648

DataComm/MMD Voice/Paper -.9091 .72754 -2.9370 1.1188

DataComm/Paper .3636 .74939 -1.7252 2.4524

DataComm/Route -.9545 .79010 -3.1568 1.2477

DataComm/Route Voice/Paper .0455 .83585 -2.2843 2.3752

DataComm/Paper 1.3182 .85493 -1.0648 3.7012

DataComm/MMD .9545 .79010 -1.2477 3.1568

PM Voice/Paper DataComm/Paper .6364 .90236 -1.8788 3.1515

DataComm/MMD 1.0909 .88162 -1.3665 3.5483

DataComm/Route .5909 .90763 -1.9390 3.1208

DataComm/Paper Voice/Paper -.6364 .90236 -3.1515 1.8788

DataComm/MMD .4545 .85741 -1.9353 2.8444

DataComm/Route -.0455 .88413 -2.5098 2.4189

DataComm/MMD Voice/Paper -1.0909 .88162 -3.5483 1.3665

DataComm/Paper -.4545 .85741 -2.8444 1.9353

DataComm/Route -.5000 .86295 -2.9053 1.9053

DataComm/Route Voice/Paper -.5909 .90763 -3.1208 1.9390

DataComm/Paper .0455 .88413 -2.4189 2.5098

DataComm/MMD .5000 .86295 -1.9053 2.9053

Dunnett C

Based on observed means.
Appendix O: Post-Scenario Results 210

Table 69. SART ratings: Pairwise comparisons for surface arrival operations

Role (I) Conditions (J) Conditions

Mean 

Difference 

(I-J) Std. Error

95% Confidence Interval

Upper Bound Lower Bound

PF Voice/Paper DataComm/Paper 2.4091(*) .72869 .3780 4.4402

DataComm/MMD .6818 .76402 -1.4478 2.8114

DataComm/Route .5909 .71561 -1.4037 2.5856

DataComm/Paper Voice/Paper -2.4091(*) .72869 -4.4402 -.3780

DataComm/MMD -1.7273 .78316 -3.9102 .4557

DataComm/Route -1.8182 .73601 -3.8697 .2333

DataComm/MMD Voice/Paper -.6818 .76402 -2.8114 1.4478

DataComm/Paper 1.7273 .78316 -.4557 3.9102

DataComm/Route -.0909 .77101 -2.2400 2.0581

DataComm/Route Voice/Paper -.5909 .71561 -2.5856 1.4037

DataComm/Paper 1.8182 .73601 -.2333 3.8697

DataComm/MMD .0909 .77101 -2.0581 2.2400

PM Voice/Paper DataComm/Paper 2.9091(*) .86926 .4862 5.3320

DataComm/MMD 2.4091(*) .81463 .1385 4.6797

DataComm/Route 2.8182(*) .74913 .7301 4.9063

DataComm/Paper Voice/Paper -2.9091(*) .86926 -5.3320 -.4862

DataComm/MMD -.5000 .85930 -2.8951 1.8951

DataComm/Route -.0909 .79748 -2.3137 2.1319

DataComm/MMD Voice/Paper -2.4091(*) .81463 -4.6797 -.1385

DataComm/Paper .5000 .85930 -1.8951 2.8951

DataComm/Route .4091 .73755 -1.6467 2.4649

DataComm/Route Voice/Paper -2.8182(*) .74913 -4.9063 -.7301

DataComm/Paper .0909 .79748 -2.1319 2.3137

DataComm/MMD -.4091 .73755 -2.4649 1.6467

Dunnett C

Based on observed means.

* The mean difference is significant at the .05 level.

Table 70. SART ratings: During arrival scenario by message altitude

MsgAltitude N Mean Std. Deviation Minimum Maximum

Low SART_InFlight 22 8.6364 2.23704 4.00 12.00

SART_Surface Ops 22 6.5455 2.21955 2.00 10.00

Condition 22 3.0000 .87287 2.00 4.00

Medium SART_DF 22 7.5909 2.51962 4.00 11.00

SART_SurfaceArrival 22 5.4545 2.95566 .00 12.00

Condition 22 3.0909 .81118 2.00 4.00

High SART_DF 22 8.8182 2.63016 4.00 13.00

SART_SurfaceArrival 20 6.7000 2.31926 2.00 12.00

Condition 22 2.9091 .81118 2.00 4.00
Appendix O: Post-Scenario Results 211

Table 71. SART ratings: During arrival scenario by message altitude, test on conditions

MsgAltitude

SART 

Surface Ops 

in Arrivals

SART Inflight 

Operations

Low Chi-Square 3.012 1.714

df 2 2

Asymp. Sig. .222 .424

Medium Chi-Square 6.162 1.307

df 2 2

Asymp. Sig. .046 .520

High Chi-Square 1.719 1.230

df 2 2

Asymp. Sig. .423 .541

Table 72. SART ratings: Binomial test for PF and PM by condition

Role Category N

Observed 

Prop. Test Prop.

Asymp. Sig. 

(1-tailed)

PF SART Inflight Operations Group 1 ≤ 4 10 .11 .75 .000(a,b)

Group 2 > 4 78 .89

Total 88 1.00

SART Surface Operations Group 1 ≤ 4 23 .13 .75 .000(a,b)

Group 2 > 4 153 .87

Total 176 1.00

SART Surface Ops in 

Departures

Group 1 ≤ 4 10 .11 .75 .000(a,b)

Group 2 > 4 78 .89

Total

88 1.00

SART Surface Ops in 

Arrivals

Group 1 ≤ 4 13 .15 .75 .000(a,b)

Group 2 > 4 75 .85

Total 88 1.00

PM SART Inflight Operations Group 1 ≤ 4 1 .01 .75 .000(a,b)

Group 2 > 4 87 .99

Total 88 1.00

SART Surface Operations Group 1 ≤ 4 22 .13 .75 .000(a,b)

Group 2 > 4 154 .88

Total 176 1.00

SART Surface Ops in 

Departures

Group 1 ≤ 4 10 .11 .75 .000(a,b)

Group 2 > 4 78 .89

Total 88 1.00

SART Surface Ops in 

Arrivals

Group 1 ≤ 4 12 .14 .75 .000(a,b)

Group 2 > 4 76 .86

Total 88 1.00

a Alternative hypothesis states that the proportion of cases in the first group < .75.

b Based on Z Approximation.
Appendix O: Post-Scenario Results 212

O.3 Acceptability of “Expected Taxi” and “Taxi” clearances 

1) Did the display of the OWNSHIP POSITION on the navigation display make the taxi clearance 

easier to understand and to carry out? (1 – Easier, 7 – Not Easier, 8 – NA )

2) Did the display of the ROUTE on the navigation display make the taxi clearance easier to 

understand and to carry out? (1 – Easier, 7 – Not Easier, 8 – NA)

3) Did you have confidence that the taxi route was accurately depicted based on the Data Comm ATC 

instruction? (1 – confident route was accurate, 7 – not confident route was accurate, 8 – NA)

4) Did you have a sufficient amount of time to respond to the Voice or Data Comm transmitted 

messages? (1 – More than enough time, 7 – did not have enough time, 8 – NA)

1) Ownship Position Make Clearance Easier To Understand 2) Route Displayed Make Clearance Easier To Understand

3) Confidence That Route Was Accurate 4) Sufficient Time to Respond to Messages
Appendix O: Post-Scenario Results 213

5) Was the amount of heads-down time required to receive and respond to just the “Expected Taxi” 

Data Comm messages acceptable in this scenario? (1 – minimal increase in Head Down time, 7 – too 

much Head Down time, 8 – NA)

6) Was the amount of heads-down time required to receive and respond to other non-time critical Data 

Comm messages acceptable in this scenario? (1 – minimal increase in Head Down time, 7 – too much 

Head Down time, 8 – NA)

7) Overall, was the communication mode (Voice or Data Comm) for receiving Expected and Taxi 

clearances acceptable during this scenario? (1 – Completely acceptable, 7 – completely unacceptable) 

[Note: this question was presented to the subjects only during Data Comm scenarios]

8) How much operational risk was introduced by the communication mode (Voice or Data Comm) 

used during this scenario? (1 – Extremely low risk, 7 – extremely high risk) 

5) Head Down Time to Respond to Taxi Messages 6) Head Down Time to Respond to Info Messages

7) Communication Mode Acceptable 8) Operational Risk
Appendix O: Post-Scenario Results 214

9) Was there a point at which you did not feel that the transmitted taxi instructions were accurate? 

(1 – The message was accurate, 7 – the message was not accurate)

Table 73. Acceptability ratings: Ownship helpful to understand clearance

Role Conditions Mean Std. Deviation N

PF DataComm/MMD 1.9070 1.28756 43

DataComm/Route 1.4545 .72991 44

Total 1.6782 1.06197 87

PM DataComm/MMD 1.6364 1.01365 44

DataComm/Route 1.6136 1.22410 44

Total 1.6250 1.11739 88

Table 74. Acceptability ratings: Route helpful to understand clearance

Role Conditions Mean Std. Deviation N

PF DataComm/Route 1.3256 .74709 43

Total 1.3256 .74709 43

PM DataComm/Route 1.3636 1.01365 44

Total 1.3636 1.01365 44

Table 75. Acceptability ratings: Confidence in route depiction

Role Conditions Mean Std. Deviation N

PF DataComm/Route 1.6047 1.07215 43

Total 1.6047 1.07215 43

PM DataComm/Route 1.4651 .66722 43

Total 1.4651 .66722 43

9) Accuracy of Taxi Instructions
Appendix O: Post-Scenario Results 215

Table 76. Acceptability ratings: Sufficient time to respond to Voice or Data Comm message

Role Conditions Mean Std. Deviation N

PF Voice/Paper 1.6429 .85029 42

DataComm/Paper 3.0227 1.48619 44

DataComm/MMD 2.5227 1.06724 44

DataComm/Route 2.7955 1.59329 44

Total 2.5057 1.38007 174

PM Voice/Paper 1.3810 .69677 42

DataComm/Paper 2.3636 1.46416 44

DataComm/MMD 2.0682 1.16933 44

DataComm/Route 2.3864 1.49753 44

Total 2.0575 1.30677 174

Table 77. Acceptability ratings: Head down time acceptable for “Expected Taxi” messages

Role Conditions Mean Std. Deviation N

PF DataComm/Paper 3.3409 1.37998 44

DataComm/MMD 2.8409 1.27486 44

DataComm/Route 3.0000 1.52499 44

Total 3.0606 1.40206 132

PM DataComm/Paper 2.6136 1.46614 44

DataComm/MMD 2.2955 1.35680 44

DataComm/Route 2.5000 1.35544 44

Total 2.4697 1.38938 132

Table 78. Acceptability ratings: Head down time for non-time-critical messages

Role Conditions Mean Std. Deviation N

PF DataComm/Paper 3.3023 1.55126 43

DataComm/MMD 2.8605 1.30167 43

DataComm/Route 3.0000 1.38093 44

Total 3.0538 1.41592 130

PM DataComm/Paper 2.3409 1.39673 44

DataComm/MMD 2.2500 1.33164 44

DataComm/Route 2.0698 1.16282 43

Total 2.2214 1.29668 131
Appendix O: Post-Scenario Results 216

Table 79. Acceptability ratings: Overall acceptability of Data Comm

Role Conditions Mean Std. Deviation N

PF DataComm/Paper 3.0682 1.26487 44

DataComm/MMD 2.4651 1.03162 43

DataComm/Route 2.4318 1.40427 44

Total 2.6565 1.26959 131

PM DataComm/Paper 2.1860 1.20031 43

DataComm/MMD 1.8182 .92190 44

DataComm/Route 1.9773 1.15111 44

Total 1.9924 1.09892 131

Table 80. Acceptability ratings: Operational risk imposed by communication mode

Role Conditions Mean Std. Deviation N

PF Voice/Paper 2.2558 1.19708 43

DataComm/Paper 3.1591 1.34585 44

DataComm/MMD 2.9070 1.30592 43

DataComm/Route 2.8182 1.41869 44

Total 2.7874 1.34966 174

PM Voice/Paper 1.7045 .90424 44

DataComm/Paper 2.2326 1.17184 43

DataComm/MMD 2.2500 1.25984 44

DataComm/Route 2.4318 1.35368 44

Total 2.1543 1.20543 175

Table 81. Acceptability ratings: Taxi instructions considered accurate

Role Conditions Mean Std. Deviation N

PF Voice/Paper 1.6667 1.18253 42

DataComm/Paper 1.8864 1.29787 44

DataComm/MMD 1.7674 .99612 43

DataComm/Route 1.5455 1.13002 44

Total 1.7168 1.15417 173

PM Voice/Paper 1.4091 .92304 44

DataComm/Paper 1.4186 1.00552 43

DataComm/MMD 1.5455 1.08809 44

DataComm/Route 1.4091 1.04143 44

Total 1.4457 1.00925 175
Appendix O: Post-Scenario Results 217

Table 82. Acceptability ratings: PF and PM differences

Chi-Square df Asymp. Sig.

Ownship Helpful To Understand Clearance .759 1 .383

Route Helpful To Understand Clearance .058 1 .809

Confidence in Route Depiction .122 1 .727

Sufficient Time to Respond to Data Comm message 12.639 1 .000

Head Down Time Acceptable for Expected Taxi 12.159 1 .000

Head Down Time Acceptable for Non-Critical messages 24.162 1 .000

Overall Acceptability of Data Comm 20.665 1 .000

Operational Risk Imposed 20.966 1 .000

Taxi Instructions Considered Accurate 12.102 1 .001

Table 83. Acceptability ratings: PF and PM differences by condition

Role

PF PM

ChiSquare dfAsymp. 

Sig.

ChiSquare dfAsymp. 

Sig.

Ownship Helpful To Understand Clearance 3.656 1 .056 .787 1 .375

Route Helpful To Understand Clearance 27.653 3 .000 18.974 3 .000

Confidence in Route Depiction 3.138 2 .208 1.188 2 .552

Sufficient Time to Respond to Data Comm message 1.822 2 .402 .556 2 .757

Head Down Time Acceptable for Expected Taxi 7.958 2 .019 1.891 2 .389

Head Down Time Acceptable for Non-Critical msgs 10.673 3 .014 9.946 3 .019

Overall Acceptability of Data Comm 4.616 3 .202 .874 3 .832
Appendix O: Post-Scenario Results 218

Table 84. Acceptability ratings: Pairwise comparisons for sufficient time to respond by condition

Role (I) Conditions (J) Conditions

Mean 

Difference 

(I-J) Std. Error

95% Confidence Interval

Upper Bound Lower Bound

PF Voice/Paper DataComm/Paper -1.3799(*) .25964 -2.0741 -.6857

DataComm/MMD -.8799(*) .20761 -1.4351 -.3246

DataComm/Route -1.1526(*) .27370 -1.8844 -.4208

DataComm/Paper Voice/Paper 1.3799(*) .25964 .6857 2.0741

DataComm/MMD .5000 .27584 -.2372 1.2372

DataComm/Route .2273 .32847 -.6505 1.1051

DataComm/MMD Voice/Paper .8799(*) .20761 .3246 1.4351

DataComm/Paper -.5000 .27584 -1.2372 .2372

DataComm/Route -.2727 .28910 -1.0453 .4999

DataComm/Route Voice/Paper 1.1526(*) .27370 .4208 1.8844

DataComm/Paper -.2273 .32847 -1.1051 .6505

DataComm/MMD .2727 .28910 -.4999 1.0453

PM Voice/Paper DataComm/Paper -.9827(*) .24552 -1.6391 -.3263

DataComm/MMD -.6872(*) .20648 -1.2393 -.1351

DataComm/Route -1.0054(*) .25005 -1.6739 -.3369

DataComm/Paper Voice/Paper .9827(*) .24552 .3263 1.6391

DataComm/MMD .2955 .28248 -.4595 1.0504

DataComm/Route -.0227 .31574 -.8665 .8211

DataComm/MMD Voice/Paper .6872(*) .20648 .1351 1.2393

DataComm/Paper -.2955 .28248 -1.0504 .4595

DataComm/Route -.3182 .28643 -1.0837 .4473

DataComm/Route Voice/Paper 1.0054(*) .25005 .3369 1.6739

DataComm/Paper .0227 .31574 -.8211 .8665

DataComm/MMD .3182 .28643 -.4473 1.0837

Dunnett C

Based on observed means.

* The mean difference is significant at the .05 level.

Table 85. Acceptability ratings: Pairwise comparisons for “Expected Taxi” message head down

time 

Role (I) Conditions (J) Conditions

Mean 

Difference 

(I-J) Std. Error

95% Confidence Interval

Upper Bound Lower Bound

PF DataComm/Paper DataComm/MMD .5000 .28323 -.1875 1.1875

DataComm/Route .3409 .31006 -.4117 1.0936

DataComm/MMD DataComm/Paper -.5000 .28323 -1.1875 .1875

DataComm/Route -.1591 .29965 -.8865 .5683

DataComm/Route DataComm/Paper -.3409 .31006 -1.0936 .4117

DataComm/MMD .1591 .29965 -.5683 .8865

PM DataComm/Paper DataComm/MMD .3182 .30115 -.4128 1.0492

DataComm/Route .1136 .30101 -.6171 .8443

DataComm/MMD DataComm/Paper -.3182 .30115 -1.0492 .4128

DataComm/Route -.2045 .28913 -.9064 .4973

DataComm/Route DataComm/Paper -.1136 .30101 -.8443 .6171

DataComm/MMD .2045 .28913 -.4973 .9064

Dunnett C 

Based on observed means.
Appendix O: Post-Scenario Results 219

Table 86. Acceptability ratings: Pairwise comparisons for non-time-critical message head down

time 

Role (I) Conditions (J) Conditions

Mean 

Difference 

(I-J) Std. Error

95% Confidence Interval

Upper Bound Lower Bound

PF DataComm/Paper DataComm/MMD .4419 .30882 -.3084 1.1921

DataComm/Route .3023 .31512 -.4630 1.0676

DataComm/MMD DataComm/Paper -.4419 .30882 -1.1921 .3084

DataComm/Route -.1395 .28765 -.8381 .5590

DataComm/Route DataComm/Paper -.3023 .31512 -1.0676 .4630

DataComm/MMD .1395 .28765 -.5590 .8381

PM DataComm/Paper DataComm/MMD .0909 .29093 -.6153 .7971

DataComm/Route .2711 .27529 -.3973 .9396

DataComm/MMD DataComm/Paper -.0909 .29093 -.7971 .6153

DataComm/Route .1802 .26786 -.4702 .8307

DataComm/Route DataComm/Paper -.2711 .27529 -.9396 .3973

DataComm/MMD -.1802 .26786 -.8307 .4702

Dunnett C 

Based on observed means.

Table 87. Acceptability ratings: Pairwise comparisons for overall acceptability of Data Comm

Role (I) Conditions (J) Conditions

Mean 

Difference 

(I-J) Std. Error

95% Confidence Interval

Upper Bound Lower Bound

PF DataComm/Paper DataComm/MMD .6031(*) .24721 .0028 1.2033

DataComm/Route .6364 .28492 -.0553 1.3280

DataComm/MMD DataComm/Paper -.6031(*) .24721 -1.2033 -.0028

DataComm/Route .0333 .26376 -.6071 .6737

DataComm/Route DataComm/Paper -.6364 .28492 -1.3280 .0553

DataComm/MMD -.0333 .26376 -.6737 .6071

PM DataComm/Paper DataComm/MMD .3679 .22983 -.1903 .9261

DataComm/Route .2088 .25223 -.4038 .8213

DataComm/MMD DataComm/Paper -.3679 .22983 -.9261 .1903

DataComm/Route -.1591 .22233 -.6988 .3806

DataComm/Route DataComm/Paper -.2088 .25223 -.8213 .4038

DataComm/MMD .1591 .22233 -.3806 .6988

Dunnett C

Based on observed means.

* The mean difference is significant at the .05 level.
Appendix O: Post-Scenario Results 220

Table 88. Acceptability ratings: Pairwise comparisons for operational risk by condition

Role (I) Conditions (J) Conditions

Mean 

Difference 

(I-J) Std. Error

95% Confidence Interval

Upper Bound Lower Bound

PF Voice/Paper DataComm/Paper -.9033(*) .27293 -1.6330 -.1736

DataComm/MMD -.6512 .27016 -1.3738 .0715

DataComm/Route -.5624 .28119 -1.3141 .1894

DataComm/Paper Voice/Paper .9033(*) .27293 .1736 1.6330

DataComm/MMD .2521 .28430 -.5080 1.0122

DataComm/Route .3409 .29480 -.4469 1.1287

DataComm/MMD Voice/Paper .6512 .27016 -.0715 1.3738

DataComm/Paper -.2521 .28430 -1.0122 .5080

DataComm/Route .0888 .29224 -.6925 .8701

DataComm/Route Voice/Paper .5624 .28119 -.1894 1.3141

DataComm/Paper -.3409 .29480 -1.1287 .4469

DataComm/MMD -.0888 .29224 -.8701 .6925

PM Voice/Paper DataComm/Paper -.5280 .22476 -1.1290 .0730

DataComm/MMD -.5455 .23379 -1.1702 .0793

DataComm/Route -.7273(*) .24542 -1.3831 -.0714

DataComm/Paper Voice/Paper .5280 .22476 -.0730 1.1290

DataComm/MMD -.0174 .26078 -.7147 .6798

DataComm/Route -.1993 .27126 -.9245 .5260

DataComm/MMD Voice/Paper .5455 .23379 -.0793 1.1702

DataComm/Paper .0174 .26078 -.6798 .7147

DataComm/Route -.1818 .27878 -.9268 .5632

DataComm/Route Voice/Paper .7273(*) .24542 .0714 1.3831

DataComm/Paper .1993 .27126 -.5260 .9245

DataComm/MMD .1818 .27878 -.5632 .9268

Dunnett C

Based on observed means.

* The mean difference is significant at the .05 level.
Appendix O: Post-Scenario Results 221

Table 89. Acceptability ratings: Pairwise comparisons for taxi instruction accuracy by condition

Role (I) Conditions (J) Conditions

Mean 

Difference 

(I-J) Std. Error

95% Confidence Interval

Upper Bound Lower Bound

PF Voice/Paper DataComm/Paper -.2197 .26754 -.9353 .4959

DataComm/MMD -.1008 .23742 -.7362 .5347

DataComm/Route .1212 .24963 -.5466 .7890

DataComm/Paper Voice/Paper .2197 .26754 -.4959 .9353

DataComm/MMD .1189 .24771 -.5433 .7811

DataComm/Route .3409 .25943 -.3524 1.0342

DataComm/MMD Voice/Paper .1008 .23742 -.5347 .7362

DataComm/Paper -.1189 .24771 -.7811 .5433

DataComm/Route .2220 .22825 -.3882 .8322

DataComm/Route Voice/Paper -.1212 .24963 -.7890 .5466

DataComm/Paper -.3409 .25943 -1.0342 .3524

DataComm/MMD -.2220 .22825 -.8322 .3882

PM Voice/Paper DataComm/Paper -.0095 .20707 -.5632 .5441

DataComm/MMD -.1364 .21511 -.7112 .4385

DataComm/Route .0000 .20979 -.5607 .5607

DataComm/Paper Voice/Paper .0095 .20707 -.5441 .5632

DataComm/MMD -.1268 .22455 -.7272 .4735

DataComm/Route .0095 .21946 -.5772 .5963

DataComm/MMD Voice/Paper .1364 .21511 -.4385 .7112

DataComm/Paper .1268 .22455 -.4735 .7272

DataComm/Route .1364 .22706 -.4704 .7432

DataComm/Route Voice/Paper .0000 .20979 -.5607 .5607

DataComm/Paper -.0095 .21946 -.5963 .5772

DataComm/MMD -.1364 .22706 -.7432 .4704

Dunnett C 

Based on observed means.
Appendix O: Post-Scenario Results 222

Table 90. Acceptability ratings: By message altitude band

MsgAltitude N Mean Std. Deviation Minimum Maximum

Low Sufficient Time to 

Respond to message 44 3.1136 1.67354 1.00 6.00

Overall Acceptability of 

Data Comm 44 2.4545 1.37172 1.00 5.00

Operational Risk 

Imposed 44 2.9318 1.46902 1.00 5.00

Taxi Instructions 

Accurate 44 1.7273 1.42018 1.00 7.00

Med Sufficient Time to 

Respond to message 44 2.8182 1.60338 1.00 7.00

Overall Acceptability of 

Data Comm 44 2.7045 1.51856 1.00 6.00

Operational Risk 

Imposed 44 3.0227 1.54752 1.00 7.00

Taxi Instructions 

Accurate 44 1.8636 1.24995 1.00 5.00

High Sufficient Time to 

Respond to message 44 2.5682 1.26487 1.00 6.00

Overall Acceptability of 

Data Comm 43 2.4884 1.22226 1.00 5.00

Operational Risk 

Imposed 43 2.9070 1.34189 1.00 5.00

Taxi Instructions 

Accurate 43 1.8605 1.42397 1.00 6.00

Table 91. Acceptability ratings: Differences by altitude band

MsgAltitude

Sufficient 

Time to 

Repond

Overall 

Acceptable

Operational 

Risk Imposed

Taxi 

Instructions 

Accurate

Low Chi-Square 6.507 1.159 .336 2.433

df 2 2 2 2

Asymp. Sig. .039 .560 .845 .296

Med Chi-Square 3.997 6.509 3.117 4.034

df 2 2 2 2

Asymp. Sig. .136 .039 .210 .133

High Chi-Square .937 5.163 .098 3.273

df 2 2 2 2

Asymp. Sig. .626 .076 .952 .195
Appendix O: Post-Scenario Results 223

Table 92. Acceptability ratings: Binomial test

Role Category N

Observed 

Prop.

Test 

Prop.

Asymp. 

Sig. (1-

tailed)

PF Ownship Helpful To Understand Clearance Group 1 ≤ 4 84 .97 .75 .000(a)

Group 2 > 4 3 .03

Total 87 1.00

Route Helpful To Understand Clearance Group 1 ≤ 4 42 .98 .75 .000(a)

Group 2 > 4 1 .02

Total 43 1.00

Confidence in Route Depiction Group 1 ≤ 4 42 .98 .75 .000(a)

Group 2 > 4 1 .02

Total 43 1.00

Sufficient Time to Respond Group 1 ≤ 4 157 .90 .75 .000(a)

Group 2 > 4 17 .10

Total 174 1.00

Head Down Time Acceptable for Taxi msg Group 1 ≤ 4 114 .86 .75 .001(a)

Group 2 > 4 18 .14

Total 132 1.00

Head Down Time Acceptable for Info msg Group 1 ≤ 4 107 .82 .75 .031(a)

Group 2 > 4 23 .18

Total 130 1.00

Overall Acceptability Group 1 ≤ 4 118 .90 .75 .000(a)

Group 2 > 4 13 .10

Total 131 1.00

Operational Risk Imposed Group 1 ≤ 4 152 .87 .75 .000(a)

Group 2 > 4 22 .13

Total 174 1.00

Taxi Instructions Accurate Group 1 ≤ 4 167 .97 .75 .000(a)

Group 2 > 4 6 .03

Total 173 1.00

PM Ownship Helpful To Understand Clearance Group 1 ≤ 4 84 .95 .75 .000(a)

Group 2 > 4 4 .05

Total 88 1.00

Route Helpful To Understand Clearance Group 1 ≤ 4 43 .98 .75 .000(a)

Group 2 > 4 1 .02

Total 44 1.00

Confidence in Route Depiction Group 1 ≤ 4 43 1.00 .75 .000(a)

Group 2 > 4 0 .00

Total 43 1.00

Sufficient Time to Respond Group 1 ≤ 4 163 .94 .75 .000(a)

Group 2 > 4 11 .06

Total 174 1.00

Head Down Time Acceptable for Taxi msg Group 1 ≤ 4 119 .90 .75 .000(a)

Group 2 > 4 13 .10

Total 132 1.00

Head Down Time Acceptable for Info msg Group 1 ≤ 4 120 .92 .75 .000(a)

Group 2 > 4 11 .08

Total 131 1.00

Overall Acceptability Group 1 ≤ 4 125 .95 .75 .000(a)

Group 2 > 4 6 .05

Total 131 1.00

Operational Risk Imposed Group 1 ≤ 4 162 .93 .75 .000(a)

Group 2 > 4 13 .07

Total 175 1.00

Taxi Instructions Accurate Group 1 ≤ 4 168 .96 .75 .000(a)

Group 2 > 4 7 .04

Total 175 1.00

a Based on Z Approximation.
Appendix P: Post-Experiment Results 224

.312 .025 .263 .361

.337 .025 .288 .386

.537 .025 .488 .586

.514 .025 .465 .563

.879 .025 .830 .928

.873 .025 .824 .922

.216 .025 .167 .265

.233 .025 .184 .282

Seat

PF

PM

PF

PM

PF

PM

PF

PM

DisplayCond

DP

MMD

RTE

VP

Mean Std. Error Lower Bound Upper Bound

95% Confidence Interval

Appendix P: Post-Experiment Questionnaire Results

This Appendix presents results from the Post-Experiment Questionnaire (Appendix E). 

P.1 Workload Comparison

Table 93. Workload ratings: Levene's test of equality

Table 94. Workload ratings: Tests of between subjects effects

Table 95. Workload ratings: By Condition and PF and PM

Levene's Test of Equality of Error Variancesa

Dependent Variable: Workload

.938 7 80 .482

F df 1 df 2 Sig.

Tests the null hypothesis that the error variance of the

dependent variable is equal across groups.

Design:

Intercept+DisplayCond+Seat+DisplayCond * Seat

a. 

Tests of Between-Subjects Effects

Dependent Variable: Workload

5.466a 7 .781 116.871 .000 .911

20.920 1 20.920 3130.952 .000 .975

5.459 3 1.820 272.309 .000 .911

.000 1 .000 .030 .862 .000

.008 3 .003 .379 .768 .014

.535 80 .007

26.921 88

6.001 87

Source

Corrected Model

Intercept

DisplayCond

Seat

DisplayCond * Seat

Error

Total

Corrected Total

Type III Sum

of Squares df Mean Square F Sig.

Partial Eta

Squared

a. R Squared = .911 (Adjusted R Squared = .903)
Appendix P: Post-Experiment Results 225

.486 .012 .462 .511

.489 .012 .465 .514

Seat

PF

PM

Mean Std. Error Lower Bound Upper Bound

95% Confidence Interval

Table 96. Workload ratings: By display condition

Table 97. Workload ratings: By crew position

Table 98. Workload ratings: Multiple comparisons of display condition

1. DisplayCond

Dependent Variable: Workload

.325 .017 .290 .359

.526 .017 .491 .560

.876 .017 .841 .911

.224 .017 .190 .259

DisplayCond

D P

MMD

RTE

VP

Mean Std. Error Lower Bound Upper Bound

95% Confidence Interv al

Multiple Comparisons

Dependent Variable: Workload

Tukey HSD

-.2010* .02465 .000 -.2657 -.1364

-.5514* .02465 .000 -.6161 -.4867

.1001* .02465 .001 .0355 .1648

.2010* .02465 .000 .1364 .2657

-.3504* .02465 .000 -.4150 -.2857

.3012* .02465 .000 .2365 .3658

.5514* .02465 .000 .4867 .6161

.3504* .02465 .000 .2857 .4150

.6515* .02465 .000 .5869 .7162

-.1001* .02465 .001 -.1648 -.0355

-.3012* .02465 .000 -.3658 -.2365

-.6515* .02465 .000 -.7162 -.5869

(J) DisplayCond

MMD

RTE

VP

D P

RTE

VP

D P

MMD

VP

D P

MMD

RTE

(I) DisplayCond

D P

MMD

RTE

VP

Mean

Diff erence

(I-J) Std. Error Sig. Lower Bound Upper Bound

95% Confidence Interv al

Based on observ ed means.

*. The mean diff erence is signif icant at the .05 lev el.
Appendix P: Post-Experiment Results 226

.486 .008 .471 .501

.488 .008 .473 .503

Seat

PF

PM

Mean Std. Error Lower Bound Upper Bound

95% Confidence Interval

P.2 Situation Awareness

Table 99. SA ratings: Levene's test of equality

Table 100. SA ratings: Test of between subject effects

Table 101. SA ratings: Means by display condition

Table 102. SA ratings: Means by crew position

Levene's Test of Equality of Error Variancesa

Dependent Variable: SA

2.106 7 80 .052

F df 1 df 2 Sig.

Tests the null hypothesis that the error variance of the

dependent variable is equal across groups.

Design:

Intercept+DisplayCond+Seat+DisplayCond * Seat

a. 

Tests of Between-Subjects Effects

Dependent Variable: SA

5.877a 7 .840 333.695 .000 .967

20.861 1 20.861 8290.713 .000 .990

5.866 3 1.955 777.067 .000 .967

9.46E-005 1 9.46E-005 .038 .847 .000

.012 3 .004 1.543 .210 .055

.201 80 .003

26.940 88

6.079 87

Source

Corrected Model

Intercept

DisplayCond

Seat

DisplayCond * Seat

Error

Total

Corrected Total

Type III Sum

of Squares df Mean Square F Sig.

Partial Eta

Squared

a. R Squared = .967 (Adjusted R Squared = .964)

1. DisplayCond

Dependent Variable: SA

.323 .011 .302 .344

.520 .011 .499 .541

.892 .011 .870 .913

.213 .011 .192 .235

DisplayCond

D P

MMD

RTE

VP

Mean Std. Error Lower Bound Upper Bound

95% Confidence Interv al
Appendix P: Post-Experiment Results 227

.310 .015 .280 .340

.335 .015 .305 .365

.511 .015 .481 .541

.529 .015 .499 .559

.909 .015 .879 .939

.874 .015 .844 .904

.213 .015 .183 .243

.213 .015 .183 .243

Seat

PF

PM

PF

PM

PF

PM

PF

PM

DisplayCond

DP

MMD

RTE

VP

Mean Std. Error Lower Bound Upper Bound

95% Confidence Interval

Table 103. SA ratings: Means by display condition and by crew position

Table 104. SA ratings: Mean differences by display condition

Multiple Comparisons

Dependent Variable: SA

Tukey HSD

-.1972* .01512 .000 -.2369 -.1575

-.5687* .01512 .000 -.6084 -.5290

.1095* .01512 .000 .0699 .1492

.1972* .01512 .000 .1575 .2369

-.3715* .01512 .000 -.4112 -.3319

.3067* .01512 .000 .2670 .3464

.5687* .01512 .000 .5290 .6084

.3715* .01512 .000 .3319 .4112

.6783* .01512 .000 .6386 .7179

-.1095* .01512 .000 -.1492 -.0699

-.3067* .01512 .000 -.3464 -.2670

-.6783* .01512 .000 -.7179 -.6386

(J) DisplayCond

MMD

RTE

VP

D P

RTE

VP

D P

MMD

VP

D P

MMD

RTE

(I) DisplayCond

D P

MMD

RTE

VP

Mean

Diff erence

(I-J) Std. Error Sig. Lower Bound Upper Bound

95% Confidence Interv al

Based on observ ed means.

*. The mean diff erence is signif icant at the .05 lev el.
Appendix P: Post-Experiment Results 228

P.3 Acceptability of Expected Taxi Data Comm message

Controller to send Expected 

Taxi msg

Flight crew to respond to 

message (1)

YES NO YES NO

Data Comm with paper

Above 10,000 feet MSL 22 20

Below 10,000 feet MSL 17 5 11 9

Final Approach Fix through roll-out 3 19 20

Taxiing Surface Operations 20 2 17 3

Data Comm with Moving Map

Above 10,000 feet MSL 22 20

Below 10,000 feet MSL 18 4 13 7

Final Approach Fix through roll-out 3 19 1 19

Taxiing Surface Operations 22 19 1

Data Comm with MMD and route

Above 10,000 feet MSL 22 20

Below 10,000 feet MSL 18 4 12 8

Final Approach Fix through roll-out 4 18 2 18

Taxiing Surface Operations 21 1 18 2

1) Used outdated questionnaire for Crew 1, therefore no question about flight crew response

No change as a function of display mode:

1A, 1B, 2A, 2B, 3B, 4A, 5A, 5B, 6A, 6B, 7B,8A, 9A, 9B, 10A, 10B, 11A, 11B

Change: 

3A: felt crew could respond between FAF and rollout in MMD + route mode

4A: felt “Expected Taxi” messages should not be sent during taxi when in paper mode

7A: felt messages during paper mode, and flight crew should not have to respond < 10,000 feet

7B, 8A: same regardless of mode, however thought crews should not have to respond <10,000 feet

8B: response depended on mode, no Data Comm FAF to roll out in paper mode

Error (?): 

4B said no “Expected Taxi” messages when in MMD + route mode, otherwise okay.
Appendix P: Post-Experiment Results 229

P.4 Trust in the System

Display comparisons were made by each crew member, comparing one display against each other. The analysis sought to determine the pilot‟s 

preference when considering least workload, highest situation awareness, highest crew coordination and highest trust. A consistency index 

subsequent to the AHP analysis suggests that the scores were inconsistent, meaning that the rater‟s priorities were loaded toward an actual 

preference. The AHP indicates the MMD+Route, in nearly all cases, was the preferred display across the constructs. The consistency index 

confirms this conclusion with variable scores across the displays and the highest preference for the MMD+Route. 

Table 105. Trust ratings: AHP preference analysis

Workload SA Crew Coordination Trust

Crew Sub 

No Seat WLVP WLDP WLMMD WLRTE SAVP SADP SAMMD SARTE CCVP CCDP CCMMD CCRTE TRVP TRDP TRMMD TRRTE

1 1 PF 0.06 0.08 0.26 0.60 0.06 0.11 0.25 0.58 0.06 0.08 0.26 0.60 0.06 0.11 0.25 0.58

1 2 PM 0.05 0.10 0.22 0.63 0.04 0.11 0.21 0.63 0.05 0.10 0.22 0.63 0.04 0.11 0.21 0.63

2 3 PF 0.05 0.08 0.24 0.63 0.05 0.10 0.24 0.61 0.08 0.09 0.24 0.59 0.08 0.09 0.19 0.65

2 4 PM 0.06 0.08 0.25 0.61 0.05 0.10 0.24 0.62 0.07 0.13 0.15 0.66 0.07 0.13 0.27 0.53

3 5 PF 0.04 0.09 0.24 0.64 0.04 0.08 0.24 0.64 0.03 0.09 0.24 0.63 0.04 0.09 0.21 0.66

3 6 PM 0.04 0.10 0.25 0.61 0.04 0.09 0.24 0.63 0.03 0.09 0.24 0.63 0.04 0.09 0.25 0.63

4 7 PF 0.04 0.09 0.25 0.62 0.05 0.08 0.28 0.60 0.05 0.09 0.25 0.61 0.06 0.10 0.22 0.62

4 8 PM 0.04 0.11 0.23 0.62 0.04 0.11 0.27 0.57 0.06 0.12 0.25 0.58 0.05 0.10 0.22 0.63

5 9 PF 0.04 0.09 0.24 0.64 0.04 0.09 0.23 0.64 0.04 0.08 0.24 0.64 0.04 0.09 0.23 0.65

5 10 PM 0.05 0.09 0.24 0.63 0.04 0.08 0.22 0.65 0.05 0.09 0.24 0.62 0.07 0.13 0.27 0.53

6 11 PF 0.04 0.09 0.24 0.63 0.04 0.10 0.25 0.61 0.05 0.09 0.22 0.64 0.04 0.11 0.31 0.54

6 12 PM 0.04 0.09 0.24 0.63 0.04 0.10 0.25 0.61 0.04 0.09 0.22 0.65 0.04 0.11 0.31 0.54

7 13 PF 0.08 0.20 0.48 0.24 0.06 0.09 0.21 0.64 0.05 0.11 0.24 0.60 0.05 0.10 0.25 0.59

7 14 PM 0.04 0.10 0.25 0.61 0.04 0.09 0.23 0.65 0.05 0.11 0.24 0.60 0.07 0.13 0.27 0.53

8 15 PF 0.04 0.09 0.25 0.62 0.05 0.09 0.24 0.63 0.04 0.10 0.25 0.61 0.04 0.10 0.24 0.61

8 16 PM 0.05 0.08 0.24 0.63 0.04 0.08 0.26 0.61 0.04 0.08 0.24 0.63 0.04 0.08 0.24 0.63

9 17 PF 0.05 0.07 0.23 0.66 0.05 0.10 0.23 0.63 0.05 0.10 0.22 0.63 0.04 0.10 0.23 0.62

9 18 PM 0.06 0.07 0.20 0.67 0.04 0.11 0.23 0.62 0.04 0.10 0.22 0.64 0.06 0.11 0.25 0.58

10 19 PF 0.04 0.09 0.23 0.65 0.04 0.11 0.24 0.61 0.04 0.10 0.23 0.62 0.07 0.13 0.27 0.53

10 20 PM 0.13 0.35 0.28 0.24 0.08 0.23 0.40 0.28 0.07 0.10 0.35 0.48 0.05 0.09 0.23 0.62

11 21 PF 0.05 0.10 0.24 0.62 0.04 0.09 0.23 0.65 0.04 0.09 0.23 0.65 0.05 0.12 0.26 0.58

11 22 PM 0.04 0.10 0.26 0.59 0.04 0.12 0.26 0.58 0.04 0.11 0.26 0.60 0.05 0.11 0.24 0.61
Appendix P: Post-Experiment Results 230

Table 106. Trust ratings: Weighted responses for consistency

Workload SA Crew Coordination Trust

Crew Sub 

No Seat WLVP WLDP WLMMD WLRTE SAVP SADP SAMMD SARTE CCVP CCDP CCMMD CCRTE TRVP TRDP TRMMD TRRTE

1 1 PF 0.05 0.10 0.25 0.60 0.05 0.10 0.62 0.22 0.06 0.09 0.27 0.58 0.06 0.13 0.25 0.55

1 2 PM 0.10 0.12 0.25 0.54 0.05 0.11 0.22 0.62 0.06 0.12 0.23 0.60 0.05 0.13 0.22 0.60

2 3 PF 0.06 0.08 0.25 0.61 0.05 0.11 0.25 0.59 0.09 0.10 0.24 0.57 0.08 0.10 0.21 0.62

2 4 PM 0.06 0.08 0.27 0.59 0.05 0.11 0.24 0.60 0.07 0.14 0.18 0.61 0.07 0.15 0.27 0.51

3 5 PF 0.04 0.12 0.25 0.58 0.05 0.11 0.26 0.59 0.04 0.13 0.25 0.58 0.05 0.11 0.22 0.62

3 6 PM 0.05 0.12 0.26 0.57 0.05 0.11 0.25 0.59 0.04 0.13 0.25 0.58 0.05 0.12 0.25 0.58

4 7 PF 0.05 0.12 0.26 0.58 0.05 0.09 0.29 0.57 0.05 0.10 0.26 0.58 0.07 0.14 0.23 0.57

4 8 PM 0.05 0.14 0.23 0.58 0.05 0.14 0.27 0.55 0.06 0.13 0.25 0.56 0.06 0.12 0.23 0.59

5 9 PF 0.04 0.12 0.25 0.58 0.04 0.12 0.24 0.59 0.05 0.11 0.26 0.59 0.05 0.11 0.24 0.60

5 10 PM 0.05 0.10 0.25 0.60 0.05 0.11 0.23 0.61 0.06 0.11 0.24 0.59 0.07 0.15 0.27 0.51

6 11 PF 0.05 0.11 0.25 0.59 0.04 0.12 0.25 0.58 0.05 0.11 0.23 0.61 0.04 0.14 0.31 0.51

6 12 PM 0.05 0.11 0.25 0.59 0.04 0.12 0.25 0.58 0.05 0.12 0.23 0.61 0.04 0.14 0.31 0.51

7 13 PF 0.08 0.18 0.42 0.32 0.06 0.11 0.23 0.61 0.06 0.13 0.25 0.56 0.06 0.12 0.27 0.55

7 14 PM 0.05 0.12 0.25 0.58 0.05 0.11 0.24 0.60 0.06 0.13 0.25 0.56 0.07 0.15 0.27 0.51

8 15 PF 0.04 0.12 0.26 0.58 0.05 0.10 0.24 0.60 0.05 0.12 0.25 0.58 0.05 0.12 0.25 0.58

8 16 PM 0.05 0.09 0.26 0.60 0.05 0.10 0.27 0.58 0.05 0.10 0.25 0.60 0.05 0.10 0.25 0.60

9 17 PF 0.05 0.08 0.26 0.62 0.05 0.11 0.24 0.59 0.05 0.13 0.23 0.59 0.05 0.13 0.24 0.58

9 18 PM 0.06 0.08 0.23 0.63 0.04 0.13 0.23 0.59 0.04 0.12 0.23 0.60 0.06 0.13 0.25 0.55

10 19 PF 0.05 0.11 0.24 0.60 0.05 0.13 0.25 0.57 0.05 0.12 0.24 0.59 0.07 0.15 0.27 0.51

10 20 PM 0.14 0.35 0.27 0.24 0.08 0.24 0.40 0.28 0.08 0.12 0.35 0.44 0.06 0.12 0.24 0.58

11 21 PF 0.05 0.13 0.25 0.57 0.05 0.11 0.24 0.60 0.05 0.11 0.24 0.60 0.06 0.14 0.26 0.54

11 22 PM 0.05 0.13 0.26 0.55 0.05 0.14 0.26 0.55 0.04 0.13 0.26 0.57 0.06 0.13 0.24 0.57

Abbreviations:

WLVP Workload, Voice/Paper SAVP SA, Voice/Paper CCVP Crew coordination,

Voice/Paper TRVP Trust, Voice/Paper

WLDP Workload, 

DataComm/Paper SADP SA, DataComm/Paper CCDP Crew coordination,DataComm/Paper TRDP Trust, DataComm/Paper

WLMMD Workload, Moving Map 

Display SAMMD SA, Moving Map Display CCMMD Crew coordination,Moving Map Display TRMMD Trust, Moving Map Display

WLRTE Workload, MMD+Route SARTE SA, MMD+Route CCRTE Crew coordination,

MMD+Route TRRTE Trust, MMD+Route
Appendix P: Post-Experiment Results 231 

.487 .005 .477 .496

.488 .005 .478 .497

Seat

PF

PM

Mean Std. Error Lower Bound Upper Bound

95% Confidence Interval

P.5 Crew Coordination 

Table 107. Crew coordination: Levene's test of equality 

Table 108. Crew coordination: Test for between subject effects

Table 109. Crew coordination: Mean ratings by display condition

Table 110. Crew coordination: Mean rating by crew position

p

1.498 7 80 .180

F df1 df2 Sig.

Tests the null hypothesis that the error variance of the

dependent variable is equal across groups.

Design:

Intercept+DisplayCond+Seat+DisplayCond * Seat

a. 

6.021a 7 .860 821.140 .000 .986

20.885 1 20.885 19938.362 .000 .996

6.019 3 2.006 1915.420 .000 .986

3.34E-005 1 3.34E-005 .032 .859 .000

.002 3 .001 .562 .642 .021

.084 80 .001

26.990 88

6.105 87

Source

Corrected Model

Intercept

DisplayCond

Seat

DisplayCond * Seat

Error

Total

Corrected Total

Type III Sum

of Squares df Mean Square F Sig.

Partial Eta

Squared

a. R Squared = .986 (Adjusted R Squared = .985)

p

.317 .007 .303 .331

.509 .007 .495 .523

.903 .007 .889 .916

.220 .007 .206 .234

DisplayCond

DP

MMD

RTE

VP

Mean Std. Error Lower Bound Upper Bound

95% Confidence Interval
Appendix P: Post-Experiment Results 232

Table 111. Crew coordination: Mean rating by display condition and crew position

Table 112. Crew coordination: Pairwise comparison by display condition

3. DisplayCond * Seat

Dependent Variable: CrewCoord

.309 .010 .290 .329

.324 .010 .305 .344

.509 .010 .490 .529

.509 .010 .489 .528

.908 .010 .888 .927

.898 .010 .878 .917

.220 .010 .200 .239

.220 .010 .201 .240

Seat

1.00

2.00

1.00

2.00

1.00

2.00

1.00

2.00

DisplayCond

D P

MMD

RTE

VP

Mean Std. Error Lower Bound Upper Bound

95% Confidence Interv al

Multiple Comparisons

Dependent Variable: CrewCoord

Tukey HSD

-.1922* .00976 .000 -.2178 -.1666

-.5858* .00976 .000 -.6114 -.5602

.0971* .00976 .000 .0714 .1227

.1922* .00976 .000 .1666 .2178

-.3936* .00976 .000 -.4192 -.3680

.2892* .00976 .000 .2636 .3148

.5858* .00976 .000 .5602 .6114

.3936* .00976 .000 .3680 .4192

.6829* .00976 .000 .6573 .7085

-.0971* .00976 .000 -.1227 -.0714

-.2892* .00976 .000 -.3148 -.2636

-.6829* .00976 .000 -.7085 -.6573

(J) DisplayCond

MMD

RTE

VP

D P

RTE

VP

D P

MMD

VP

D P

MMD

RTE

(I) DisplayCond

D P

MMD

RTE

VP

Mean

Diff erence

(I-J) Std. Error Sig. Lower Bound Upper Bound

95% Confidence Interv al

Based on observ ed means.

*. The mean diff erence is signif icant at the .05 lev el.
Appendix P: Post-Experiment Results 233

P.6 Summary

The results of the post-experiment summary questions (Appendix E.6) are tabulated here. 

#1: To what degree did the scenarios in this experiment accurately simulate a complex, high-workload 

environment? If not, what was missing? (Scale of 1 “realistic” to 7 “unrealistic”)

Rating Number of 

responses

Comments (number of crews making the comment)

1 5 very good (one crew rated ground ops 1 and inflight ops 5)

2 12 add flight crew interaction (1) ; have more radio comm while airborne (1)

3 1 allow use of auto-pilot, have more comm while airborne (1)

4 1 ground operations very good, however airborne operations were generic

5 2 need more radio comm while airborne (ground ops very good)

6 0

7 0

Mean=2.6 N = 21 The first crew (2 pilots) were not asked this question, one crew scored twice

#2: What is your overall assessment of the potential of communicating clearance updates or changes using 

data link while an aircraft is taxiing or in busy terminal area? (Scale of 1 “realistic” to 7 “unrealistic”)

Rating Number of 

responses

Comments (number of crews making the comment)

1 3

2 13 if immediate response not required (2); close to implementable as in experiment (2)

3 2

4 0

5 2 getting new clearance so close to taxiway intersection is problematic (1); prefer Voice

so I am in the communication loop of what other aircraft are doing (1)

6 2 fairly unrealistic (1)

7 0

Mean=2.6 N = 22

#3: Should the dotted cyan lines for an “Expected Taxi” clearance include red hold short bars?

Rating Number of 

responses

Comments (number of crews making the comment)

Yes 17

No 3

- N = 20 The first crew (2 pilots) were not asked this question

#4: Will the solid magenta line for a Taxi clearance on the ND encourage crew members to begin taxiing

prior to receiving the Voice message from ATC?

Rating Number of 

responses

Comments (number of crews making the comment)

Yes 4 need to add text to end of Data Comm message saying “Contact ATC xxx.xx” (1)

Maybe 11 however training and operational procedures should be sufficient (11)

No 5

- N = 20 The first crew (2 pilots) were not asked this question
Appendix P: Post-Experiment Results 234

#5: Was the simultaneous Voice and Data Comm instructions to cross active runway clear? Was there a 

delay in the FO updating the graphical display on the ND? Was the delay important?

Rating Number of 

responses

Comments (number of crews making the comment)

Yes 16 but not a good time to be Head Down (1); high workload as implemented (1); Voice

comm should have priority (1); delay responding until across the runway (1)

No 3 Data Comm probably not necessary (1); did not like going Head Down (1)

- N = 19 The first crew (2 pilots) were not asked this question, one crew did not respond

#6: How would CDTI impact your workload, SA, and acceptability of using Data Comm messages in 

terminal airspace or surface operations?

Rating

n/a

Number of 

responses

Comments (number of crews making the comment)

- 16 helpful, increase in SA outweighs possible increase in workload, less Voice comm

- 1 may cause overload, but very useful in low visibility conditions

- 2 might cause more Head Down time, but eliminate confusion, less radio congestion

- 1 would slow down operations if information too cluttered

- N = 20 The first crew (2 pilots) were not asked this question

#7: Was the use of Voice by the controller for critical or time sensitive information (such as crossing the 

runway) appropriate and necessary?

Rating Number of 

responses

Comments (number of crews making the comment)

Yes 20 Voice is quicker; has priority for crew attention; hard time trusting Data Comm

No 0

- N = 20 The first crew (2 pilots) were not asked this question

#8: Were there any challenges with Data Comm unique to your flight duties as PF or PM?

Rating

n/a

Number of 

responses

Comments (number of crews making the comment)

- 4 significant increase in Head Down time and workload for the PM

- 2 difficult for the PF to stay in the information loop; a challenge to keep crew member 

informed; prioritizing messages and tasks

- 2 difficult keeping CDU and ND aligned; respond on second CDU page caused errors 

and too much Head Down time

- N = 8 The first crew (2 pilots) were not asked this question, many crews did not respond

#9: Do you have any other comments?

Rating

n/a

Number of 

responses

Comments (number of crews making the comment)

- 4 like and prefer Data Comm, very useful if integrated with MMD and route, otherwise 

limited benefit except for language barriers

- 4 no Data Comm messages when time critical, safety related, or on runway

- 2 entire airspace system would greatly benefit from this enhancement in safety, the 

sooner the better for all, looking forward to seeing this on the flight line

- 1 use a different color or bold text to show most recent clearance

- N = 11 The first crew (2 pilots) were not asked this question, many crews did not respond
REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

2. REPORT TYPE 

Technical Publication

 4. TITLE AND SUBTITLE

Flight Crew Workload, Acceptability, and Performance When Using Data 

Comm in a High-Density Terminal Area Simulation

5a. CONTRACT NUMBER

 6. AUTHOR(S)

Norman, R. Michael; Baxley, Brian T.; Adams, Catherine A.; Ellis, Kyle 

K. E.; Latorella, Kara A.; Comstock, James R., Jr

 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

NASA Langley Research Center

Hampton, VA 23681-2199

 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

National Aeronautics and Space Administration

Washington, DC 20546-0001

 8. PERFORMING ORGANIZATION

 REPORT NUMBER

L-19907

10. SPONSOR/MONITOR'S ACRONYM(S)

NASA

13. SUPPLEMENTARY NOTES

12. DISTRIBUTION/AVAILABILITY STATEMENT

Unclassified - Unlimited

Subject Category 03

Availability: NASA CASI (443) 757-5802

19a. NAME OF RESPONSIBLE PERSON

STI Help Desk (email: help@sti.nasa.gov)

14. ABSTRACT

NASA and the FAA collaborated to conduct a simulator experiment using 22 commercial airline pilots to determine the effect of using Data Comm to issue 

messages during busy, terminal area operations. Four equipage conditions were used: Voice communication only, Data Comm only, Data Comm with Moving 

Map Display, and Data Comm with Moving Map displaying taxi route. Each condition was used in an arrival and a departure scenario at Boston Logan 

Airport. Of particular interest was the flight crew response to D-TAXI, the use of Data Comm by Air Traffic Control (ATC) to send taxi instructions. 

Quantitative data was collected on subject reaction time, flight technical error, operational errors, and eye tracking information. Questionnaires collected 

subjective feedback on workload, situation awareness, and acceptability to the flight crew. Results showed that 95% of the Data Comm messages were 

responded to by the flight crew within one minute and 97% of the messages within two minutes, and flight crews reported the Expected D-TAXI messages as 

useful. Flight crews reported the use of Data Comm as unacceptable when issuing instructions to cross an active runway, and issuing D-TAXI messages 

between the Final Approach Fix and 80 knots during landing roll. 

15. SUBJECT TERMS

Air traffic control; Data communication; Datalinked taxi routes; Pilot workload

18. NUMBER

 OF 

 PAGES

254

19b. TELEPHONE NUMBER (Include area code)

(443) 757-5802

a. REPORT

U

c. THIS PAGE

U

b. ABSTRACT

U

17. LIMITATION OF 

 ABSTRACT

UU

Prescribed by ANSI Std. Z39.18

Standard Form 298 (Rev. 8-98)

3. DATES COVERED (From - To)

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

031102.02.07.06.9D36.09

11. SPONSOR/MONITOR'S REPORT

 NUMBER(S)

NASA/TP-2013-218007

16. SECURITY CLASSIFICATION OF:

The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, 

gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this 

collection of information, including suggestions for reducing this burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and 

Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person 

shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY)

01- 06 - 2013