ote technical note techn

Document is available to the public

through the National Technical Information

Service, Springfield, Virginia 22161

U.S. Department of Transportation

Federal Aviation Administration

William J. Hughes Technical Center

Atlantic City International Airport, NJ 08405

ote technical note techn

Document is available to the public

through the National Technical Information

Service, Springfield, Virginia 22161

U.S. Department of Transportation

Federal Aviation Administration

William J. Hughes Technical Center

Atlantic City International Airport, NJ 08405

ote technical note techn

Document is available to the public

through the National Technical Information

Service, Springfield, Virginia 22161

U.S. Department of Transportation

Federal Aviation Administration

William J. Hughes Technical Center

Atlantic City International Airport, NJ 08405

ote technical note techn

Document is available to the public

through the National Technical Information

Service, Springfield, Virginia 22161

U.S. Department of Transportation

Federal Aviation Administration

William J. Hughes Technical Center

Atlantic City International Airport, NJ 08405

ote technical note techn

Document is available to the public

through the National Technical Information

Service, Springfield, Virginia 22161

U.S. Department of Transportation

Federal Aviation Administration

William J. Hughes Technical Center

Atlantic City International Airport, NJ 08405

ote technical note techn

Document is available to the public

through the National Technical Information

Service, Springfield, Virginia 22161

U.S. Department of Transportation

Federal Aviation Administration

William J. Hughes Technical Center

Atlantic City International Airport, NJ 08405

ote technical note techn

Document is available to the public

through the National Technical Information

Service, Springfield, Virginia 22161

U.S. Department of Transportation

Federal Aviation Administration

William J. Hughes Technical Center

Atlantic City International Airport, NJ 08405

ote technical note techn

Document is available to the public

through the National Technical Information

Service, Springfield, Virginia 22161

U.S. Department of Transportation

Federal Aviation Administration

William J. Hughes Technical Center

Atlantic City International Airport, NJ 08405

Human-in-the-Loop Simulation 

Evaluating the Collocation of the User 

Request Evaluation Tool, Traffic 

Management Advisor, and ControllerPilot Data Link Communications: 

Experiment I – Tool Combinations 

Randy L. Sollenberger, Ph.D., ATO-P, 

Ben Willems, ATO-P, 

Pamela S. Della Rocco, Ph.D., ATO-P, 

Anton Koros, Northrop Grumman Information Technology 

Todd Truitt, Ph.D., ATO-P 

February 2005 

DOT/FAA/CT-TN04/28 
NOTICE 

This document is disseminated under the sponsorship of the U.S. 

Department of Transportation in the interest of information 

exchange. The United States Government assumes no liability for 

the contents or use thereof. The United States Government does 

not endorse products or manufacturers. Trade or manufacturer's 

names appear herein solely because they are considered essential 

to the objective of this report. 

This report is available at the Federal Aviation Administration, 

William J. Hughes Technical Center’s full text, technical reports 

web site: http://actlibrary.tc.faa.gov in Adobe Acrobat portable 

document format (PDF). 
Technical Report Documentation Page 

1. Report No. 

DOT/FAA/CT-TN04/28 

2. Government Accession No. 3. Recipient’s Catalog No. 

5. Report Date 

February 2005 

4. Title and Subtitle 

Human-In-The-Loop Simulation Evaluating the Collocation of the User Request 

Evaluation Tool, Traffic Management Advisor, and Controller-Pilot Data Link 

Communications: Experiment I – Tool Combinations 6. Performing Organization Code 

7. Author(s) 

Randy L. Sollenberger, Ph.D., ATO-P, Ben Willems, ATO-P, Pamela S. Della Rocco, 

Ph.D., ATO-P, Anton Koros, Northrop Grumman Information Technology, and Todd 

Truitt, Ph.D., ATO-P 

8. Performing Organization Report No. 

DOT/FAA/CT-TN04/28 

9. Performing Organization Name and Address 10. Work Unit No. (TRAIS) 

Federal Aviation Administration 

NAS Human Factors Group, ATO-P 

William J. Hughes Technical Center 

Atlantic City International Airport, NJ 08405 

11. Contract or Grant No. 

13. Type of Report and Period Covered 

Technical Note 

12. Sponsoring Agency Name and Address 

Federal Aviation Administration 

Human Factors Division 

800 Independence Avenue S.W. 

Washington, DC 20591 

14. Sponsoring Agency Code 

ATO-P 

15. Supplementary Notes 

16. Abstract 

The Federal Aviation Administration (FAA) Free Flight Program successfully deployed the User Request Evaluation Tool (URET), 

Traffic Management Advisor (TMA), and Controller-Pilot Data Link Communications (CPDLC) to a limited number of Air Route 

Traffic Control Centers (ARTCCs). As deployment expands nationwide, several facilities may eventually receive all three tools. 

Before this occurs, it is important to identify any potential human factors issues that may arise due to the collocation of these tools at 

the controller’s workstation. In this report, we present the first of three high fidelity human-in-the-loop simulation experiments we 

conducted to evaluate the impact of URET, TMA, and CPDLC collocation on controller workload, situational awareness, and 

teamwork. We examined collocation issues with a “stovepipe” independent configuration where none of the tools were integrated or 

directly communicated with each other. In this first experiment, twelve Air Traffic Control Specialists (ATCSs) participated as Rside/D-side controller teams operating a high altitude generic sector using all combinations of the three tools. The most important 

collocation issue identified was that controllers had difficulty accessing important information on the D-side display when URET and 

CPDLC were both operational (i.e., display clutter). Although neither tool alone caused display clutter, both tools in combination 

made it difficult for D-side controllers to find the information they needed quickly. This was especially true for accessing CPDLC 

windows, which became covered when controllers used URET. Another collocation issue was that D-side controllers had to access 

TMA delay time information from the R-side display. Controllers thought it was important to have TMA information available on the 

D-side display where it could be easily accessed by D-side controllers. However, controllers were concerned that simply showing the 

TMA List on the D-side might add to the D-side display clutter. Good human factors design principles prescribe that users must have 

immediate access to important information and that critical information should never be covered. A “stovepipe” independent 

deployment of these tools will result in impaired access to timely information. The results of this study indicated that better efforts 

should be made towards integrating the information from URET, TMA, and CPDLC on the D-side monitor prior to deployment. 

17. Key Words 

Air Traffic Control, Aviation Human Factors, Computer-Human 

Interaction, Controller-Pilot Data Link Communications, Human-in-theLoop Simulation, Traffic Management Advisor, and User Request 

Evaluation Tool. 

18. Distribution Statement 

This document is available to the public through 

the National Technical Information Service, 

Springfield, Virginia, 22161 

19. Security Classif. (of this report) 

Unclassified 

20. Security Classif. (of this page) 

Unclassified 

21. No. of Pages 

84 

22. Price 

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized 
iii

Table of Contents 

Page 

Executive Summary.........................................................................................................................v 

1. Introduction..................................................................................................................................1 

1.1 Background............................................................................................................................1 

1.1.1 Previous Research...........................................................................................................1 

1.1.2 Human Factors Issues .....................................................................................................3 

1.1.3 Real-Time Human-in-the-Loop Simulation....................................................................3 

1.2 Assumptions...........................................................................................................................3 

1.3 Purpose...................................................................................................................................4 

1.4 Specific Objectives ................................................................................................................4 

2. Method .........................................................................................................................................5 

2.1 Participants.............................................................................................................................5 

2.2 Test Facility and Equipment ..................................................................................................5 

2.2.1 Airspace ..........................................................................................................................6 

2.2.2 Traffic Scenarios.............................................................................................................8 

2.3 Experiments ...........................................................................................................................8 

2.3.1 Independent Variables.....................................................................................................8 

2.3.2 Dependent Variables.......................................................................................................9 

2.4 Training................................................................................................................................10 

2.5 Procedure .............................................................................................................................10 

3. Results........................................................................................................................................12 

3.1 Objective #1: Assess whether controllers have access to information when needed and 

whether they interpret available information correctly .......................................................12 

3.2 Objective #2: Assess R-side/D-side Roles and Responsibilities .........................................17 

3.3 Objective #3: Assess R-side/D-side Workload and Situational Awareness ........................18 

3.4 Objective #4: Access R-side/D-side Teamwork and Communications...............................23 

3.5 Other Measures....................................................................................................................26 

3.5.1 System Effectiveness Measures....................................................................................26 

3.5.2 Exit Questionnaire Ratings ...........................................................................................27 

4. Discussion..................................................................................................................................28 

5. Conclusion .................................................................................................................................30 

References......................................................................................................................................31 

Acronyms.......................................................................................................................................33 

Appendices 

A - Informed Consent Form 

B - Background Questionnaire 

C - Generic Airspace 

D - Post-Scenario Questionnaire 

E - Observer Rating Form 

F - Exit Questionnaire 
iv

List of Illustrations 

Figures Page 

Figure 1. Generic high altitude airspace. .........................................................................................7 

Figure 2. Mean controller ratings for how often CPDLC caused display clutter provided by 

the R-side and D-side positions for each of the CPDLC tool conditions.....................13 

Figure 3. Mean controller ratings for how often URET caused display clutter provided by 

the D-side position for each of the URET tool conditions. ..........................................14 

Figure 4. Typical configuration of URET and CPDLC on the D-side display..............................15 

Figure 5. URET aircraft list covering CPDLC windows on the D-side display............................15 

Figure 6. Mean observer ratings of overall prioritization effectiveness for controller teams 

for each of the tool conditions. .....................................................................................16 

Figure 7. Mean number of CPDLC TOC messages sent by R-side and D-side for each of the 

CPDLC tool conditions. ...............................................................................................17 

Figure 8. Mean ATWIT workload ratings provided by the R-side and D-side positions for 

each of the tool conditions............................................................................................18 

Figure 9. Mean NASA-TLX mental demand workload ratings provided by the R-side and Dside positions for each of the tool conditions. ..............................................................19 

Figure 10. Mean number of ground-to-air transmissions made by the R-side controller for 

each of the tool conditions............................................................................................20 

Figure 11. Mean number of airspeed changes made by the R-side controller for each of the 

tool conditions. .............................................................................................................21

Figure 12. Mean SA ratings made by the R-side and D-side positions for each of the tool 

conditions......................................................................................................................22 

Figure 13. Mean number of handoffs accepted by the R-side and D-side positions for each of 

the tool conditions. .......................................................................................................23 

Figure 14. Mean number of handoffs initiated by the R-side and D-side positions for each of 

the eight tool conditions. ..............................................................................................24 

Figure 15. Mean number of flight plan readouts by the R-side and D-side positions for each 

of the eight tool conditions. ..........................................................................................25 

Figure 16. Mean flight distance per aircraft for controller teams for each of the tool 

conditions......................................................................................................................26 

Tables Page 

Table 1. Controller Tools.................................................................................................................6 

Table 2. Experimental Design..........................................................................................................8 

Table 3. Objective Measures............................................................................................................9 

Table 4. Subjective Measures ..........................................................................................................9 

Table 5. Daily Schedule.................................................................................................................11 

Table 6. Experimental Condition Counterbalancing .....................................................................11 

Table 7. Controller Ratings of Simulation Realism.......................................................................27 
v

Executive Summary 

The Federal Aviation Administration (FAA) established the Free Flight Program in collaboration 

with the aviation community to increase capacity (airport and airspace) and improve efficiency 

(flight times and fuel consumption) while maintaining the current high level of safety. An 

important goal of the Free Flight Program was the delivery of new air traffic control (ATC) 

technologies focused on early benefits to users of the National Airspace System (NAS). These 

capabilities included the User Request Evaluation Tool (URET), Traffic Management Advisor 

(TMA), and Controller-Pilot Data Link Communications (CPDLC) as en route controller tools. 

Under the Free Flight Program, the FAA deployed these tools independently to a limited number 

of Air Route Traffic Control Centers (ARTCCs) nationwide. As deployment expands to other 

facilities, all three tools may be collocated at the sector workstation. 

Different designers developed URET, TMA, and CPDLC with the assumption that each system 

would operate independently. As tool deployment expands nationwide, several facilities may 

eventually receive all three tools. Before this occurs, it is important to identify any potential 

human factors issues that may arise due to the collocation of these tools at the controller’s 

workstation. For example, will controllers be able to access the information they need quickly 

without confusing data from different systems? How will controller communications between 

team members be affected? How will the collocation of these tools change the roles and 

responsibilities of team members? What new training or procedures may be required? The FAA 

Free Flight Program Office (AOZ) and the Human Factors Research and Engineering Division 

(ATO-P) sponsored this research to address these important questions. 

In this report, we present the first of three human-in-the-loop simulation experiments we 

conducted to evaluate the impact of URET, TMA, and CPDLC collocation on controller 

workload, situational awareness, and teamwork. The first experiment examined R-side/D-side 

controller teams working a high altitude sector using different combinations of the three tools at 

a single sector. The second experiment examined controller teams interacting with each other 

while working a high and a low altitude sector and using all of the tools. The third experiment 

examined controllers working a high altitude sector alone without a D-side and using all of the 

tools. We will present the second and third experiments in a subsequent report. 

Twelve Air Traffic Control Specialists (ATCSs) from Level 11 and Level 12 ARTCCs 

nationwide participated in this study. We recruited six participants from ARTCCs where URET 

is operational and six participants from ARTCCs where TMA is operational. All six ATCSs 

from the URET facilities were URET current and proficient. However, only five ATCSs from 

the TMA facilities were TMA current and proficient. The participant who was not TMA 

qualified, received TMA training on our ATC simulator. We trained all twelve participants in 

CPDLC after arriving at the FAA, William J. Hughes Technical Center (WJHTC). Also, all 

participants received some cross-training in URET and TMA. Each controller team consisted of 

one TMA-qualified ATCS operating the R-side (Radar) and one URET-qualified ATCS 

operating the D-side (Data) position. 

We conducted the experiment in the FAA, WJHTC Research, Development, and Human Factors 

Laboratory (RDHFL) using our high fidelity ATC simulator, the Distributed Environment for 

Simulation, Rapid Engineering, and Experimentation (DESIREE). DESIREE emulated en route 
vi

Display System Replacement (DSR) functions and was configured with URET and TMA 

prototypes, as well as CPDLC Build 1A functionality. We deployed TMA and CPDLC on the 

R-side Sony 2K monitor and URET and CPDLC on the D-side 21” flat-panel monitor. The 

CPDLC services were: Transfer of Communications (TOC), Altimeter Setting (AS), Initial 

Contact (IC), and Menu Text (MT) Messages. 

The study consisted of three 2-week sessions with a different group of four ATCSs participating 

in each session. In the first week, controllers completed 18 hours of practice scenarios to 

become familiar with the generic high altitude sector selected for this simulation and the three 

tools. In the second week, controllers completed eight test scenarios under experimental 

conditions with different combinations of URET, TMA, and CPDLC. In a baseline condition, 

participants controlled traffic without any tools. In other conditions, participants completed 

scenarios using each tool separately, as well as two and three tools together. We 

counterbalanced the presentation order of the eight tool combinations to experimentally control 

for practice effects. 

The most important collocation issue identified in this experiment was that controllers had 

difficulty accessing important information on the D-side display when URET and CPDLC were 

both operational (i.e., display clutter). Controller ratings indicated that CPDLC caused a great 

deal of display clutter on the D-side monitor. Neither URET alone nor CPDLC alone caused 

display clutter. However, both tools in combination made it difficult for D-side controllers to 

find the information they needed quickly. This was especially true for accessing CPDLC 

windows, which became covered when controllers used URET. 

It is important to note the controllers identified this D-side display clutter issue using the D-side 

CPDLC CHI we developed in the RDHFL for use in this simulation study. We designed the 

D-side CHI to be consistent with a “stovepipe” independent deployment of the tools with simple 

features to help controllers manage the multiple windows associated with each tool. This 

specific D-side CHI was not intended to be the interface that will be deployed to ARTCCs in the 

future. 

Another collocation issue identified in this experiment was that D-side controllers had to access 

TMA delay time information from the R-side display. Controllers thought it was important to 

have TMA information available on the D-side display where it could be easily accessed by Dside controllers. However, controllers were concerned that simply showing the TMA List on the 

D-side might add to the D-side display clutter. 

Controller workload ratings indicated that D-side workload tended to increase when two and 

three tools were operational. However, D-side workload ratings were only moderate and never 

reached a high level for the moderate traffic scenarios we used in the simulation. We also 

examined the number of ground-to-air voice transmissions and airspeed, heading, and altitude 

changes as additional indicators of controller workload. None of these measures increased 

greatly with multiple tool use. 

In general, controllers rated their situational awareness as very high during the simulation. 

However, there was a situational awareness issue with the CPDLC TOC service. R-side 

controllers sent most of the CPDLC TOCs to aircraft. Although D-side controllers did not use 
vii

the TOC service very often, controllers still expressed concern about not knowing what their 

team member was doing with CPDLC. Unlike voice communications, there were no audible 

cues with CPDLC to help controllers maintain situational awareness of their team member’s 

actions. Controllers had to visually monitor the CPDLC Message Out window to know when 

their team member sent a TOC message. If the CPDLC display was covered by URET, the 

D-side controller could easily miss a sent message. 

Good human factors design principles prescribe that users must have immediate access to 

important information and that critical information should never be covered. A “stovepipe” 

independent deployment of these tools will result in impaired access to timely information. The 

results of this study indicated that better human factors efforts should be made towards 

integrating the information from URET, TMA, and CPDLC. Even if these systems cannot be 

entirely integrated, we should explore integrating the displays on the D-side monitor prior to 

deployment. 
1 

1. Introduction

In 1998, the Federal Aviation Administration (FAA) established the Free Flight Program in 

collaboration with the aviation community to increase capacity (airport and airspace) and 

improve efficiency (flight times and fuel consumption) while maintaining the current high level 

of safety. An important goal of the Free Flight Program was the delivery of new air traffic 

control (ATC) technologies focused on early benefits to users of the National Airspace System 

(NAS). These capabilities included the User Request Evaluation Tool (URET), Traffic 

Management Advisor (TMA), and Controller-Pilot Data Link Communications (CPDLC) as en 

route controller tools. Under the Free Flight Program, the FAA deployed these tools 

independently to a limited number of Air Route Traffic Control Centers (ARTCCs) nationwide. 

As deployment expands to other facilities, all three tools may be collocated at the sector 

workstation. 

Designers developed the three systems independently; therefore, we need to investigate how Air 

Traffic Control Specialists (ATCSs) will interact with the three tools before they are deployed 

together at the same sector. When evaluating tools that will introduce changes or add to the 

number of systems used in ATC, it is important to identify any potential problems that might 

arise from the introduction of these new tools. Identifying problems and correcting them before 

they can negatively impact performance in the field is critical in ATC where safety is potentially 

at stake. Therefore, the FAA Free Flight Program Office (AOZ) and the Human Factors 

Research and Engineering Division (ATO-P) sponsored this study to examine the impact of 

collocating URET, TMA, and CPDLC at the controller’s workstation. 

In this report, we present the first of three separate experiments we conducted to assess the 

human factors issues of collocating URET, TMA, and CPDLC. The first experiment examined 

R-side/D-side controller teams working a high altitude sector using different combinations of the 

three tools at a single sector. The second experiment examined controller teams interacting with 

each other while working a high and a low altitude sector and using all of the tools. The third 

experiment examined controllers working a high altitude sector alone without a D-side and using 

all of the tools. We will present the second and third experiments in a subsequent report. 

1.1 Background

1.1.1 Previous Research

Several studies have examined issues related to the collocation of URET, TMA, and CPDLC. 

Desenti, Gross, and Toma (2000) examined the use of URET and TMA. The authors questioned 

whether there was an emerging concept of use for URET and TMA in which the trial planning 

capability of URET may be compatible to meet the metering times of TMA. A potential human 

factors issue arising from this study was that URET and TMA used different algorithms to 

compute an aircraft’s future location. 

The NAS Advanced Concepts Branch at the FAA, William J. Hughes Technical Center 

(WJHTC) performed the second study. These researchers conducted a real-time human-in-theloop (HITL) simulation that examined the use of URET Core Capability Limited Deployment 

(CCLD) and TMA. They determined that the URET list sequences were different from the 
2 

aircraft sequences on TMA. However, they concluded that, because URET and TMA were 

independent tools, there should be little or no negative impact on safety. 

In a third study, Kerns (2001) of the MITRE Center for Advanced Aviation System 

Development examined the human factors issues related to collocating URET and CPDLC. She 

concluded that the Data (D)-side workload would increase, and the design of the Computer 

Human Interface (CHI) on the D-side was critical to the successful deployment of URET and 

CPDLC. Kerns also noted there would be changes in the roles and responsibilities of the Radar 

(R)-side and D-side positions. 

Della Rocco, Panjwani, Friedman-Berg, Kopardekar, and Hah (in press) performed a fourth 

study. These researchers conducted a “cognitive walkthrough” that explored the collocation of 

URET, TMA, and CPDLC. The cognitive walkthrough is a technique that human factors 

researchers use to evaluate the design of a user interface, with special attention to how well the 

interface supports first-time use without formal training. Della Rocco et al. raised questions 

about how the three tools update the NAS. They also identified a number of CHI inconsistencies 

across the three tools. They concurred with Kerns (2001) that when all three tools were 

deployed together, the roles and responsibilities of the R-side and D-side positions needed 

clarification. Further, applying human factors principles to these issues will help ATCSs use the 

tools as intended and attain the expected benefits. 

As stated previously, researchers have conducted subjective evaluations of these tools (Desenti, 

Gross, & Toma, 2000; Kerns, 2001). Whereas Desenti, Gross, and Toma (2000) investigated the 

collocation of URET and TMA, and Kerns (2001) examined the collocation of URET and 

CPDLC; neither study involved a real-time HITL simulation. In the study performed at the NAS 

Advanced Concepts Branch (2001), researchers conducted a real-time HITL simulation that 

examined the collocation of URET and TMA. That study, however, gathered no objective 

performance measures. Della Rocco et al. (in press) explored the collocation of URET, TMA, 

and CPDLC by performing a cognitive walkthrough using Subject Matter Experts (SMEs) to 

identify potential human factors issues related to the deployment of the three tools at the same 

sector. To date, there are no real-time HITL simulation studies that have objectively examined 

ATCS performance while employing all three tools. 

Researchers have made several recommendations related to human factors issues involving realtime HITL simulations. First, Della Rocco et al. (in press) recommended that the inconsistencies 

in the CHIs across systems should be examined for their impact on actual ATC performance. 

Second, a real-time HITL simulation should examine the procedures defining the roles and 

responsibilities for the R-side and D-side positions and where to implement those procedures. 

More specifically, they further recommended the need to examine the effect of the increased 

information from all three tools (i.e., URET, TMA, and CPDLC), the effect of the increased 

communication requirements, the effect of the different tools updating NAS, any issues related to 

clutter on both the R-side and D-side, and any issues unique to a single R-side operation. Thus, 

an HITL simulation was the logical next step to evaluate the collocation of these three tools at 

the en route sector workstation. 
3 

1.1.2 Human Factors Issues

In this study, we examined the primary issues of whether there will be any change in controller 

performance, workload, or situational awareness (SA) due to the collocation of URET, TMA, 

and CPDLC. Best human factors practices recommend that information be provided to ATCS in 

a timely manner that is consistent with other information and easily understood by the ATCS. 

Otherwise, there may be a detrimental impact on performance, workload, and SA. In addition, 

when URET and CPDLC are collocated on the D-side, there may be an increase in 

communication between the R-side and D-side, which again may have a significant impact on 

controller performance, workload, and SA (Cardosi & Murphy, 1995). Furthermore, a 

“stovepipe” independent deployment of the three tools will result in information being presented 

over a more diverse area requiring the controller to not only search for the appropriate data, but 

also to integrate it. 

1.1.3 Real-Time Human-in-the-Loop Simulation

One of the best ways to gain valuable insight into the impact of new automation and controller 

tools is to conduct real-time HITL simulations (Manning, 2000), where ATCSs interact with 

realistic models of the tools and perform as they would in an actual Air Route Traffic Control 

Center (ARTCC). As Manning points out, real-time HITL simulations “allow controllers to 

direct the activities of a sample of simulated air traffic, performing characteristic functions such 

as ordering changes in aircraft speed or flight path, all within a relatively standardized work 

sample.” In this study, we conducted a real-time HITL simulation to explore how the collocation 

of URET, TMA, and CPDLC will affect ATCS performance. HITL simulations surface issues 

that would not otherwise be apparent until after deployment. 

1.2 Assumptions

The assumptions for the current study were the same as those adopted for the cognitive 

walkthrough study conducted by Della Rocco et al. (in press). Specifically, 

a. The system will represent today’s Display System Replacement (DSR) environment 

with the BCC21 CHI designed by the Air Traffic Design Evolution Team (ATDET). 

The D-side position will have a 21” flat panel monitor that is used to display URET and 

the D-side Computer Readout Display (CRD). This same monitor will display CPDLC 

information in our simulation. 

b. The system will be implemented with URET, TMA, and CPDLC being independent and 

not having direct communication with each other, but each tool will communicate to and 

from the simulated host computer. 

c. Controllers will procedurally use CPDLC only for non-time-critical instructions to 

pilots, and the tool will be available at both the R-side and D-side positions. 

d. To implement CPDLC at both positions for the present study, the D-side position will 

have CPDLC windows on the D-side monitor similar to the ATDET CPDLC Build IA 

windows on the R-side. These will be the CPDLC Message Out, Menu Text, and 

History windows with the same interactive capability in these windows allowed on the 

R-side. 
4 

e. In addition, the D-side controller will be able to use text-based commands in the D-side 

CRD as well as “hot keys” on the D-side keyboard for CPDLC functions. 

f. The D-side position will not have interactive aircraft data blocks for CPDLC functions. 

g. The URET Aircraft List will replace paper flight progress strips in CCLD (FAA, 2002) 

as in the deployments at Memphis and Indianapolis ARTCCs. 

1.3 Purpose

We intended the present study to be the first in a series of studies that will examine the 

collocation of URET, TMA, and CPDLC with the following general purposes: 

a. assess the human factors implications of collocating these three tools at the same sector, 

b. understand the cumulative effects from a research perspective on workload, SA, and 

teamwork, 

c. identify areas in which mitigation strategies are needed before the tools are deployed, 

and 

d. assess the efficiency and capacity benefits of collocating these three tools at the same 

sector. 

We used the work performed by Della Rocco et al. (in press) and others (Desenti et al., 2000; 

Kerns, 2001) to design and conduct a real-time HITL simulation that explored whether the 

collocation of URET, TMA, and CPDLC tools had any impact on ATCS performance, workload, 

and SA. In the first experiment, we examined all possible combinations of these three tools with 

both R-side and D-side controller teams. In a second experiment, we examined a high altitude 

and a low altitude sector configuration to determine if ATCS performance differs as a function of 

sector characteristics. In a third experiment, we investigated ATCS performance when working 

the R-side position unassisted. The results of the second and third experiments will follow in a 

subsequent report. In this report, we present the results from the first experiment. 

1.4 Specific Objectives

The specific objectives of the present study are 

1. to assess whether controllers have access to information when needed and whether they 

interpret available information correctly, 

2. to assess R-side/D-side roles and responsibilities, 

3. to assess R-side/D-side teamwork and communications, 

4. to assess R-side/D-side workload and workload distribution, 

5. to identify information interactions between the tools that require resolution, and 

6. to surface any other important human factors issues not previously anticipated. 
5 

2. Method

2.1 Participants

Ten male and two female ATCSs from Level 11 and Level 12 ARTCCs nationwide participated 

in this study. We recruited six participants from ARTCCs where URET is operational and six 

participants from ARTCCs where TMA is operational. All six ATCSs from the URET facilities 

were URET current and proficient. However, only five ATCSs from the TMA facilities were 

TMA current and proficient. The participant who was not TMA qualified, received TMA 

training on our ATC simulator. We trained all twelve participants in CPDLC after arriving at the 

WJHTC. Also, all participants received some cross-training in URET and TMA. Each 

controller team consisted of one TMA-qualified ATCS operating the R-side and one URETqualified ATCS operating the D-side position. 

All participants were nonsupervisory, certified professional controllers who were qualified at 

their facility and held a current medical certificate. The medical certification ensured that all 

participants were in good health and had normal or corrected-to-normal vision and hearing. The 

controllers completed an Informed Consent Form (Appendix A) prior to participating in the 

study. The consent form described the study and stated that participation was voluntary and that 

controllers may withdraw from the study at any time for any reason. 

Each controller completed a Background Questionnaire (Appendix B) to describe the general 

demographic characteristics of participants in the study. The controllers ranged in age from 28 

to 47 years old with a mean of 40.52 years and ranged in FAA experience from 5 to 21 years of 

active service with a mean of 15.10 years. All participants actively controlled traffic for the past 

12 months. 

2.2 Test Facility and Equipment

We conducted the simulation in the FAA, WJHTC Research, Development, and Human Factors 

Laboratory (RDHFL). The simulation configuration consisted of the Distributed Environment 

for Simulation, Rapid Engineering, and Experimentation (DESIREE) and the Target Generator 

Facility (TGF). DESIREE emulated en route DSR functions and received input from the TGF to 

display radar targets. The DESIREE simulation support team connected the URET CCLD and 

TMA prototypes and emulated CPDLC functionality. The TGF maneuvers aircraft based upon 

simulation pilot commands and scripted flight plan data. Table 1 identifies the specific tools, the 

version, and the functions that DESIREE provided. 
6 

Table 1. Controller Tools 

Tool Version Primary Capabilities 

URET MITRE URET prototype, 

version 4.1 

● Automated conflict detection 

● Trial planning 

● Trajectory modeling 

● Routing changes 

● Electronic flight data management 

TMA NASA Center TRACON 

Automation System 

(CTAS), version 6.0.1 

● Estimated times of arrival to outer meter arc, meter fix, final 

approach fix, and runway threshold 

● Aircraft sequence and scheduled times of arrival 

● Time each aircraft must gain/lose to meet maximally efficient 

arrival rates 

CPDLC DESIREE emulation, 

(formerly Build IA) 

● Transfer of communications (TOC) 

● Altimeter setting (AS) 

● Initial contact (IC) 

● Menu text messages (MT) 

Each R-side workstation consisted of a high-resolution Sony 2K monitor, DSR R-side keyboard, 

and 3-button trackball. TMA and CPDLC were on the R-side Sony 2K monitor. Each D-side 

workstation consisted of a 21” flat-panel monitor, DSR D-side keyboard, and mouse. We 

deployed URET and the CPDLC windows on the D-side 21” flat-panel monitor. The voice 

communications system consisted of individual relay switchboxes, controller headsets with 

microphones, and push-to-talk handsets or foot pedals. Flight strip marking was optional in our 

simulation; however, we informed controllers that strips could be posted to their position when 

URET was not operational. Only one of the controller teams requested and marked flight 

progress strips. 

We set up the controller workstations in one experiment room. Engineering research 

psychologists and a software engineer operated the data collection equipment and monitored the 

simulation from an adjacent room. An SME observed over-the-shoulder of each controller team 

and provided ratings and comments on controller interaction with the tools. Three experienced 

simulation pilots supported each controller team from pilot workstations in a remote room of the 

same building. The simulation pilots communicated with controllers using proper ATC 

phraseology and procedures and maneuvered aircraft using simple keyboard commands. 

2.2.1 Airspace

We selected a generic high altitude en route sector as the airspace for this simulation (Figure 1). 

Appendix C describes the merits of using generic airspace in real-time HITL simulations. We 

decided upon a high altitude sector after visiting Indianapolis and Los Angeles ARTCCs and 
7 

consulting with the user groups for each of the tools. SMEs with specific knowledge of URET 

and TMA operation in the field advised the research team to use a high altitude sector. The 

rationale was that both tools are probably more effective in high altitude sectors where URET 

conflict detection is less complex and TMA delays are longer. Also, we discussed the generic 

high altitude sector with the CPDLC Program Office and National Air Traffic Controller 

Association (NATCA) representatives of the CPDLC user team who were satisfied that the 

sector would be adequate for CPDLC usage as well. 

Researchers and SMEs with the NAS Human Factors Group (ATO-P) designed this generic high 

altitude sector to be a realistic environment for controlling traffic and easy for ATCSs to learn 

(Guttman & Stein, 1997). The airspace consisted of easily remembered “fix” names and 

simplified operating procedures to facilitate learning. The generic sector was roughly 

rectangular in shape and extended for approximately 120 nm from north to south, approximately 

100 nm from east to west, and from Flight Level (FL) 240 and above in altitude. Arrival routes 

flowed in a southbound direction and departure routes flowed in a northbound direction. The 

sector contained several intersections that contributed to complexity and crossing restrictions for 

realism. The high altitude sector was adjacent to a generic low altitude sector. The low altitude 

sector served as transition airspace controlling traffic flow into the terminal region that had one 

major airport and three satellite airports. In this experiment, the sectors adjacent to the high 

altitude sector were automated by “ghost” controller functionality. 

Figure 1. Generic high altitude airspace. 

CHIGO

INDIN

PAWNE LINCO

SIOUX

BUTTE

DESMN BOSTN

DETRO

TOPKA

IND

KANCY

COLUM

LOUIV

WHEEL

J26

J79

J12

J13

J79

J79

J23 J23 J23

J30

J1

J13

J1

J13

J30

J12

J26

J22

J22

J21

J100

J100

J100

J15

J21

J12

J12

OHO

R07

FL240-999

120.07

R22

FL240-999

120.22

R02

FL240-999

120.02

A

B

R33

SFC-999

120.33

GEND2

N

J21

J25

J23

R05

FL240-999

120.05

IOW CIN

SPL

NEB

OHS

OHN

Sector 08

A FL230-999

B FL240-999

120.08

CHIGO

INDIN

PAWNE LINCO

SIOUX

BUTTE

DESMN BOSTN

DETRO

TOPKA

IND

KANCY

COLUM

LOUIV

WHEEL

J26

J79

J12

J13

J79

J79

J23 J23 J23

J30

J1

J13

J1

J13

J30

J12

J26

J22

J22

J21

J100

J100

J100

J15

J21

J12

J12

OHO

R07

FL240-999

120.07

R22

FL240-999

120.22

R02

FL240-999

120.02

A

B

R33

SFC-999

120.33

GEND2

N

J21

J25

J23

R05

FL240-999

120.05

IOW CIN

SPL

NEB

OHS

OHN

Sector 08

A FL230-999

B FL240-999

120.08
8 

2.2.2 Traffic Scenarios

The researchers selected traffic scenarios from previous studies in the RDHFL and modified 

them to meet the objectives of the present study. We developed eight similar test scenarios and 

18 practice scenarios. We designed the test and practice scenarios with similar traffic patterns 

and levels. All test scenarios were 45 minutes in duration. The practice scenarios were 45 to 60 

minutes in duration. 

SMEs designed the traffic scenarios with moderate traffic levels. The operational definition of 

moderate traffic for this study was traffic that was busy enough to require metering, but not so 

busy as to overwhelm controllers who were not experienced with all of the tools. We prepared 

one basic test scenario with 81 total aircraft (43 arrivals, 6 departures, and 32 overflights). We 

designed the other test scenarios based on the basic scenario with the same number of arrivals, 

departures, and overflights. However, we changed the aircraft entry times and assigned different 

callsigns to the aircraft in each of the other scenarios. This ensured that all test scenarios were 

similar in traffic, but not recognizable as the same scenario. 

When CPDLC was in use, 40% of the aircraft in the scenarios were equipped. The default 

setting for CPDLC was manual Transfer of Communications (TOC) mode that was operational 

with auto handoff mode in DSR. In manual TOC mode, CPDLC will generate a held TOC 

message that must be released by either the R-side or D-side controller for the system to uplink a 

new frequency to aircraft. We allowed the participants to decide for themselves which team 

member would release aircraft with held TOCs. In the CPDLC system, controllers can always 

override the manual TOC mode for an individual aircraft and allow the system to automatically 

uplink a new frequency when the aircraft is handed off. 

2.3 Experiments

2.3.1 Independent Variables

The experiment represented a 2-factor design with R-side and D-side controller positions as the 

first factor and eight different tool combinations as the second factor (Table 2). The ATCSs 

participated in eight scenarios, with each scenario representing a different tool combination. We 

counterbalanced the presentation of the tool combinations to experimentally control for practice 

effects. Although there were two teams of ATCSs participating simultaneously, each team 

independently controlled different traffic scenarios in the high altitude sector. 

Table 2. Experimental Design 

Positions R-side and D-side Controller Teams 

Tools None U T C UT UC TC UTC 

 None-No Tools (Baseline) U-URET T-TMA C-CPDLC 
9 

2.3.2 Dependent Variables

In our real-time HITL simulations, we collect a large set of standard system effectiveness 

measures for ATC simulation research that include safety, capacity, efficiency, and 

communications indicators (Buckley, DeBaryshe, Hitchner, & Kohn, 1983; Stein & Buckley, 

1992). We examined all the objective measures we collected to understand what controllers did 

in our simulation. For this report, we selected a few measures that were directly influenced by 

these tools. Table 3 presents these objectives measures. 

Table 3. Objective Measures 

Number of CPDLC TOC messages sent 

Number of controller ground-to-air voice communications 

Number of controller assigned airspeed, heading, and altitude changes 

Number of controller handoffs accepted and initiated 

Number of controller flight plan readouts 

Flight distance and flight time (per aircraft) within the sector 

Aircraft loss-of-separation within the sector 

In our real-time HITL simulations, we also collect a large set of subjective measures that include 

controller workload and SA measures, SME ratings and observations, and participant 

questionnaires. We examined all the subjective measures we collected to identify any potential 

human factors issues in this study. For this report, we selected a few measures that were directly 

influenced by these tools. Table 4 presents these subjective measures. 

Table 4. Subjective Measures 

Controller ratings of display clutter 

Controller workload ratings obtained from ATWIT 

Controller workload ratings obtained from NASA-TLX: Mental Demand 

Controller situational awareness ratings 

Observer ratings of controller overall prioritization 

Controller simulation realism ratings 

The ATCSs provided workload ratings using two different techniques. The first technique was 

the ATWIT, a real-time uni-dimensional workload rating method. ATWIT provides an 

unobtrusive and reliable means for collecting self-report workload ratings as the ATCS manages 
10 

air traffic (Stein, 1985, 1991). A personal computer and 10-button keypad collected and 

recorded participant responses. The participants indicated their current workload by pressing 

one of the keypad buttons labeled from 1 (low workload) to 10 (high workload). The system 

prompted participants for input every five minutes by emitting several beeps and lighting the 

buttons on the keypad. Participants had 20 seconds to respond by pressing one of the 10 buttons. 

If the ATCS did not respond within 20 seconds, ATWIT recorded a default symbol indicating 

there was no response. 

The second technique we used to measure workload was the NASA-TLX (Hart & Staveland, 

1987). The ATCSs completed this multi-dimensional workload rating method at the conclusion 

of each scenario. The six dimensions on which the NASA-TLX focuses are mental demand, 

physical demand, temporal demand, performance, effort, and frustration. We used a modified 

procedure that included all six workload subscales, but did not use the sorting technique that 

assigns subjective weights to the subscales. 

An SME unobtrusively observed each controller team and made over-the-shoulder ratings of 

ATCS performance during the scenarios. The SMEs used an observation form specially 

designed for ATC performance evaluation research (Sollenberger, Stein, & Gromelski, 1997; 

Vardaman & Stein, 1998). The observation form consists of 27 different rating scales organized 

into 6 major ATC performance categories. The SMEs received extensive training in the use of 

these rating scales. 

2.4 Training

The SMEs also acted as instructors and trained the participants to become proficient with the 

generic airspace and familiar with the URET, TMA, and CPDLC systems. Training on the 

airspace and tools consisted of an overview presentation and 18 hours of practice scenarios. 

Participants completed airspace and tool training over the course of four days. 

2.5 Procedure

Table 5 shows the daily schedule for each 2-week session. Four controllers participated in each 

testing session. Monday of the first week and Thursday of the second week were scheduled for 

travel. On the first Tuesday, the researchers briefed the participants on the project goals, 

airspace operations, and support tools followed by three 60-minute practice scenarios and a 

question and answer period at the end of the day. Wednesday, Thursday, and Friday of the first 

week consisted of five 60-minute practice scenarios each day. The following week, Monday, 

Tuesday, and Wednesday consisted of four test scenarios and a group discussion at the end of 

each day. Participants worked from 8:00 AM to 4:30 PM with a 90-minute lunch period and a 

30-minute break after each scenario. On the first day of the study, participants signed an 

Informed Consent Form (Appendix A) and completed a Background Questionnaire 

(Appendix B). The participants completed a Post-Scenario Questionnaire (Appendix D), and the 

SMEs completed Observer Rating Forms (Appendix E) after each test scenario. On the last day 

of the study, the participants completed an Exit Questionnaire (Appendix F). 
11 

Table 5. Daily Schedule 

Tues Wed, Thurs, Fri Mon, Tues, Wed 

Time Activity Time Activity Time Activity 

8:00 - 10:00 Project Briefing, 8:00 - 9:00 Practice Scenario 8:00 - 8:30 Warm-up Scenario 

 Overview, and 9:00 - 9:30 Break 8:30 - 9:00 Break 

 Initial Forms 9:30 - 10:30 Practice Scenario 9:00 - 9:45 Test Scenario 

10:00 - 10:30 Break 10:30 - 11:00 Break 9:45 - 10:15 Break 

10:30 - 11:30 Practice Scenario 11:00 - 12:00 Practice Scenario 10:15 - 11:00 Test Scenario 

11:30 - 1:00 Lunch 12:00 - 1:30 Lunch 11:00 - 11:30 Break 

1:00 - 2:00 Practice Scenario 1:30 - 2:30 Practice Scenario 11:30 - 12:15 Test Scenario 

2:00 - 2:30 Break 2:30 - 3:00 Break 12:15 - 1:45 Lunch 

2:30 - 3:30 Practice Scenario 3:00 - 4:00 Practice Scenario 1:45 - 2:30 Test Scenario 

3:30 - 4:00 Break 4:00 - 4:30 Group Discussion 2:30 - 3:00 Break 

4:00 - 4:30 Question & Answer 3:00 - 4:30 Group Discussion 

Table 6 shows the counterbalancing order for the experimental conditions. We presented eight 

scenarios for each R-side/D-side team of participants. The conditions were randomized 

differently for each group of participants. The scenarios were similar in traffic level, but we 

changed the aircraft entry times and assigned different callsigns to each aircraft in the scenarios. 

The participants completed four test scenarios each day after a preliminary warm-up scenario. 

Before each scenario, the researchers informed the participants which tool or tools would be 

operational. 

Table 6. Experimental Condition Counterbalancing 

 Presentation Order of Experimental Conditions 

Controllers 1 2 3 4 5 6 7 8 

S01, S02 TC U UC C UT UTC T None 

S03, S04 UT C U UTC UC None TC T 

S05, S06 UC TC C U T UT None UTC 

S07, S08 U UC None UT TC T UTC C 

S09, S10 T UTC TC UC None U C UT 

S11, S12 C None UT T UTC TC U UC 

 None-No Tools (Baseline) U-URET T-TMA C-CPDLC 
12 

3. Results

We present the results of the experiment in a series of graphs depicting the means of the eight 

tool conditions for all participants. For most of the measures, we present the means separately 

for the R-side and D-side; however, in some cases, we show R-side/D-side team means. The 

graphs also depict error bars for each mean representing +/- 1 standard error of the mean as an 

indicator of between-subject or between-team variability. We used the following formula for 

computing the standard error. 

Standard Error = Standard Deviation / Square Root (N), where N is the number of 

observations used to compute the mean, (i.e., 6 R-side participants, 6 D-side participants, 

or 6 teams) 

We used inferential statistics to analyze the data using two different approaches. In the first 

approach, we performed a 2-way analysis of variance (ANOVA) on each dependent measure 

with Position (R-side, D-side) as a between-subjects factor and Tools (None, U, T, C, UT, UC, 

TC, UTC) as a repeated measures factor. For the measures that produced team data, we 

performed a 1-way ANOVA on the Tools factor. This first approach allowed us to analyze the 

data in terms of differences between the tool conditions. A significant interaction between the 

factors would indicate that the trends were different for the two controller positions. 

In the second approach, we analyzed the R-side and D-side data separately in two different 3-

way ANOVAs with URET (Absent, Present), TMA (Absent, Present), and CPDLC (Absent, 

Present) as repeated measures factors. For the measures that produced team data, we performed 

only one 3-way ANOVA. This second approach allowed us to analyze the data in terms of the 

impact of each tool. A significant interaction between the tools would indicate a potential 

collocation issue. In each ANOVA for both approaches, the standard significance level was 

p < .05. When there were significant effects, we performed Tukey HSD post hoc comparisons to 

determine the means that were different using a group comparisons significance level of p < .05. 

3.1 Objective #1: Assess whether controllers have access to information when needed and 

whether they interpret available information correctly

We examined controller ratings of whether they were able to quickly access needed information 

or whether the collocation of tools made access to information difficult. We also conducted 

structured group debriefings with controllers to discuss any information access issues that 

occurred during simulation and why they may have occurred. Finally, our SMEs made ratings 

and comments about their observations on how controllers were using the tools during the 

simulation to further clarify any information access issues. 
13 

Figure 2 shows controller ratings for how often CPDLC caused display clutter on both the R-side 

and D-side displays for each of the four tool conditions with CPDLC operational. R-side and Dside controller ratings were different depending upon the tools in use, [Position x Tools 

interaction: F(3,30) = 4.36, p = .012]. For the R-side, controller ratings were rather low 

indicating that they did not think CPDLC caused clutter very often. For the D-side, controller 

ratings varied from low to moderate depending upon the tools displayed, [D-side, Tools simple 

main effect: F(3,15) = 4.03, p = .027]. When CPDLC was displayed alone and when CPDLC 

and TMA were displayed together, ratings of display clutter were very low. Ratings were much 

higher when CPDLC and URET were both displayed and when all three tools were displayed 

together (confirmed by Tukey HSD comparisons). 

Figure 2. Mean controller ratings for how often CPDLC caused display clutter provided by the 

R-side and D-side positions for each of the CPDLC tool conditions. 

1

2

3

4

5

6

7

8

9

10

Tools

How often did CPDLC cause display clutter?

R-Side D-Side D-Side

C UC TC UTC C UC TC UTC

Always

Never
14 

Figure 3 shows controller ratings for how often URET caused display clutter on the D-side 

display for each of the four tool conditions with URET operational. In this case, D-side 

controller ratings were rather low indicating that they did not think URET caused clutter on the 

D-side display whether used alone or in combination with TMA and CPDLC. The D-side 

controllers in our study were used to operating URET alone at their facility. This may have 

influenced their choice in identifying CPDLC as the source of display clutter with their ratings, 

not URET. 

Figure 3. Mean controller ratings for how often URET caused display clutter provided by the Dside position for each of the URET tool conditions. 

In our group discussions, controllers frequently commented that when URET and CPDLC were 

collocated on the D-side, there was display clutter. Figure 4 illustrates this issue and shows a 

typical configuration of URET and CPDLC on the D-side display. Controllers expand the URET 

Aircraft List to cover most of the display so that as much aircraft information can be seen as 

possible. The CPDLC Message Out and Menu Text windows are shown along with the D-side 

CRD. URET and CPDLC windows overlap each other because in this study there was no 

integration of the information displays for different tools. The URET Aircraft List can be resized 

and made smaller, but controllers do not often do this because important information would be 

truncated from the window and would not be visible on the display. 

Figure 5 illustrates what happens on the D-side display during typical use of URET and CPDLC. 

When a controller selects the URET Aircraft List as the active window, it becomes front and the 

CPDLC windows move to the back and become covered. In this configuration, updated 

information in the CPDLC Message Out window is not visible to controllers. 

1

2

3

4

5

6

7

8

9

10

Tools

How often did URET cause display clutter?

U UT UC UTC

D-Side

Always

Never
15 

Figure 4. Typical configuration of URET and CPDLC on the D-side display. 

Figure 5. URET aircraft list covering CPDLC windows on the D-side display. 

#LINE

5

D >>

FULL

TEXT CID MT ALT HDG SPD SNT WIL

PRINT

TEXT MID RTE TOC IC AS ROG NRR

SEND KEYS DELETE

120 COA662 011 HZ 120.01 HLD S

111 NWA553 015 HZ 120.06 HLD

114 ALO290 018 HZ 120.06 HLD

105 USA375 025 ALTIMETER 29.92 SNT

106 N347V 028 ALTIMETER 29.92 SNT T

EXIT

- DL MSG OUT §

#LINE

5

D >>

FULL

TEXT CID MT ALT HDG SPD SNT WIL

PRINT

TEXT MID RTE TOC IC AS ROG NRR

SEND KEYS DELETE

120 COA662 011 HZ 120.01 HLD S

111 NWA553 015 HZ 120.06 HLD

114 ALO290 018 HZ 120.06 HLD

105 USA375 025 ALTIMETER 29.92 SNT

106 N347V 028 ALTIMETER 29.92 SNT T

EXIT

- DL MSG OUT §

#LINE

5

P >>

VIEW SUP

KEYS

S

T

MIC – CHECK STUCK MICROPHONE

FOG2 – FOG 2 MI VISIBILITY

XCHI – CROSS CHICAGO @ FL240

XDAR – CROSS DARIO @ FL240

CALL – CALL COMPANY

SHOW SEND ALL

PRINT

TEXT

FULL

TEXT

- DL MENU TEXT §

EXIT

#LINE

5

P >>

VIEW SUP

KEYS

S

T

MIC – CHECK STUCK MICROPHONE

FOG2 – FOG 2 MI VISIBILITY

XCHI – CROSS CHICAGO @ FL240

XDAR – CROSS DARIO @ FL240

CALL – CALL COMPANY

SHOW SEND ALL

PRINT

TEXT

FULL

TEXT

- DL MENU TEXT §

EXIT

ACL 26 DEP 5 GPD PLANS WIND CRD-UA CRD-MCA CRD-MCA STATUS T 0004:56 8 0004:56 8 MSG OUT MSG OUT MENU HISTORY HISTORY

#LINE

5

D >>

FULL

TEXT CID MT ALT HDG SPD SNT WIL

PRINT

TEXT MID RTE TOC IC AS ROG NRR

SEND KEYS DELETE

120 COA662 011 HZ 120.01 HLD S

111 NWA553 015 HZ 120.06 HLD

114 ALO290 018 HZ 120.06 HLD

105 USA375 025 ALTIMETER 29.92 SNT

106 N347V 028 ALTIMETER 29.92 SNT T

EXIT

- DL MSG OUT §

#LINE

5

D >>

FULL

TEXT CID MT ALT HDG SPD SNT WIL

PRINT

TEXT MID RTE TOC IC AS ROG NRR

SEND KEYS DELETE

120 COA662 011 HZ 120.01 HLD S

111 NWA553 015 HZ 120.06 HLD

114 ALO290 018 HZ 120.06 HLD

105 USA375 025 ALTIMETER 29.92 SNT

106 N347V 028 ALTIMETER 29.92 SNT T

EXIT

- DL MSG OUT §

#LINE

5

P >>

VIEW SUP

KEYS

S

T

MIC – CHECK STUCK MICROPHONE

FOG2 – FOG 2 MI VISIBILITY

XCHI – CROSS CHICAGO @ FL240

XDAR – CROSS DARIO @ FL240

CALL – CALL COMPANY

SHOW SEND ALL

PRINT

TEXT

FULL

TEXT

- DL MENU TEXT §

EXIT

#LINE

5

P >>

VIEW SUP

KEYS

S

T

MIC – CHECK STUCK MICROPHONE

FOG2 – FOG 2 MI VISIBILITY

XCHI – CROSS CHICAGO @ FL240

XDAR – CROSS DARIO @ FL240

CALL – CALL COMPANY

SHOW SEND ALL

PRINT

TEXT

FULL

TEXT

- DL MENU TEXT §

EXIT

ACL 26 DEP 5 GPD PLANS WIND CRD-UA CRD-MCA CRD-MCA STATUS T 0004:56 8 0004:56 8 MSG OUT MSG OUT MENU HISTORY HISTORY
16 

In our group discussions, some controllers commented that they wanted TMA delay time 

information available on the D-side display. In the simulation, controllers positioned the TMA 

list in a location near the edge of the DSR where D-side controllers could see the information. 

The participants commented that the R-side should be able to position the TMA list wherever is 

most convenient for him/her and that D-side controllers should have access to this information 

on the D-side display. During the simulation, one controller used the URET Aircraft List Special 

Emphasis Area to sort arrival aircraft in the same order as the TMA list. This controller 

commented that having arrival aircraft sorted in this manner helped with monitoring their status. 

Also, it would be even better if TMA delay time information was available on the URET Aircraft 

List. Finally, controllers commented that they were concerned about adding to the clutter on the 

D-side display, but thought there should be a smart way to show TMA information without 

causing more clutter. Therefore, despite the already cluttered display, controllers thought it was 

important to have access to key information about all of the tools on the D-side display. 

Observer ratings and comments supported the idea that TMA delay time information was very 

important to both R-side and D-side controllers and influenced controller overall prioritization of 

tasks. Figure 6 shows mean observer ratings of controller overall prioritization effectiveness for 

controller teams for each of the eight tool conditions. The observers rated controller 

prioritization as slightly less effective when TMA was operational [TMA effect: F(1,5) = 12.00, 

p = .018]. The observers commented that controllers were paying a great deal of attention to the 

TMA List which made them fall behind on other tasks like accepting handoffs. 

Figure 6. Mean observer ratings of overall prioritization effectiveness for controller teams for 

each of the tool conditions. 

1

2

3

4

5

6

7

8

Tools

Observer Rating: Prioritization Effectiveness

Low None U T C UT UC TC UTC

High
17 

3.2 Objective #2: Assess R-side/D-side Roles and Responsibilities

For this report, we were interested in the number of CPDLC TOC messages sent by R-side and 

D-side controllers to describe the roles and responsibilities for CPDLC usage during the 

simulation. We did not create a procedure for which team member should send CPDLC TOCs to 

aircraft. Therefore, this measure indicated how controllers decided for themselves which team 

member would send CPDLC TOC messages when different tools were operational. 

Figure 7 shows the mean number of CPDLC TOC messages sent by the R-side and D-side 

positions for each of the four tool conditions with CPDLC operational. The R-side sent from 

90% to 95% of the CPDLC TOC messages in each of the tool combinations [Position effect: 

F(1,10) = 194.96, p < .001]. Therefore, the R-side retained the responsibility to issue the change 

of frequency to aircraft. 

Figure 7. Mean number of CPDLC TOC messages sent by R-side and D-side for each of the 

CPDLC tool conditions. 

In our group discussions, controllers commented that the R-side and D-side were frequently not 

aware of the CPDLC messages sent by each other and received by pilots. There was no auditory 

feedback while using CPDLC, therefore, controllers had to visually monitor the CPDLC 

Message Out window to know when their team member sent a message or verbally coordinate 

with each other when messages were sent. This may become a multi-tool issue when one 

controller sends a CPDLC TOC message early and, based on URET or TMA information, the 

other team member decides to take (or recommends) action on the aircraft. 

0

5

10

15

20

25

Tools

CPDLC Transfer of Communications (TOC)

C UC TC UTC C UC TC UTC

R-Side D-Side D-Side
18 

3.3 Objective #3: Assess R-side/D-side Workload and Situational Awareness

We examined controller ATWIT and NASA-TLX ratings as direct subjective measures of 

workload and number of ground-to-air voice transmissions and airspeed, heading, and altitude 

changes as indirect objective measures of workload. High workload ratings when multiple tools 

are operational would be a warning signal that indicates a collocation issue. Large increases in 

ground-to-air transmissions or airspeed, heading, and altitude changes when more than one tool 

is in use would indicate a collocation issue. We also examined controller SA ratings to identify 

any decrease in SA that would indicate a collocation issue when the tools are used together. 

Figure 8 shows mean ATWIT workload ratings provided by the R-side and D-side positions for 

each of the eight tool conditions. For the R-side, ATWIT ratings were moderate and there were 

few differences between the tool combinations. For the D-side, ATWIT ratings were low to 

moderate and tended to differ for different tool combinations. D-side ATWIT ratings were the 

lowest in the baseline condition without any tools; however, this was not statistically significant. 

We must emphasize that 5 of the 6 controller teams did not want to use flight strips during the 

simulation. Therefore, workload for this baseline condition may have been lower than actual 

field conditions where flight strips are actively used. Ratings tended to increase slightly in the 

single tool conditions and further increase in the two and three tool combinations. An additional 

analysis indicated that D-side controller ratings were the highest in conditions where URET was 

operational [D-side, URET effect: F(1,5) = 13.39, p = .015]. 

Figure 8. Mean ATWIT workload ratings provided by the R-side and D-side positions for each 

of the tool conditions. 

1

2

3

4

5

6

7

8

9

10

Tools

Air Traffic Workload Input Technique (ATWIT) Rating

None U T C UT UC TC UTC None U T C UT UC TC UTC

R-Side D-Side D-Side

Low

High
19 

Figure 9 shows mean NASA-TLX mental demand workload ratings provided by the R-side and 

D-side positions for each of the eight tool conditions. For the R-side, mental demand ratings 

were moderate and there were few differences between the tool combinations. For the D-side, 

mental demand ratings were low to moderate and tended to differ for different tool combinations. 

For both R-side and D-side, mental demand ratings were the lowest in the baseline condition 

without any tools, [Tools effect: F(7,70) = 2.54, p = .022]. However, the only statistically 

significant difference was between the no tools condition and the TMA-CPDLC combination 

(confirmed by Tukey HSD comparisons). Again, D-side workload ratings may have been higher 

in the No Tools condition if controllers had used flight strips as they do in actual field conditions. 

Ratings tended to increase slightly in the single tool conditions and greatly increase in the two 

and three tool combinations. An additional analysis indicated that D-side controller ratings 

were the highest in conditions where CPDLC was operational [D-side, CPDLC effect: F(1,5) = 

7.50, p = .041]. 

Figure 9. Mean NASA-TLX mental demand workload ratings provided by the R-side and D-side 

positions for each of the tool conditions. 

1

2

3

4

5

6

7

8

9

10

Tools

NASA-TLX: Mental Demand

None U T C UT UC TC UTC None U T C UT UC TC UTC

R-Side D-Side D-Side

Low

High
20 

Figure 10 shows the mean number of ground-to-air transmissions made by the R-side controller 

for each of the eight tool conditions. Controllers made 9.59% fewer ground-to-air transmissions 

when CPDLC was operational [CPDLC effect: F(1,5) = 165.40, p < .001]. This result was 

expected because CPDLC was designed to decrease ground-to-air voice transmissions. Fewer 

ground-to-air transmissions were expected when CPDLC was used alone and also when it was 

collocated with other tools. Therefore, this result did not indicate a collocation issue when 

CPDLC was used with other tools. 

Figure 10. Mean number of ground-to-air transmissions made by the R-side controller for each 

of the tool conditions. 

0

20

40

60

80

100

120

140

160

180

200

220

240

Tools

Number of Ground-to-Air Voice Transmissions

None U T C UT UC TC UTC
21 

Figure 11 shows the mean number of airspeed changes made by the R-side controller for each of 

the eight tool conditions. Controllers made 22.35% more airspeed changes when TMA was 

operational [TMA effect: F(1,5) = 14.45, p = .013]. This result was expected because controllers 

frequently used airspeed changes as a control technique to meet TMA metering times. More 

airspeed changes were expected when TMA was used alone and also when it was collocated with 

other tools. Therefore, this result did not indicate a collocation issue when TMA was used with 

other tools. 

Figure 11. Mean number of airspeed changes made by the R-side controller for each of the tool 

conditions. 

In addition, we examined the mean number of heading and altitude changes made by the R-side 

controller for each of the tool combinations. The results indicated that there were no differences 

in either heading or altitude changes between the tool combinations. Controllers made very few 

heading changes during the simulation because vectoring was not necessary to meet TMA 

metering times or miles-in-trail restrictions. Also, controllers used altitude changes to descend 

landing aircraft, but not to meet TMA metering times or miles-in-trail restrictions. 

0

5

10

15

20

25

30

35

40

45

Tools

Number of Airspeed Changes

None U T C UT UC TC UTC
22 

Figure 12 shows mean SA ratings made by the R-side and D-side positions for each of the eight 

tool conditions. In general, controller SA ratings were very high. For the R-side, ratings were 

the highest when URET alone was operational and the lowest in the four conditions where 

CPDLC was operational. For the D-side, ratings were the highest when URET alone was 

operational and the lowest in the condition where URET and CPDLC were combined; however, 

these trends were not statistically significant. 

Figure 12. Mean SA ratings made by the R-side and D-side positions for each of the tool 

conditions. 

1

2

3

4

5

6

7

8

9

10

Tools

Overall Situational Awareness Rating

None U T C UT UC TC UTC None U T C UT UC TC UTC

R-Side D-Side D-Side

Low

High
23 

3.4 Objective #4: Access R-side/D-side Teamwork and Communications

We examined the number of handoffs accepted and initiated by R-side and D-side controllers to 

describe controller teamwork during the simulation. These measures indicated how controllers 

decided which team member would accept and initiate handoffs when different tools were 

operational. We used these measures to determine if there were any changes in the way team 

members interacted with each other with additional tools. Also, we examined the number of 

flight plan readouts to assess controller needs for flight plan data when different tools were 

operational. We will examine controller communications between R-side and D-side team 

members in more detail in a future report. 

Figure 13 shows the mean number of handoffs accepted by the R-side and D-side positions for 

each of the eight tool conditions. The R-side accepted more than twice as many handoffs as the 

D-side [Position effect: F(1,10) = 30.31, p < .001]. However, there were no differences in the 

number of handoffs accepted with different tool combinations. Therefore, the R-side retained 

the majority of the handoff duties. 

Figure 13. Mean number of handoffs accepted by the R-side and D-side positions for each of the 

tool conditions. 

0

10

20

30

40

50

60

70

80

90

100

110

120

Tools

Number of Handoffs Accepted

None U T C UT UC TC UTC None U T C UT UC TC UTC

R-Side D-Side D-Side
24 

Figure 14 shows the mean number of handoffs initiated by the R-side and D-side positions for 

each of the eight tool conditions. In general, the D-side initiated the majority of the handoffs; 

however, this trend was not statistically significant. Also, there were no differences in the 

number of handoffs initiated with different tool combinations. 

Figure 14. Mean number of handoffs initiated by the R-side and D-side positions for each of the 

eight tool conditions. 

0

5

10

15

20

25

30

35

40

45

50

Tools

Number of Handoffs Initiated

None U T C UT UC TC UTC None U T C UT UC TC UTC

R-Side D-Side D-Side
25 

Figure 15 shows the mean number of flight plan readouts by the R-side and D-side positions for 

each of the eight tool conditions. R-side and D-side number of flight plan readouts were 

different depending upon the tools in use, [Position x Tools interaction: F(7,70) = 4.61, 

p < .001]. The R-side controller performed some flight plan readouts, but there were no 

differences between the tool combinations. The D-side controller performed a different number 

of flight plan readouts depending upon the tool combination, [D-side, Tools simple main effect: 

F(7,35) = 7.64, p < .001]. The D-side controller made the most flight plan readouts in the No 

Tools condition and the least number of flight plan readouts in the four conditions where URET 

was operational (confirmed by Tukey HSD comparisons). An additional analysis indicated that 

D-side controllers made the fewest flight plan readouts when URET was operational [D-side, 

URET effect: F(1,5) = 12.82, p = .016]. This result was expected because D-side controllers did 

not need to perform flight plan readouts when URET was operational because URET provided 

the flight data for aircraft. 

Figure 15. Mean number of flight plan readouts by the R-side and D-side positions for each of 

the eight tool conditions. 

0

5

10

15

20

25

30

35

40

45

Tools

Number of Flight Plan Readouts

None U T C UT UC TC UTC None U T C UT UC TC UTC

R-Side D-Side D-Side
26 

3.5 Other Measures

3.5.1 System Effectiveness Measures

Figure 16 shows the mean flight distance per aircraft for controller teams for each of the eight 

tool conditions. In general, flight distance per aircraft was the longest in the No Tools condition 

and at least slightly shorter in each of the conditions with tools; however, these trends were not 

statistically significant. 

Figure 16. Mean flight distance per aircraft for controller teams for each of the tool conditions. 

In addition, we examined the mean flight time per aircraft for controller teams for each of the 

eight tool conditions. The pattern of data for mean flight time was similar to mean flight 

distance per aircraft. Flight time per aircraft was the longest in the No Tools condition and at 

least slightly shorter in each of the conditions with tools; however, these trends were not 

statistically significant. 

There were two cases where aircraft lost separation during the entire experiment. In the first 

loss-of-separation, CPDLC was the only tool in operation. The controllers were establishing an 

arrival sequence with two aircraft when they lost separation. The controllers were aware that the 

distance was going to be close with the aircraft, but they thought they had enough separation. In 

the second loss-of-separation, TMA and CPDLC were operational. The D-side pointed out the 

aircraft conflict to the R-side. The R-side attempted to separate the aircraft with heading changes 

to one of the aircraft, but didn’t turn the aircraft sharp enough to ensure separation. In both cases 

where there was loss-of-separation, the closest-point-of-approach was over 4 nm. 

54

56

58

60

62

64

66

68

70

Tools

Flight Distance (nm) per Aircraft

None U T C UT UC TC UTC
27 

3.5.2 Exit Questionnaire Ratings

Table 7 shows the controller ratings of simulation realism. In general, controllers rated the 

simulation realism as moderate. Controller mean ratings ranged from 4.20 to 7.67 on a 10-point 

scale depending upon the specific aspect of the simulation. The lowest realism rating was for the 

TMA implementation. Controllers who were experienced with TMA commented that the delay 

times generated in our simulation were not realistic. In field operations, typical TMA delay 

times are positive and are often resolved by slowing aircraft. In our simulation, TMA delay 

times were frequently small and sometimes negative, which made controllers increase aircraft 

airspeeds. This discrepancy in TMA operation may have affected some measures like aircraft 

flight distance, flight time, and airspeed changes. However, controllers commented that it did 

not have a major impact on identifying tool collocation issues. 

Lastly, controllers rated the ATWIT workload rating technique as low in interference, 

(Mean = 1.60, SD = 0.84), where 1 = no interference at all and 10 = a great deal of interference. 

Table 7. Controller Ratings of Simulation Realism 

Question Extremely 

Unrealistic 123456789V Extremely 

Realistic 

Rate the realism of the overall simulation experience 

compared to actual ATC operations. Mean = 6.20 SD = 1.75 Range (4 – 9) 

Rate the realism of the simulation URET implementation 

compared to actual field operation. Mean = 5.60 SD = 2.30 Range (2 – 8) 

Rate the realism of the simulation TMA implementation 

compared to actual field operation. Mean = 4.20 SD = 2.39 Range (1 – 7) 

Rate the realism of the simulation CPDLC emulation 

compared to actual field operation. Mean = 7.67 SD = 2.08 Range (6 – 10)

Rate the realism of the simulation DSR hardware 

compared to actual DSR equipment. Mean = 7.00 SD = 2.31 Range (3 – 10)

Rate the realism of the simulation DSR software 

compared to actual DSR functionality. Mean = 6.80 SD = 1.69 Range (4 – 9) 

Rate the realism of the simulation generic airspace 

compared to actual NAS airspace. Mean = 7.60 SD = 1.78 Range (5 – 10)

Rate the realism of the simulation traffic scenarios 

compared to actual NAS traffic. Mean = 7.20 SD = 1.32 Range (5 – 9) 
28 

4. Discussion

The RDHFL ATC simulator collects a very large set of raw data about what controllers do in 

simulation. This report presents a quick look at data we could readily access and quickly 

summarize to provide timely answers to important research issues concerning the collocation of 

URET, TMA, and CPDLC. 

The most important collocation issue identified in this experiment was that controllers had 

difficulty accessing important information on the D-side display when URET and CPDLC were 

both operational (i.e., display clutter). Controller ratings indicated the cause of this display 

clutter problem was CPDLC displays on the D-side monitor. They did not identify URET 

displays as causing display clutter. All the D-side controllers were experienced URET users and 

none had worked with CPDLC prior to the simulation; therefore, it is not surprising that these 

controllers would attribute the display clutter to CPDLC and not URET. Also, controller ratings 

indicated that neither URET alone nor CPDLC alone caused D-side display clutter. Finally, Rside controller ratings indicated that CPDLC caused very little display clutter on the DSR 

display. 

It is important to note the controllers identified this display clutter problem using the D-side 

CPDLC CHI we developed in the RDHFL for use in this simulation study. We designed the Dside CHI to be consistent with a “stovepipe” independent deployment of the tools with simple 

features to help controllers manage the multiple windows associated with each tool. Members of 

the CPDLC users group helped us develop the D-side CHI. We then asked the ATDET to 

review the interface for use in this study. However, this specific D-side CHI was not intended to 

be the interface that will be deployed to ARTCCs in the future. 

Another collocation issue identified in this experiment was that D-side controllers had to access 

TMA delay time information on the R-side display. In the simulation, controllers positioned the 

TMA list in a location near the edge of the DSR where D-side controllers could see the 

information. Both controllers spent a great deal of time viewing the TMA list. Controllers 

thought it was important to have TMA information available on the D-side display where it 

could be more easily accessed by D-side controllers. One controller used the URET Aircraft List 

Special Emphasis Area to sort arrival aircraft in the same order as the TMA list. However, 

controllers were concerned that simply showing the TMA list on the D-side might add to the Dside display clutter. 

Good human factors design principles prescribe that users must have immediate access to 

important information and that critical information should never be covered. A “stovepipe” 

independent deployment of these tools will result in impaired access to timely information. The 

results of this study indicated that better human factors efforts should be made towards 

integrating the information from URET, TMA, and CPDLC. Even if these systems cannot be 

entirely integrated, we should explore integrating the displays on the D-side monitor prior to 

deployment. 

The number of CPDLC TOC messages indicated that R-side controllers sent most of the TOCs. 

There were no mandatory procedures for using CPDLC by the R-side and D-side in our 

simulation. We allowed each controller team to practice together and decide for themselves who 
29 

would send the CPDLC TOCs to aircraft. Although the results indicated that D-side controllers 

did not use the TOC service very often, controllers still expressed concerns about not knowing 

what their team member was doing with CPDLC. Unlike voice communications, there were no 

audible cues with CPDLC to help controllers maintain situational awareness of their team 

member’s actions. Controllers had to visually monitor the CPDLC Message Out window to 

know when their team member sent a TOC message. If the CPDLC display was covered by 

URET, the D-side controller could miss a sent message. In addition, CPDLC messages were 

visible in the Message Out window for only a few seconds after a pilot “Wilcox” was received. 

Therefore, controllers had to be very vigilant. An even greater concern for safety exists if 

CPDLC is used to issue control instructions (e.g., altitude, heading or airspeed future services). 

D-side controllers could recommend actions that create aircraft conflicts when they are not aware 

of the CPDLC messages sent. More research needs to be conducted examining the potential 

risks, roles, and responsibilities for CPDLC usage, especially when other tools are being used. 

Controller workload ratings using ATWIT and NASA-TLX indicated that D-side workload and 

mental demand tended to increase when two and three tools were operational. However, D-side 

workload ratings were only moderate and never reached an excessively high level. We designed 

the scenarios with moderate traffic levels so that controllers would not be overwhelmed with 

traffic and stop using the tools. We thought moderate traffic levels would allow controllers to 

use the tools more often and better identify any collocation issues. However, the results of this 

experiment may have been different with higher traffic levels and greater workload demands on 

controllers. 

D-side controller workload was rather low in the baseline condition without any tools. Workload 

for this baseline condition may have been lower than actual field conditions where flight strips 

are actively used because 5 of the 6 controller teams did not want flight strips posted at their 

sector. All of our D-side controllers came from URET facilities that do not require flight strips. 

The baseline condition was intended to be an experimental comparison for the conditions with 

tools. Without flight strips, controllers used flight plan readout to obtain aircraft routing 

information. Finally, R-side ATWIT and NASA-TLX ratings indicated that controller workload 

and mental demand were only moderate and did not change with tool use. 

We used the number of ground-to-air voice transmissions and airspeed, heading, and altitude 

changes as additional indicators of controller workload. As expected, controllers made fewer 

ground-to-air voice transmissions when CPDLC was operational. Also, as expected, controllers 

made more airspeed changes when TMA was operational because controllers frequently used 

airspeed changes to meet metering times. However, there were no differences in ground-to-air 

transmissions or airspeed, heading, and altitude changes when tools were used in combination. 

Therefore, the changes in these measures were due to each tool’s expected effect and not due to 

collocation. 

Finally, controllers rated their SA as very high and did not vary with different tool combinations. 

Although sometimes self-ratings can potentially misrepresent true SA, there was no indication 

that this actually occurred during the simulation. With the exception of two cases where aircraft 

separation was lost, controllers maintained safety throughout the scenarios. In both cases, 

controllers were aware of the loss of aircraft separation. For the present study, we used self-
30 

ratings of SA as a technique to elicit controller concerns and identify collocation issues for group 

discussion. 

5. Conclusion

The purpose of this simulation study was to identify collocation issues when URET, TMA, and 

CPDLC were implemented together at the same sector. In a high fidelity HITL simulation, the 

main concern was D-side display clutter when URET and CPDLC were being used together. 

Another concern was the need for TMA information on the D-side display without adding to the 

display clutter. In addition to identifying these important display issues, controllers provided 

some ideas that could resolve these problems. From a human factors perspective, URET, TMA, 

and CPDLC should be integrated on the D-side display. This solution avoids the problem of 

multiple windows or displays that were a consequence of the “stovepipe” implementation of 

these tools in our simulation. With URET, TMA, and CPDLC information integrated, 

controllers should have easier access to information when needed without having to monitor and 

manage multiple displays. 

Controllers also provided some non-integration ideas including a larger D-side display and an 

improved D-side CHI that would make it easier to manage multiple information displays. 

Although there would appear to be advantages to all of these ideas, there may also be pitfalls that 

can only be assessed in an HITL simulation. Therefore, we recommend that more research be 

conducted to investigate URET, TMA, and CPDLC collocation issues and potential solutions. 

Future simulations should examine the best presentation of the information, specific procedures 

for R-side and D-side tool use, and higher traffic levels for scenarios. 

In this report, we presented the first of three experiments that examined human factors issues in 

the collocation of URET, TMA, and CPDLC. In subsequent reports, we will present the two 

additional experiments and address teamwork and communications issues with the collocation of 

these tools. In addition, we will examine human factors issues related to using the three tools 

with an R-side only staffing configuration. Finally, we will also examine eye tracking data from 

the R-side position to explore the information controller’s access when these tools are in use. 
31 

References 

Buckley, E. P., DeBaryshe, B. D., Hitchner, N., & Kohn, P. (1983). Methods and measurements 

in real-time air traffic control system simulation (DOT/FAA/CT-83/26). Atlantic City 

International Airport, NJ: DOT/FAA Technical Center. 

Cardosi, K. M., & Murphy, E. D. (1995). Human factors in the design and evaluation of air 

traffic control systems (Technical Report No. DOT/FAA/RD-95/3). Washington, DC: 

Federal Aviation Administration Office of Aviation Research. 

Della Rocco, P. S., Punjabi, G., Friedman-Berg, F., Kopardekar, P., & Hah, S. (in press). 

Collocation of User Request Evaluation Tool, Traffic Management Advisor, and 

Controller-Pilot Data Link Communications: An initial human factors evaluation 

(DOT/FAA/CT-TN04/06). Atlantic City International Airport, NJ: DOT/FAA, William J. 

Hughes Technical Center. 

Desenti, C. T., Gross, A. E., & Toma, N. E. (2000). Analysis of Atlanta ARTCC operations for 

ATL arrival flows (MTR00W30). McLean, VA: The MITRE Corporation. 

Federal Aviation Administration. (2002). Operational Evolution Plan. Retrieved October 1, 

2002, from http://faa.gov/programs/oep/History.htm 

Guttman, J. A., & Stein, E. S. (1997). En route generic airspace evaluation (DOT/FAA/CTTN97/7). Atlantic City International Airport, NJ: DOT/FAA William J. Hughes Technical 

Center. 

Guttman, J. A., Stein, E. S., & Gromelski, S. (1995). The influence of generic airspace on air 

traffic controller performance (DOT/FAA/CT-TN95/38). Atlantic City International 

Airport, NJ: DOT/FAA Technical Center. 

Hart, S. G., & Staveland, L. E. (1987). Development of NASA-TLX (Task Load Index): Results 

of empirical and theoretical research. In P. A. Hancock and N. Meshkati (Eds.), Human 

Mental Workload. Amsterdam: North-Holland. 

Kerns, K. (2001). User Request Evaluation Tool (URET)/Controller-Pilot Data Link 

Communications (CPDLC) collocation: Development of an operational concept, 

identification of potential issues, and recommended solutions (MTR01W0000022). 

McLean VA: The MITRE Corporation. 

Manning, C. (2000). Measuring Air Traffic Controller performance in a high-fidelity simulation

(DOT/FAA/AM-00/2). Oklahoma City: DOT/FAA Office of Aviation Medicine. 

Sollenberger, R. L., Stein, E. S., & Gromelski, S. (1997). The development and evaluation of a 

behaviorally-based rating form for assessing air traffic controller performance 

(DOT/FAA/CT-TN96/16). Atlantic City International Airport, NJ: DOT/FAA, William J. 

Hughes Technical Center. 
32 

Stein, E. S. (1985). Air traffic controller workload: an examination of workload probe

(DOT/FAA/CT-TN84/24). Atlantic City International Airport, NJ: DOT/FAA Technical 

Center. 

Stein, E. S. (1991). Air traffic controller memory -- a field survey (DOT/FAA/CT-TN90/60). 

Atlantic City International Airport, NJ: DOT/FAA, Technical Center. 

Stein, E. S., & Buckley, E. P. (1992). Human factors at the FAA Technical Center: Bibliography 

1958-1994 (DOT/FAA/CT-TN94/50). Atlantic City International Airport, NJ: DOT/FAA, 

Technical Center. 

Vardaman, J. J., & Stein, E. S. (1998). The development and evaluation of a behaviorally based 

rating form for the assessment of en route air traffic controller performance

(DOT/FAA/CT-TN98/5). Atlantic City International Airport, NJ: DOT/FAA, William J. 

Hughes Technical Center. 
33 

Acronyms 

ANOVA Analysis of Variance 

ARTCC 

AS 

Air Route Traffic Control Center 

Altimeter Setting 

ATC Air Traffic Control 

ATCS Air Traffic Control Specialist 

ATDET Air Traffic Design Evolution Team 

ATWIT Air Traffic Workload Input Technique 

CCLD Core Capability Limited Deployment 

CHI Computer Human Interface 

CPDLC Controller-Pilot Data Link Communications 

CRD 

CTAS 

Computer Readout Display 

Center TRACON Automation System 

D-side 

DESIREE 

Data (or Radar Associate) Controller Position 

Distributed Environment for Simulation, Rapid Engineering, and Experimentation 

DSR Display System Replacement 

FAA Federal Aviation Administration 

FL Flight Level 

HITL Human-In-The-Loop 

IC 

MT 

NAS 

Initial Contact 

Menu Text 

National Airspace System 

NASA National Aeronautics and Space Administration 

NATCA National Air Traffic Controller Association 

R-side 

RDHFL 

Radar Controller Position 

Research Development and Human Factors Laboratory 

SA Situational Awareness 

SME Subject Matter Expert 

TGF Target Generation Facility 

TLX Taskload Index 

TMA Traffic Management Advisor 

TOC Transfer of Communications 

URET User Request Evaluation Tool 

WJHTC William J. Hughes Technical Center 
34 
A-1 

APPENDIX A 

Informed Consent Form 

I, ______________________________, understand that this study, entitled "Human-in-the-Loop 

Evaluation of the Collocation of User Request Evaluation Tool (URET), Traffic Management 

Advisor (TMA), and Controller-Pilot Data Link Communications (CPDLC)” is sponsored by the 

Federal Aviation Administration and is being directed by Dr. Randy Sollenberger. 

Nature and Purpose:

I have been recruited to volunteer as a participant in this project. The purpose of the study is to 

identify the human factors issues of collocating URET, TMA, and CPDLC in a high-fidelity, 

controller-in-the-loop simulation. The researchers will use the results of this study to provide 

guidance for deploying the three support tools in the field. 

Experimental Procedures:

Participants will arrive at the simulation laboratory in groups of four controllers for each 

two-week session. For most of the simulation, controllers will work in R-side/D-side teams 

without changing positions. However, all participants will work two scenarios as an R-side 

controller without D-side assistance. The first week of the simulation will consist of a project 

briefing, airspace familiarization, support tool training, and practice scenarios. Participants will 

stay over the weekend. The second week of the simulation will consist of twelve 45-minute test 

scenarios and an exit debriefing. Participants will work from 8:00 AM to 4:30 PM each day with 

a rest break after each scenario and a lunch break.

Participants will control traffic while using different combinations of support tools. After each 

scenario, the controllers will complete questionnaires to evaluate the impact of the tools on 

controller performance, workload, and situational awareness. In addition, subject-matter experts 

will make over-the-shoulder observations during the simulation to further assess aspects of the 

impact of the support tools. Finally, an automated data collection system will record system 

operations and generate a set of standard ATC simulation measures, which include safety, 

capacity, efficiency, and communications measures. The simulation will be audio-video 

recorded in case researchers need to re-examine any important simulation events. 

Discomfort and Risks:

I understand that I will not be exposed to any foreseeable risks or intrusive measurement 

techniques. 

Confidentiality:

My participation is strictly confidential, and no individual names or identities will be recorded or 

released in any reports. 

Benefits:

I understand that the only benefit to me is that I will be able to provide the researchers with 

valuable feedback and insight into the impact of using different support tools. My data will help 

the FAA to identify the human factors issues of collocating URET, TMA, and CPDLC. 
A-2 

Participant Responsibilities:

I am aware that to participate in this study I must be a certified professional controller who is 

qualified at an air traffic control facility and holds a current medical certificate. I will control 

traffic and answer any questions asked during the study to the best of my abilities. I will not 

discuss the content of the experiment with anyone until the study is completed. 

Participant's Assurances:

I understand that my participation in this study is completely voluntary and I have the freedom to 

withdraw at any time without penalty. I also understand that the researchers in this study may 

terminate my participation if they feel this to be in my best interest. I understand that if new 

findings develop during the course of this research that may relate to my decision to continue 

participation, I will be informed. 

I have not given up any of my legal rights or released any individual or institution from liability 

for negligence. 

Dr. Sollenberger has adequately answered all the questions I have asked about this study, my 

participation, and the procedures involved. I understand that Dr. Sollenberger or another 

member of the research team will be available to answer any questions concerning procedures 

throughout this study. 

If I have questions about this study or need to report any adverse effects from the research 

procedures, I will contact Dr. Sollenberger at (609) 485-7169. 

Compensation and Injury:

I agree to immediately report any injury or suspected adverse effect to Dr. Randy Sollenberger at 

(609) 485-7169. Local clinics and hospitals will provide any treatment, if necessary. I agree to 

provide, if requested, copies of all insurance and medical records arising from any such care for 

injuries/medical problems. 

Signature Lines:

I have read this informed consent form. I understand its contents, and I freely consent to 

participate in this study under the conditions described. I understand that, if I want to, I may 

have a copy of this form. 

Research Participant:________________________________________ Date:__________ 

Investigator:_______________________________________________ Date:__________ 

Witness:__________________________________________________ Date:__________ 
B-1 

APPENDIX B 

Background Questionnaire 

Instructions: 

This questionnaire is designed to obtain information about your background and 

experience as an air traffic control specialist (ATCS). The information will be used to 

describe the participants in this study as a group. You will not be identified by name. 

Indicate your response by filling in the circle with a pen or pencil (or mark with an X ), 

writing on the blank line, or circling the percentage number where appropriate. Some rating 

scales have labels at either end and 10 numbered circles that represent different levels of 

response. Fill in one circle with a pen or pencil (or mark with an X) to indicate your level of 

response. 

Demographic Information and Experience 

1. What is your gender?  Male  Female

2. Will you be wearing corrective lenses during this 

experiment?  Yes  No

3. What is your age? _____ years _____ months

4. How long have you worked as an ATCS (include both FAA 

and military experience)? _____ years _____ months

5. How long have you worked as an ATCS for the FAA? _____ years _____ months

6. How long have you been a Certified Professional Controller

(or Full Performance Level Controller)? _____ years _____ months

7. How long have you actively controlled traffic in the en route 

environment? _____ years _____ months

8. How many of the past 12 months have you actively controlled 

traffic? _____ months
B-2 

Sector Characteristics 

9. Rate the complexity of the sector that you 

work most frequently. 

Low 

Complexity 123456789 High Complexity

10. Provide the approximate dimensions of your 

most frequently worked sector. 

_____ nautical miles 

wide

_____ nautical miles 

long

11. Does your most frequently worked sector

contain special use or restricted airspace?  Yes  No

12. Describe the traffic type of your most frequently worked sector by assigning a 

percentage to the following categories:

(A). Transitional Traffic to/from a Major Airport 0 10 20 30 40 50 60 70 80 90 100 

(B). High Altitude En Route (FL240 and above) 0 10 20 30 40 50 60 70 80 90 100 

(C). Low Altitude En Route (FL230 and below) 0 10 20 30 40 50 60 70 80 90 100 

13. Describe the traffic mix of your most frequently worked sector by assigning a 

percentage to the following categories (percentages must sum to 100%):

(A). Air Carriers/Corporate Jets 0 10 20 30 40 50 60 70 80 90 100 

(B). Air Taxi/Commuters 0 10 20 30 40 50 60 70 80 90 100 

(C). Cargo 0 10 20 30 40 50 60 70 80 90 100 

(D). General Aviation 0 10 20 30 40 50 60 70 80 90 100 

(E). Military 0 10 20 30 40 50 60 70 80 90 100 

(F). Other, specifiy ___________________ 0 10 20 30 40 50 60 70 80 90 100 

Sum Total 100% 

14. Rate how often you use paper flight strips. Never 123456789 Always 

General Ratings 

15. Rate your current skill as an ATCS. 

Not 

Skilled 123456789 Extremely Skilled 

16. Rate your current level of stress. 

Not 

Stressed 123456789 Extremely Stressed 

17. Rate your level of motivation to participate in 

this study. 

Not 

Motivated 123456789 Extremely Motivated 
C-1 

APPENDIX C 

Generic Airspace 

Guttman & Stein (1997) performed a study that compared ATCS performance in generic and 

home airspaces. The generic airspaces were designed to be comparable to the home airspaces on 

a number of factors, including number of airways, route length, sector size, traffic mix, and the 

number and altitudes of restricted areas. However, the generic airspace, as compared to the 

home sector, had a different direction of traffic flow, different sector boundaries and, different 

Letters-of-Agreement (LOAs). These differences were purposely introduced to induce learning 

on the part of the ATCS. In this study, the researchers designed traffic scenarios to realistically 

model the traffic of a moderately busy sector. They reported high correlations between a number 

of different ATC performance measurements taken in the simulations performed in the ATCSs 

home airspace and the simulations performed in the generic airspace. The performance 

measurements with high correlations between home and generic airspace included controller 

self-ratings, controller workload ratings, and system effectiveness variables (e.g. number and 

duration of push-to-talk communications, average transmission time, flight time, and distance 

flown). They also determined that the majority of controllers considered the simulations using 

generic airspace to be very realistic and that the generic sector itself was very representative of a 

typical sector. They concluded that airspace sectors used for simulations need to be easy to learn 

and have properties similar to that of real sectors. Earlier, Guttman, Stein, & Gromelski (1995) 

performed a similar study evaluating the use of a generic terminal radar approach control 

(TRACON) with comparable results. 

Manning (2000) also evaluated the use of generic airspace in high-fidelity simulations. She 

found that a number of factors were important to consider when: a) designing traffic scenarios 

for real-time HITL simulations, b) training ATCSs on generic airspace, and c) training observers 

to make performance ratings during simulations. Manning stated, “The likelihood of accurately 

measuring ATCS performance increases with the extent to which one is able to place ATCSs in a 

standardized and realistic environment in which they must control traffic, and that affords 

reliable measurement of their performance.” First, Manning pointed out that scenarios should 

have moderately busy or busy traffic so ATCSs will be required to actively control traffic. 

Second, she felt that ATCSs should be trained to a certain level of proficiency on the airspace 

before testing begins. This training would ensure that all ATCSs are equally familiar with the 

generic airspace. In her study, ATCSs were required, after training, to take an airspace 

familiarization test and receive a score of 70% on both recall and recognition items before they 

were able to participate in the testing phase of the experiment. Finally, she concluded that 

simulation observers should receive extensive training with the rating scale to ensure a high level 

of inter-rater reliability. 

At the US-European Action Plan 5 Practitioners Workshop hosted by the FAA WJHTC in 2002, 

researchers involved in conducting simulations formally discussed the advantages and 

disadvantages of using generic airspace. They also discussed how scenarios should be developed 

for use in experiments. A best practices paper from this workshop is currently in preparation that 

discusses the advantages and disadvantages of using generic airspace (see Table C-1). After 

evaluating the pros and cons of using generic airspace, we decided that because this study will 
C-2 

examine systems which will be deployed across the NAS, a generic airspace would best allow us 

to generalize results to a variety of different airspaces (Willems, 2002). 

Table C-1. Advantages and Disadvantages of Generic Airspace 

Advantages Disadvantages 

● Generic airspace is easy to learn 

● Controllers start at a level playing field 

● Using generic airspace enables researchers to 

create their own standard operating procedures 

● A greater number of controllers are available to be

recruited for experiments 

● Results from experiments using generic airspace 

can be generalized to the NAS 

● All participants must be trained on the generic 

airspace 

● Researchers need to ensure that generic airspace 

has features that are consistent with actual 

airspace 

● Developing generic airspace can be difficult and 

costly 

● When using generic airspace, it may be difficult to

generalize the results to a specific airspace 

By using generic airspace for the real-time HITL simulations, we will be able to capture real 

ATCS performance while controlling for individual differences related to an ATCS -- length of 

experience and home sector complexity (Guttman & Stein, 1997; Manning, 2000). As Manning 

points out, testing ATCSs in their home airspaces is not a viable option. This is due to inherent 

differences in traffic density and airspace layout that would not allow us to compare ATCS 

performance across different levels of difficulty. To the extent that ATCS performance in 

generic airspace is comparable to ATCS performance in real airspace, researchers can 

extrapolate results across different airspaces. This greatly reduces the costs of performing ATC 

studies while increasing realism by using real-time HITL simulations. 
D-1 

APPENDIX D 

Post-Scenario Questionnaire 

R-side Version

Instructions: 

Please answer the following questions based upon your experience in the scenario just 

completed. The rating scales have labels at either end and 10 numbered circles that represent 

different levels of response. Fill in one circle with a pen or pencil (or mark with an X) to

indicate your level of response OR mark  N/A if the question deals with something you did 

not try at all during the simulation. 

Overall Performance, Workload, Situational Awareness, and Simulation Ratings 

1. Rate your overall level of ATC performance

during this scenario. 

Extremely 

Poor 123456789 Extremely Good 

2. Rate your workload due to communications 

with pilots during this scenario. 

Extremely 

Low 123456789 Extremely High 

3. How much coordination with your D-side 

team member occurred during this scenario? 

None At 

All 123456789 A Great Deal 

4. Rate your workload due to coordination with 

your D-side team member during this 

scenario. 

Extremely 

Low 123456789 Extremely High 

5. Rate your level of situational awareness

during this scenario. 

Extremely 

Poor 123456789 Extremely Good 

6. Rate the performance of the simulation pilots 

in terms of their responding to control 

instructions and providing readbacks. 

Extremely 

Poor 123456789 Extremely Good 

7. Rate the difficulty of this scenario. Extremely 

Difficult 123456789 Extremely Easy 
D-2 

URET 

8. Overall, how easy was it to use URET during 

this scenario? 

Extremely 

Difficult 

123456789

OR  N/A 

Extremely 

Easy 

9. How easy was it to access information from 

the URET Aircraft List when needed? 

Extremely 

Difficult 

123456789

OR  N/A

Extremely 

Easy 

10. How easy was it to access information from 

the URET Graphics Plan View Display

when needed? 

Extremely 

Difficult 

123456789

OR  N/A

Extremely 

Easy 

11. How easy was it to perform URET Trial 

Plans when you wanted? 

Extremely 

Difficult 

123456789

OR  N/A

Extremely 

Easy 

12. How often did you feel URET caused clutter 

on the display? Never 123456789

OR  N/A Always 

13. What impact did URET have on your 

workload? 

Negative 

(increased) 123456789 Positive (decreased)

14. How much URET coordination with your 

D-side team member occurred? 

None At 

All 123456789 A Great Deal 

15. What impact did URET coordination with 

your D-side team member have on your 

workload? 

Negative 

(increased) 123456789 Positive (decreased)

16. What impact did URET have on your 

situational awareness? 

Negative 

(decreased) 123456789 Positive (increased)

17. What impact did URET have on your ability 

to control traffic safely? 

Negative 

(decreased) 123456789 Positive (increased)

18. Additional comments regarding URET: 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
D-3 

TMA 

19. Overall, how easy was it to use TMA during 

this scenario? 

Extremely 

Difficult 123456789 Extremely Easy 

20. How easy was it to access information from 

the TMA List when needed? 

Extremely 

Difficult 123456789 Extremely Easy 

21. How often did you feel the TMA List caused 

clutter on the display? Never 123456789 Always 

22. What impact did TMA have on your 

workload? 

Negative 

(increased) 123456789 Positive (decreased)

23. How much TMA coordination with your 

D-side team member occurred? 

None At 

All 123456789 A Great Deal 

24. What impact did TMA coordination with 

your D-side team member have on your 

workload? 

Negative 

(increased) 123456789 Positive (decreased)

25. What impact did TMA have on your 

situational awareness? 

Negative 

(decreased) 123456789 Positive (increased)

26. What impact did TMA have on your ability 

to control traffic safely? 

Negative 

(decreased) 123456789 Positive (increased)

27. Additional comments regarding TMA: 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
D-4 

CPDLC 

28. Overall, how easy was it to use CPDLC

during this scenario? 

Extremely 

Difficult 123456789 Extremely Easy 

29. How easy was it to access information from 

the CPDLC Message Out View when 

needed? 

Extremely 

Difficult 

123456789

OR  N/A

Extremely 

Easy 

30. How easy was it to access information from 

the CPDLC History Window when needed? 

Extremely 

Difficult 

123456789

OR  N/A

Extremely 

Easy 

31. How easy was it to use the CPDLC Menu 

Text Window when needed? 

Extremely 

Difficult 

123456789

OR  N/A

Extremely 

Easy 

32. How easy was it to access information from 

the Datablock CPDLC Status Line 0 when 

needed? 

Extremely 

Difficult 

123456789

OR  N/A

Extremely 

Easy 

33. How easy was it to use the Datablock 

CPDLC Fly-out Windows when needed? 

Extremely 

Difficult 

123456789

OR  N/A

Extremely 

Easy 

34. How easy was it to use the CRD CPDLC 

Text Commands when needed? 

Extremely 

Difficult 

123456789

OR  N/A

Extremely 

Easy 

35. How often did you feel CPDLC caused 

clutter on your display? Never 123456789 Always 

36. What impact did CPDLC have on your 

workload? 

Negative 

(increased) 123456789 Positive (decreased)

37. How much CPDLC coordination with your 

D-side team member occurred? 

None At 

All 123456789 A Great Deal 

38. What impact did CPDLC coordination with 

your D-side team member have on your 

workload? 

Negative 

(increased) 123456789 Positive (decreased)
D-5 

CPDLC (continued) 

39. What impact did CPDLC have on your 

situational awareness? 

Negative 

(decreased) 123456789 Positive (increased)

40. What impact did CPDLC have on your 

ability to control traffic safely? 

Negative 

(decreased) 123456789 Positive (increased)

41. What impact did CPDLC Menu Text 

crossing restrictions have on your 

workload? 

Negative 

(increased)

123456789

OR  N/A

Positive 

(decreased)

42. What impact did CPDLC Menu Text 

crossing restrictions have on your 

situational awareness? 

Negative 

(decreased)

123456789

OR  N/A

Positive 

(increased)

43. What impact did CPDLC Menu Text 

crossing restrictions have on your ability to 

control traffic safely? 

Negative 

(decreased)

123456789

OR  N/A

Positive 

(increased)

44. Additional comments regarding CPDLC: 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
D-6 

Post-Scenario Questionnaire 

D-side Version

Instructions: 

Please answer the following questions based upon your experience in the scenario just 

completed. The rating scales have labels at either end and 10 numbered circles that represent 

different levels of response. Fill in one circle with a pen or pencil (or mark with an X) to

indicate your level of response OR mark  N/A if the question deals with something you did 

not try at all during the simulation. 

Overall Performance, Workload, Situational Awareness, and Simulation Ratings 

1. Rate your overall level of ATC performance

during this scenario. 

Extremely 

Poor 123456789 Extremely Good 

2. Rate your workload due to communications 

with pilots during this scenario. 

Extremely 

Low 123456789 Extremely High 

3. How much coordination with your R-side 

team member occurred during this scenario? 

None At 

All 123456789 A Great Deal 

4. Rate your workload due to coordination with 

your R-side team member during this 

scenario. 

Extremely 

Low 123456789 Extremely High 

5. Rate your level of situational awareness

during this scenario. 

Extremely 

Poor 123456789 Extremely Good 

6. Rate the performance of the simulation pilots 

in terms of their responding to control 

instructions and providing readbacks. 

Extremely 

Poor 123456789 Extremely Good 

7. Rate the difficulty of this scenario. Extremely 

Difficult 123456789 Extremely Easy 
D-7 

URET 

8. Overall, how easy was it to use URET during 

this scenario? 

Extremely 

Difficult 123456789 Extremely Easy 

9. How easy was it to access information from 

the URET Aircraft List when needed? 

Extremely 

Difficult 123456789 Extremely Easy 

10. How easy was it to access information from 

the URET Graphics Plan View Display

when needed? 

Extremely 

Difficult 123456789 Extremely Easy 

11. How easy was it to perform URET Trial 

Plans when you wanted? 

Extremely 

Difficult 123456789 Extremely Easy 

12. How often did you feel URET caused clutter 

on the display? Never 123456789 Always 

13. What impact did URET have on your 

workload? 

Negative 

(increased) 123456789 Positive (decreased)

14. How much URET coordination with your 

R-side team member occurred? 

None At 

All 123456789 A Great Deal 

15. What impact did URET coordination with 

your R-side team member have on your 

workload? 

Negative 

(increased) 123456789 Positive (decreased)

16. What impact did URET have on your 

situational awareness? 

Negative 

(decreased) 123456789 Positive (increased)

17. What impact did URET have on your ability 

to support your R-side team member? 

Negative 

(decreased) 123456789 Positive (increased)

18. Additional comments regarding URET: 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
D-8 

TMA 

19. Overall, how easy was it to use TMA during 

this scenario? 

Extremely 

Difficult 

123456789

OR  N/A

Extremely 

Easy 

20. How easy was it to access information from 

the TMA List when needed? 

Extremely 

Difficult 

123456789

OR  N/A

Extremely 

Easy 

21. How often did you feel the TMA List caused 

clutter on the display? Never 123456789

OR  N/A Always 

22. What impact did TMA have on your 

workload? 

Negative 

(increased) 123456789 Positive (decreased)

23. How much TMA coordination with your 

R-side team member occurred? 

None At 

All 123456789 A Great Deal 

24. What impact did TMA coordination with 

your R-side team member have on your 

workload? 

Negative 

(increased) 123456789 Positive (decreased)

25. What impact did TMA have on your 

situational awareness? 

Negative 

(decreased) 123456789 Positive (increased)

26. What impact did TMA have on your ability 

to support your R-side team member? 

Negative 

(decreased) 123456789 Positive (increased)

27. Additional comments regarding TMA: 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
D-9 

CPDLC 

28. Overall, how easy was it to use CPDLC

during this scenario? 

Extremely 

Difficult 123456789 Extremely Easy 

29. How easy was it to access information from 

the CPDLC Message Out View when 

needed? 

Extremely 

Difficult 

123456789

OR  N/A

Extremely 

Easy 

30. How easy was it to access information from 

the CPDLC History Window when needed? 

Extremely 

Difficult 

123456789

OR  N/A

Extremely 

Easy 

31. How easy was it to use the CPDLC Menu 

Text Window when needed? 

Extremely 

Difficult 

123456789

OR  N/A

Extremely 

Easy 

32. How easy was it to access information from 

the Datablock CPDLC Status Line 0 when 

needed? 

Extremely 

Difficult 

123456789

OR  N/A

Extremely 

Easy 

33. How easy was it to use the Datablock 

CPDLC Fly-out Windows when needed? 

Extremely 

Difficult 
 N/A Extremely Easy 

34. How easy was it to use the CRD CPDLC 

Text Commands when needed? 

Extremely 

Difficult 

123456789

OR  N/A

Extremely 

Easy 

35. How often did you feel CPDLC caused 

clutter on your display? Never 123456789 Always 

36. What impact did CPDLC have on your 

workload? 

Negative 

(increased) 123456789 Positive (decreased)

37. How much CPDLC coordination with your 

R-side team member occurred? 

None At 

All 123456789 A Great Deal 

38. What impact did CPDLC coordination with 

your R-side team member have on your 

workload? 

Negative 

(increased) 123456789 Positive (decreased)
D-10 

CPDLC (continued) 

39. What impact did CPDLC have on your 

situational awareness? 

Negative 

(decreased) 123456789 Positive (increased)

40. What impact did CPDLC have on your 

ability to support your R-side team 

member? 

Negative 

(decreased) 123456789 Positive (increased)

41. What impact did CPDLC Menu Text 

crossing restrictions have on your 

workload? 

Negative 

(increased)

123456789

OR  N/A

Positive 

(decreased)

42. What impact did CPDLC Menu Text 

crossing restrictions have on your 

situational awareness? 

Negative 

(decreased)

123456789

OR  N/A

Positive 

(increased)

43. What impact did CPDLC Menu Text 

crossing restrictions have on your ability to 

control traffic safely? 

Negative 

(decreased)

123456789

OR  N/A

Positive 

(increased)

44. Additional comments regarding CPDLC: 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
D-11 

NASA-TLX Ratings 

Definitions 

Mental Demand – How much mental and perceptual activity was required (e.g., thinking, 

deciding, calculating, remembering, looking, searching, etc.)? Were your tasks easy or 

demanding, simple or complex, exacting or forgiving? 

Physical Demand – How much physical activity was required (e.g., data entry, strip marking, 

talking, pointing, etc.)? Were your tasks easy or demanding, slow or brisk, slack or 

strenuous, restful or laborious? 

Temporal Demand – How much time pressure did you feel due to the rate or pace at which 

your tasks occurred? Was the pace slow and leisurely or rapid and frantic? 

Performance – How successful do you think you were in accomplishing the goals of your 

tasks? How satisfied were you with your performance in accomplishing these goals? 

Effort – How hard did you have to work (mentally and physically) to accomplish this level of 

performance? 

Frustration – How insecure, discouraged, irritated, stressed, and annoyed versus secure, 

gratified, content, relaxed, and complacent did you feel in performing your tasks? 

45. Rate your mental demand during this 

scenario. 

Extremely 

Low 123456789 Extremely High 

46. Rate your physical demand during this 

scenario. 

Extremely 

Low 123456789 Extremely High 

47. Rate your temporal demand during this 

scenario. 

Extremely 

Low 123456789 Extremely High 

48. Rate your performance during this scenario. Extremely 

Low 123456789 Extremely High 

49. Rate your effort during this scenario. Extremely 

Low 123456789 Extremely High 

50. Rate your frustration during this scenario. Extremely 

Low 123456789 Extremely High 

51. Do you have any comments or clarifications about these NASA-TLX questions? 

___________________________________________________________________________ 

___________________________________________________________________________ 
E-1 

APPENDIX E 

Subject Matter Expert Observer Rating Form 

Observer Code _________ Date _________ 

Controller _________ Scenario _________ 

INSTRUCTIONS

This form is designed to be used by supervisory air traffic control specialists to evaluate the 

effectiveness of controllers working in simulation environments. SATCSs will observe and rate 

the performance of controllers in several different performance dimensions using the scale below 

as a general purpose guide. Use the entire scale range as much as possible. You will see a wide 

range of controller performance. Take extensive notes on what you see. Do not depend on your 

memory. Write down your observations. Space is provided after each scale for comments. You 

may make preliminary ratings during the course of the scenario. However, wait until the 

scenario is finished before making your final ratings and remain flexible until the end when you 

have had an opportunity to see all the available behavior. At all times please focus on what you 

actually see and hear. This includes what the controller does and what you might reasonably 

infer from the actions of the pilots. Try to avoid inferring what you think may be happening. If 

you do not observe relevant behavior or the results of that behavior, then you may leave a 

specific rating blank. Also, please write down any comments that may help improve this 

evaluation form. Do not write your name on the form itself. You will not be identified by name. 

An observer code known only to yourself and the researchers conducting this study will be 

assigned to you. The observations you make do not need to be restricted to the performance 

areas covered in this form and may include other areas that you think are important. 

ASSUMPTIONS

ATC is a complex activity that contains both observable and unobservable behavior. There 

are so many complex behaviors involved that no observational rating form can cover everything. 

A sample of the behaviors is the best that can be achieved, and a good form focuses on those 

behaviors that controllers themselves have identified as the most relevant in terms of their overall 

performance. Most controller performance is at or above the minimum standards regarding 

safety and efficiency. The goal of the rating system is to differentiate performance above this 

minimum. The lowest rating should be assigned for meeting minimum standards and also for 

anything below the minimum since this should be a rare event. It is important for the 

observer/rater to feel comfortable using the entire scale and to understand that all ratings should 

be based on behavior that is actually observed. 
E-2 

Rating Scale Descriptors 

Remove this page and keep it available while doing ratings 

SCALE QUALITY SUPPLEMENTARY

1 Least Effective

Unconfident, Indecisive, Inefficient, Disorganized,

Behind the power curve, Rough, Leaves some tasks 

incomplete, Makes mistakes 

2 Poor May issue conflicting instructions, Doesn’t plan 

completely 

3 Fair Distracted between tasks 

4 Low Satisfactory Postpones routine actions 

5 High Satisfactory Knows the job fairly well 

6 Good Works steadily, Solves most problems 

7 Very Good Knows the job thoroughly, Plans well 

8 Most Effective 

Confident, Decisive, Efficient, Organized, Ahead of the 

power curve, Smooth, Completes all necessary tasks,

Makes no mistakes 
E-3 

I - MAINTAINING SAFE AND EFFICIENT TRAFFIC FLOW

II - MAINTAINING ATTENTION AND SITUATION AWARENESS

III - PRIORITIZING

IV - PROVIDING CONTROL INFORMATION

V - TECHNICAL KNOWLEDGE

VI - COMMUNICATING
E-4 

1. What is the D-side doing? [Teamwork, Roles & Responsibilities] 

2. What URET information is being exchanged between R-side & D-side team members? 

 How does the D-side use URET to support the R-side? 

 When does the D-side use URET Trial Plans? 

 Any differences in URET usage depending upon availability of other tools?

3. What TMA information is being exchanged between R-side & D-side team members? 

 How does the D-side use TMA to support the R-side? 

 When does the D-side interact with TMA? 

 Any differences in TMA usage depending upon availability of other tools?

4. What CPDLC information is being exchanged between R-side & D-side team members?

 How does the D-side use CPDLC to support the R-side? 

 When does the R-side use CPDLC? 

 Any differences in CPDLC usage depending upon availability of other tools? 
E-5 

I - MAINTAINING SAFE AND EFFICIENT TRAFFIC FLOW

1. Maintaining Separation and Resolving Potential Conflicts................................ 1 2 3 4 5 6 7 8

• using control instructions that maintain appropriate aircraft 

and airspace separation 

• detecting and resolving impending conflicts early 

• recognizing the need for speed restrictions and wake 

turbulence separation 

2. Sequencing Aircraft Efficiently ................................................................ 1 2 3 4 5 6 7 8

• using efficient and orderly spacing techniques for arrival, 

departure, and en route aircraft 

• maintaining safe arrival and departure intervals that minimize 

delays 

3. Using Control Instructions Effectively/Efficiently...........................1 2 3 4 5 6 7 8

• providing accurate navigational assistance to pilots 

• issuing economical clearances that result in need for few 

additional instructions to handle aircraft completely

• ensuring clearances require minimum necessary flight path 

changes

4. Overall Safe and Efficient Traffic Flow Scale Rating ................................ 1 2 3 4 5 6 7 8

II - MAINTAINING ATTENTION AND SITUATION AWARENESS

5. Maintaining Awareness of Aircraft Positions................................................................ 1 2 3 4 5 6 7 8

• avoiding fixation on one area of the radar scope when other 

areas need attention 

• using scanning patterns that monitor all aircraft on the radar 

scope 

6. Giving and Taking Handoffs in a Timely Manner............................1 2 3 4 5 6 7 8

• ensuring that handoffs are initiated in a timely manner 

• ensuring that handoffs are accepted in a timely manner 

• ensuring that handoffs are made according to procedures 

7. Ensuring Positive Control ................................................................ 1 2 3 4 5 6 7 8

• tailoring control actions to situation 

• using effective procedures for handling heavy, emergency, and 

unusual traffic situations 

8. Detecting Pilot Deviations from Control Instructions......................1 2 3 4 5 6 7 8

• ensuring that pilots follow assigned clearances correctly 

• correcting pilot deviations in a timely manner 

9. Correcting Own Errors in a Timely Manner.....................................1 2 3 4 5 6 7 8

• acting quickly to correct errors 

• changing an issued clearance when necessary to expedite 

traffic flow 

10. Overall Attention and Situation Awareness Scale Rating ................................ 1 2 3 4 5 6 7 8
E-6 

III - PRIORITIZING

11. Taking Actions in an Appropriate Order of Importance...................1 2 3 4 5 6 7 8

• resolving situations that need immediate attention before 

handling low priority tasks 

• issuing control instructions in a prioritized, structured, and 

timely manner 

12. Preplanning Control Actions.............................................................1 2 3 4 5 6 7 8

• scanning adjacent sectors to plan for future and conflicting 

traffic 

• studying pending flight strips in bay 

13. Handling Control Tasks for Several Aircraft....................................1 2 3 4 5 6 7 8

• shifting control tasks between several aircraft when necessary 

• communicating in timely fashion while sharing time with 

other actions 

14. Marking Flight Strips while Performing Other Tasks ......................1 2 3 4 5 6 7 8

• marking flight strips accurately while talking or performing 

other tasks 

• keeping flight strips current 

15. Overall Prioritizing Scale Rating ......................................................1 2 3 4 5 6 7 8

IV - PROVIDING CONTROL INFORMATION

16. Providing Essential Air Traffic Control Information........................1 2 3 4 5 6 7 8

• providing mandatory services and advisories to pilots in a 

timely manner 

• exchanging essential information 

17. Providing Additional Air Traffic Control Information.....................1 2 3 4 5 6 7 8

• providing additional services when workload permits 

• exchanging additional information 

18. Providing Coordination.....................................................................1 2 3 4 5 6 7 8

• providing effective and timely coordination 

• using proper point-out procedures 

19. Overall Providing Control Information Scale Rating ................................ 1 2 3 4 5 6 7 8
E-7 

V - TECHNICAL KNOWLEDGE

20. Showing Knowledge of LOAs and SOPs .........................................1 2 3 4 5 6 7 8

• controlling traffic as depicted in current LOAs and SOPs 

• performing handoff procedures correctly 

21a.Showing Knowledge of Aircraft Capabilities and Limitations.........1 2 3 4 5 6 7 8

• using appropriate speed, vectoring, and/or altitude assignments 

to separate aircraft with varied flight capabilities 

• issuing clearances that are within aircraft performance 

parameters 

21b.Showing Effective Use of Equipment...............................................1 2 3 4 5 6 7 8

• updating data blocks 

• using equipment capabilities 

22. Overall Technical Knowledge Scale Rating .....................................1 2 3 4 5 6 7 8

VI - COMMUNICATING

23. Using Proper Phraseology................................................................ 1 2 3 4 5 6 7 8

• using words and phrases specified in the 7110.65 

• using phraseology that is appropriate for the situation 

• using minimum necessary verbiage 

24. Communicating Clearly and Efficiently ...........................................1 2 3 4 5 6 7 8

• speaking at the proper volume and rate for pilots to understand

• speaking fluently while scanning or performing other tasks 

• ensuring clearance delivery is complete, correct and timely 

• speaking with confident, authoritative tone of voice 

25. Listening to Pilot Readbacks and Requests ......................................1 2 3 4 5 6 7 8

• correcting pilot readback errors 

• acknowledging pilot or other controller requests promptly 

• processing requests correctly in a timely manner 

26. Overall Communicating Scale Rating...............................................1 2 3 4 5 6 7 8
F-1 

APPENDIX F 

Exit Questionnaire 

R-side Version

Instructions: 

Please answer the following questions based upon your overall experience in the 

simulation. The rating scales have labels at either end and 10 numbered circles that represent 

different levels of response. Fill in one circle with a pen or pencil (or mark with an X) to

indicate your level of response OR mark  Unknown if the question deals with something you 

have no field experience with. Unless otherwise stated, the questions refer to your experience 

as an R-side controller with a D-side team member. 

Simulation Realism and Research Apparatus Ratings 

1. Rate the realism of the overall simulation 

experience compared to actual ATC 

operations. 

Extremely 

Unrealistic 123456789 Extremely Realistic 

2. Rate the realism of the simulation URET 

implementation compared to actual field 

operation. 

Extremely 

Unrealistic

123456789

OR  Unknown

Extremely 

Realistic 

3. Rate the realism of the simulation TMA 

implementation compared to actual field 

operation. 

Extremely 

Unrealistic

123456789

OR  Unknown

Extremely 

Realistic 

4. Rate the realism of the simulation CPDLC 

emulation compared to actual field operation. 

Extremely 

Unrealistic

123456789

OR  Unknown

Extremely 

Realistic 

5. Rate the realism of the simulation DSR 

hardware compared to actual DSR 

equipment. 

Extremely 

Unrealistic 123456789 Extremely Realistic 

6. Rate the realism of the simulation DSR 

software compared to actual DSR 

functionality. 

Extremely 

Unrealistic 123456789 Extremely Realistic 

7. Rate the realism of the simulation generic 

airspace compared to actual NAS airspace. 

Extremely 

Unrealistic 123456789 Extremely Realistic 

8. Rate the realism of the simulation traffic 

scenarios compared to actual NAS traffic. 

Extremely 

Unrealistic 123456789 Extremely Realistic 
F-2 

9. To what extent did the oculometer interfere

with your ATC performance? 

None At 

All 

123456789

OR  N/A

A Great 

Deal 

10. To what extent did the ATWIT online 

workload rating technique interfere with 

your ATC performance? 

None At 

All 123456789 A Great Deal 

11. Do you have any comments or suggestions for improvement about our simulation 

capability? 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
F-3 

URET 

12. What URET information did you want from your D-side team member? 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

13. When did you want URET Trial Plans data from your D-side team member? 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

14. Did you want different URET information depending upon 

the availability of the other tools?  Yes  No

If yes, please explain. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

15. Please explain the difference between how you controlled traffic with and without URET. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
F-4 

TMA 

16. What TMA information (if any) did you want from your D-side team member? 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

17. When did you want assistance from your D-side team member with TMA? 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

18. Did you use TMA differently depending upon the 

availability of the other tools?  Yes  No

If yes, please explain. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

19. Please explain the difference between how you controlled traffic with and without TMA. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
F-5 

CPDLC 

20. What CPDLC information (if any) did you want from your D-side team member? 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

21. When did you want your D-side team member to use CPDLC? 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

22. Did you use CPDLC differently depending upon the 

availability of the other tools?  Yes  No

If yes, please explain. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

23. Please explain the difference between how you controlled traffic with and without CPDLC. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
F-6 

CPDLC (continued) 

24. Were there any benefits in using CPDLC Menu Text 

crossing restrictions?  Yes  No

If yes, please explain. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

25. Were there any safety or other problems in using CPDLC 

Menu Text crossing restrictions?  Yes  No

If yes, please explain. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

26. What was your overall impression of the computer-human interface of the CPDLC D-side 

design (e.g., toolbar, window cycle, and default/preferred window location buttons)? 

Do you have any ideas for improvements? 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
F-7 

Tool Combinations 

27. Did you notice any problems with displays covering 

important information in any of the tool combinations?  Yes  No

If yes, please explain which tool combinations and what the problems were. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

28. Did you notice any inconsistent display characteristics

between any of the tools that could be confusing?  Yes  No

If yes, please explain which tool combinations and what the problems were. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

29. Did you notice any inconsistency in the interaction between 

any of the tools that could be confusing?  Yes  No

If yes, please explain which tool combinations and what the problems were. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
F-8 

30. Did you notice any other problems in using any of the tool 

combinations?  Yes  No

If yes, please explain which tool combinations and what the problems were. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

31. Is there a need for information transfer between any of the 

tools that could benefit from automation?  Yes  No

If yes, please explain. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

32. Is there any additional functionality that you would like to 

have on any of the tools?  Yes  No

If yes, please explain. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
F-9 

33. Are there any procedures that should be set up due to the 

change in operations with the addition of the tools?  Yes  No

If yes, please explain. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

34. As an R-side Only controller, did you have any problems 

using the tools without D-side assistance?  Yes  No

If yes, please explain what the problems were. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

35. Is there anything else that we should have asked or that you would like to comment on? 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
F-10 

Exit Questionnaire 

D-side Version

Instructions: 

Please answer the following questions based upon your overall experience in the 

simulation. The rating scales have labels at either end and 10 numbered circles that represent 

different levels of response. Fill in one circle with a pen or pencil (or mark with an X) to

indicate your level of response OR mark  Unknown if the question deals with something you 

have no field experience with. Unless otherwise stated, the questions refer to your experience 

as a D-side controller assisting your R-side team member. 

Simulation Realism and Research Apparatus Ratings 

1. Rate the realism of the overall simulation 

experience compared to actual ATC 

operations. 

Extremely 

Unrealistic 123456789 Extremely Realistic 

2. Rate the realism of the simulation URET 

implementation compared to actual field 

operation. 

Extremely 

Unrealistic

123456789

OR  Unknown

Extremely 

Realistic 

3. Rate the realism of the simulation TMA 

implementation compared to actual field 

operation. 

Extremely 

Unrealistic

123456789

OR  Unknown

Extremely 

Realistic 

4. Rate the realism of the simulation CPDLC 

emulation compared to actual field operation. 

Extremely 

Unrealistic

123456789

OR  Unknown

Extremely 

Realistic 

5. Rate the realism of the simulation DSR 

hardware compared to actual DSR 

equipment. 

Extremely 

Unrealistic 123456789 Extremely Realistic 

6. Rate the realism of the simulation DSR 

software compared to actual DSR 

functionality. 

Extremely 

Unrealistic 123456789 Extremely Realistic 

7. Rate the realism of the simulation generic 

airspace compared to actual NAS airspace. 

Extremely 

Unrealistic 123456789 Extremely Realistic 

8. Rate the realism of the simulation traffic 

scenarios compared to actual NAS traffic. 

Extremely 

Unrealistic 123456789 Extremely Realistic 
F-11 

9. To what extent did the oculometer interfere

with your ATC performance? 

None At 

All 

123456789

OR  N/A

A Great 

Deal 

10. To what extent did the ATWIT online 

workload rating technique interfere with 

your ATC performance? 

None At 

All 123456789 A Great Deal 

11. Do you have any comments or suggestions for improvement about our simulation 

capability? 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
F-12 

URET 

12. What URET information did you provide your R-side team member? 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

13. When did you use the URET Trial Plans feature? 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

14. Did you use URET differently depending upon the 

availability of the other tools?  Yes  No

If yes, please explain. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

15. Please explain the difference between how you supported your R-side team member with and 

without URET. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
F-13 

TMA 

16. What TMA information (if any) did you provide your R-side team member? 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

17. When did you assist your R-side team member with TMA? 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

18. Did you provide different TMA information depending 

upon the availability of the other tools?  Yes  No

If yes, please explain. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

19. Please explain the difference between how you supported your R-side team member with and 

without TMA. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
F-14 

CPDLC 

20. What CPDLC information (if any) did you provide your R-side team member? 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

21. When did you use CPDLC? 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

22. Did you use CPDLC differently depending upon the 

availability of the other tools?  Yes  No

If yes, please explain. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

23. Please explain the difference between how you supported your R-side team member with and 

without CPDLC. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
F-15 

CPDLC (continued) 

24. Were there any benefits in using CPDLC Menu Text 

crossing restrictions?  Yes  No

If yes, please explain. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

25. Were there any safety or other problems in using CPDLC 

Menu Text crossing restrictions?  Yes  No

If yes, please explain. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

26. What was your overall impression of the computer-human interface of the CPDLC D-side 

design (e.g., toolbar, window cycle, and default/preferred window location buttons)? 

Do you have any ideas for improvements? 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
F-16 

Tool Combinations 

27. Did you notice any problems with displays covering 

important information in any of the tool combinations?  Yes  No

If yes, please explain which tool combinations and what the problems were. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

28. Did you notice any inconsistent display characteristics

between any of the tools that could be confusing?  Yes  No

If yes, please explain which tool combinations and what the problems were. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

29. Did you notice any inconsistency in the interaction between 

any of the tools that could be confusing?  Yes  No

If yes, please explain which tool combinations and what the problems were. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
F-17 

30. Did you notice any other problems in using any of the tool 

combinations?  Yes  No

If yes, please explain which tool combinations and what the problems were. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

31. Is there a need for information transfer between any of the 

tools that could benefit from automation?  Yes  No

If yes, please explain. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

32. Is there any additional functionality that you would like to 

have on any of the tools?  Yes  No

If yes, please explain. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 
F-18 

33. Are there any procedures that should be set up due to the 

change in operations with the addition of the tools?  Yes  No

If yes, please explain. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

34. As an R-side Only controller, did you have any problems 

using the tools without D-side assistance?  Yes  No

If yes, please explain what the problems were. 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

35. Is there anything else that we should have asked or that you would like to comment on? 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________ 

___________________________________________________________________________