# FAA/NASA Human Factors for 

# Evolving Environments:

Human Factors Attributes and

Technology Readiness Levels

> ![](media/image1.png){width="2.375in" height="1.75in"}

  ----------------------- ----------------------- -----------------------
  Paul Krois              Richard Mogford         Jacqueline Rehmann

  Human Factors Research  Advanced Aerospace      Office of System
  and Engineering         Transportation          Architecture and
  Division                Technologies            Investment Analysis

  Federal Aviation        NASA Ames Research      Federal Aviation
  Administration          Center                  Administration

  Washington, D.C.        Moffett Field, CA       Atlantic City, NJ
  ----------------------- ----------------------- -----------------------

DOT/FAA/AR-03/43

April 2003

For More Information Contact:

Human Factors Research and Engineering Division

Federal Aviation Administration\
AAR-100, Room 907

800 Independence Avenue, SW

Washington, DC 20591

# (202) 267 -- 7219 

# **FAA/NASA Human Factors for Evolving Environments:**

### Human Factors Attributes and Technology Readiness Levels

Paul Krois, Richard Mogford, and Jacqueline Rehmann

## Summary

The purpose of this paper is to provide guidance to researchers and
acquisition offices on human factors attributes important for
engineering projects as they mature through the research and development
pipeline. In order to identify the human factors attributes potentially
associated with each Technology Readiness Level (TRL), human factors
practitioners experienced with air traffic control research and
acquisitions rated a set of 23 attributes for importance for each of six
TRLs. There were general trends showing increasing importance for
specific attributes as a research capability progresses across the TRLs,
and an increasing number of attributes that are important at higher
TRLs. The findings argue for more robust early involvement of human
factors at lower TRLs, and that FAA implementing organizations engage
with researchers at TRL 3 rather than TRL 4 as typically prescribed.

**1.0 Introduction**

The goal of the FAA and NASA Interagency Air Traffic Management
Integrated Product Team (IAIPT) is to coordinate research necessary to
develop and field systems that enhance the National Airspace System
(NAS). To manage this research, FAA, NASA, and their research partners
have chosen a model of Technology Readiness Levels (TRLs) as the basis
for coordinating various activities, roles, and expectations. The TRL
model provides criteria against which the FAA and research organizations
can assess a capability's readiness to transition to the next level of
development, or TRL. As research proceeds, the TRL model provides key
milestones to ensure organizations define/refine a product to satisfy
FAA requirements. The purpose of this paper is to provide guidance on
identifying the human factors attributes potentially associated with
each TRL.

**2.0 Background**

In September 1995, the FAA and NASA strengthened their partnership by
forming the FAA/NASA IAIPT. The mission of the IAIPT is to plan and
conduct integrated research related to both air-based and ground-based
air traffic control (ATC) and air traffic management (ATM) decision
support tools and procedures. Oversight for IAIPT activities is provided
by the FAA's Research, Engineering & Development (RE&D) Advisory
Committee and NASA's Aeronautics and Space Transportation Technology
Advisory Committee. The IAIPT provides the pipeline for how emerging and
maturing research concepts and prototypes are fed into the FAA
acquisition management system.

The goal of the IAIPT is to perform the research necessary to field
systems that provide the NAS capabilities described in the Radio
Technical Communications Association **(**RTCA) National Airspace
Concept of Operations (RTCA, 2002). To provide these needed
capabilities, the FAA, NASA, and their research partners must work
together as part of a system development and implementation process,
meeting a series of milestones.

As described in the Free Flight Research Program Plan (FAA, 2000),
research goals are centered on RTCA-recommended research concepts and
achievement of an investment decision. To achieve these goals, the FAA,
NASA, and the MITRE Center for Advanced Aviation System Development
(CAASD) have chosen the TRL model as the basis for coordinating research
plans, responsibilities, and schedules. The TRL model allows the
research organization to describe, develop, and perform limited testing
of proposed concepts. As the research activity proceeds, the model
provides key milestones where the researchers work to define a product
that will satisfy FAA requirements.

The TRL model provides a framework to implement a phased approach
towards research and development and define various roles and
responsibilities of research organizations and the FAA. Transitioning
from each TRL will require a meeting/review between FAA and the research
organizations to ensure that exit criteria have been met. Outcomes of
these TRL transition reviews may consist of TRL approval, disapproval,
or termination of the R&D activity. These discussions should occur
approximately 1 month prior to the anticipated end of the transition
period for a TRL. Individual research transition plans (RTPs) should
identify this in their milestone schedules.

Technical Interchange Meetings (TIMs) are an important way for IAIPT
members to exchange information and perspectives. Human factors TIMs are
coordinated through the IAIPT System/Cross-Cutting Joint Research
Product Description (JRPD) number 12, called Human Factors for Evolving
Environments. JRPD 12 team members Paul Krois, FAA/AAR-100, Jacqueline
Rehmann, FAA/ASD-100, and Richard Mogford, NASA Ames, co-hosted an IAIPT
human factors TIM in Washington, D.C. The focus of the two-day meeting
held November 19-20, 2002, was to discuss the role of human factors in
progressing capabilities through research and development (R&D) to
acquisition and deployment.

Meeting participants came from the FAA, NASA Ames and Langley, MITRE
CAASD, and the aviation industry. Participants discussed the important
contribution of human factors in transitioning research concepts and
products through the R&D pipeline to acquisition and fielded systems.
The meeting focused on refining human factors guidance for transitioning
research capabilities across TRLs.

Meeting participants agreed that transitioning research concepts from
exploration to development and onward to acquisition products and
fielded systems should be accompanied by increasingly detailed
assessments of information requirements, display management/integration,
human-centered automation, and human performance measures such as
workload, situation awareness, communications, and human error. They
concurred that human factors considerations are just one component of a
broader integrated system engineering perspective encompassing
operational concepts, system requirements, and system engineering
methods.

Presentations focused on identifying and discussing human factors and
human performance issues across different levels of capability maturity.
Results of a recent TRL Human Factors survey were provided to meeting
participants for discussion and refinement.

**3.0 Technology Readiness Levels: A Model for Research, Development,
and Implementation**

Descriptions of the TRLs are documented in several sources including the
Integrated Plan for Air Traffic Management Research and Technology
Development, Version 6.0 (FAA, 2002), the Free Flight Research Program
Plan (FAA, 2000), and various internal guidance documents developed by
NASA and FAA. The TRL model provides for a phased approach to research
and development and defines roles and responsibilities of both the
research organization and the FAA. TRL transition reviews, during which
TRL exit criteria are discussed, may result in three possible outcomes
as described above.

Figure 1 provides an overview of TRLs 1-6 with a brief description of
what is included in each (FAA, 2002). It is important to note that while
the TRL model is recognized as a useful general approach for research,
development, and implementation, it may not lend itself to some research
products, e.g., there may not be clearly identifiable transition points
across research phases and transfer to a FAA Integrated Product Team
(IPT).

#### Figure 1: TRL Overview

![](media/image3.wmf){width="7.152777777777778in"
height="4.516666666666667in"}

While a research capability may exit TRL 6 and be transitioned to a FAA
IPT, it may continue to undergo certain further research. The TRL model
also provides for TRL 7 through TRL 9 where research may examine
integration issues and site adaptation requirements. At these TRLs,
human factors acquisition policy, processes, and best practices that
provide direction, standards, tools, data, and guidance would be
directly applicable.

## 4.0 TRL Activity Descriptions

# 

# The following discussion for TRLs 1 through 6 reflects a synthesis and summary of the current guidance available. To move from one level to another, research products must meet a number of exit criteria. Exit criteria for TRLs 1 through 6 are summarized in Table 1. 

Table 1: TRL Exit Criteria

+----+-------------------+--------------------------------------------+
| *  | **Title**         | ###### Criteria                            |
| *T |                   |                                            |
| RL |                   |                                            |
| ** |                   |                                            |
+----+-------------------+--------------------------------------------+
| 1  | Basic Principles  | Initial concept description is provided    |
|    | Observed/Reported | and is consistent with top-level Concept   |
|    |                   | of Operations; benefits, risks, and        |
|    |                   | research issues are identified.            |
+----+-------------------+--------------------------------------------+
| 2  | Technology        | Research management plan is delivered and  |
|    | Concept and/or    | FAA Research Management plan is delivered  |
|    | Application       | if applicable. Single year benefits        |
|    | Formulated        | assessment showing performance and         |
|    |                   | economic benefits, preliminary safety risk |
|    |                   | assessment, and preliminary human factors  |
|    |                   | assessment and research plan must be       |
|    |                   | completed.                                 |
+----+-------------------+--------------------------------------------+
| 3  | Analyt            | Initial Feasibility report is submitted    |
|    | ical/Experimental | showing capability is feasible from        |
|    | Critical Function | technical, benefits, safety, and human     |
|    | or Characteristic | factors perspectives. Initial analytic or  |
|    | Proof-of-Concept  | experimental quantification of technical   |
|    |                   | performance metrics shows improvement over |
|    |                   | baseline.                                  |
+----+-------------------+--------------------------------------------+
| 4  | Component and/or  | Research demonstrates capability is        |
|    | Integrated        | feasible from safety, human factors, and   |
|    | Components Tested | development perspectives, and expected     |
|    | in a Laboratory   | benefits outweigh costs based upon         |
|    | Environment       | human-in-the-loop testing with             |
|    |                   | representative potential users. A FAA      |
|    |                   | baseline Concept of Use for the capability |
|    |                   | is developed.                              |
+----+-------------------+--------------------------------------------+
| 5  | Components and/or | Pre-development prototype is developed and |
|    | Subsystems        | evaluated in a high fidelity environment.  |
|    | Verified in a     | This could involve a full mission          |
|    | Relevant          | simulation in a laboratory or a            |
|    | Environment       | demonstration or test in a field setting.  |
|    |                   | Specifications and design documentation    |
|    |                   | are updated based upon lessons learned in  |
|    |                   | testing. An updated report documents       |
|    |                   | capability feasibility from safety, human  |
|    |                   | factors, and development perspectives and  |
|    |                   | summarizes what has been learned to date.  |
|    |                   | R&D organization continues research on     |
|    |                   | as-built prototype while FAA begins        |
|    |                   | acquisition program baseline definition.   |
+----+-------------------+--------------------------------------------+
| 6  | System            | Field evaluations demonstrate technical    |
|    | Demonstrated/     | functionality of prototypes, benefits, and |
|    | Validated in a    | resolution of human factors issues. FAA    |
|    | Relevant          | and research organization review           |
|    | Environment       | capability to determine its readiness to   |
|    |                   | transfer to development organization. An   |
|    |                   | acquisition strategy is required and a     |
|    |                   | development contractor is engaged.         |
+----+-------------------+--------------------------------------------+

## 4.1 TRL 1: Basic Principles Observed/Reported

TRL 1 is the stage at which an ATM concept is initially identified and
described. The appropriate development group analyzes a deficiency or
need in the NAS for which the capability may be a solution. During this
TRL phase, the following should occur: development of an initial
operational concept, completion of a trade/risk/benefit analysis, and
identification of research issues. A key concern during this stage is
the viability of the operational concept.

The Free Flight Research Program Plan notes that the research
organization proceeds relatively independently and coordinates research
activities with the FAA during TRLs 1-3 via the IAIPT.

4.2 TRL 2: Technology Concept and/or Application Formulated

TRL 2 is the stage at which a detailed research plan is developed that
provides a definition of the technical solution to the deficiency and
identifies critical feasibility issues. The plan describes activities,
schedule, likely facilities, and resources required to address research
issues in tool development. Human factors research issues including
human effectiveness are also identified in this research plan as well as
resources necessary to resolve these human factors issues.

4.3 TRL 3: Analytical/Experimental Critical Function or Characteristic
Proof-of-Concept

TRL 3 is the stage during which a conceptual prototype of the tool is
developed with initial requirements defined. Initial laboratory
evaluations may include part-task computer-human interface (CHI)
evaluations, preliminary procedures development, functionality testing,
and performance evaluations. The proof-of-concept conceptual design
should consider the relationships of roles and responsibilities with the
conceptual design and architecture.

To successfully exit from this TRL, initial research should show the
tool to be feasible from technical, benefits, safety, and human factors
perspectives based on research to date. Also, initial quantification of
technical performance metrics should reflect an improvement over the
baseline, or if improvement cannot be demonstrated, the cause is
understood and improvement is expected at some point.

4.4 TRL 4: Component and/or Integrated Components Tested in a Laboratory
Environment

TRL 4 is the stage during which a research prototype is developed and
evaluated by representative potential users. Evaluations may consist of
medium-fidelity human-system interface evaluations, procedures
evaluation, human performance evaluations, functionality testing, and
performance evaluations. The laboratory real time simulation environment
is at a higher fidelity level than at TRL 3 using standalone or
integrated components. The research organization and the FAA participate
in the laboratory and, if appropriate, site evaluations such as a shadow
mode or back room test at a field site with user teams. Exiting from
this TRL requires development and baselining of an initial FAA Concept
of Use for the capability. The cost/benefit analysis is updated, as
appropriate.

When TRL 4 evaluations are complete, the research team generates an
updated feasibility report describing technical progress, life-cycle
cost/benefits indicated, current safety and human factors status, and
issues that might require a "return to the drawing board." The updated
feasibility report is intended as an executive summary of what the team
has learned to date and should be more reflective of the operational
environment in which the capability is expected to operate.

Formal acquisition involvement begins at TRL 4 as design and
architecture requirements are refined. Assessments of procedures along
with roles and responsibilities become more robust relative to the range
of conditions afforded in the laboratory environment.

4.5 TRL 5: Components/ Subsystems Verified in a Relevant Environment

TRL 5 is the stage during which a pre-development prototype of the tool
is developed and evaluated. The evaluation environment should be at a
high fidelity such as can be achieved through a full-mission simulation
platform or through demonstration in the field with an integrated
architecture for representative normal and off-normal traffic
conditions. Based upon lessons learned, specifications and design
documentation are updated during this TRL.

Also during this TRL, the FAA focuses on activities to prepare for
acquisition. The FAA acquisition office forms user teams to address user
inputs, specifications, maintenance concepts, concept of use issues, and
human factors concerns. FAA will begin to develop contractual
documentation such as statements of work and contract data requirements
lists.

The research organization will continue on the as-built prototype system
while the FAA begins its acquisition program baseline definition. It is
FAA policy that research prototype development and evaluation should
adhere to the same fundamental paradigm as a full-scale system
acquisition that includes early user involvement (FAA, 1999a). The FAA
uses prototyping activities in this vein to assist in assessing
alternative solutions to an identified mission need.

4.6 TRL 6: System Demonstrated/ Validated in a Relevant Environment

TRL 6 is the stage at which an operational demonstration of the
pre-production prototype system is conducted in a FAA field facility if
deemed feasible and necessary. Field evaluations may include a
substantial demonstration of the prototype's functionality, and could
involve a daily use version of the pre-development operational software
application. Comprehensive human factors assessment for the prototype
capability should be completed to exit TRL 6. The research organization
focuses on completing the technology transfer and cost/benefits
activities based on data obtained during the field test. Final system
engineering documents are produced including system specifications,
interface requirements, and design descriptions. It is critical that
when operational demonstrations and field evaluations are conducted they
involve FAA operational personnel.

During TRL 6, the final high fidelity, integrated system demonstration
of the transfer prototype is accomplished, using a large variety of
traffic nominal and off-nominal conditions. Documents produced at
earlier TRLs will be finalized for formal transfer to the FAA.

5.0 Human Factors Attributes and TRLs

The role of human factors becomes increasingly complex commensurate with
the progression of a research capability through the TRLs. A summary of
human factors considerations for TRLs 1 through 6 is shown in Table 2.

Table 2: TRLs and Human Factors Summary

+----+----------------+--------------------+--------------------------+
| *  | Title          | Pipeline Output    | HF Component             |
| *T |                |                    |                          |
| RL |                |                    |                          |
| ** |                |                    |                          |
+----+----------------+--------------------+--------------------------+
| 1  | Basic          | ATM concept        | Initial identification   |
|    | Principles     | initially          | of human factors issues  |
|    | Obs            | developed and      |                          |
|    | erved/Reported | described          |                          |
+----+----------------+--------------------+--------------------------+
| 2  | Technology     | Detailed research  | Preliminary human        |
|    | Concept and/or | plan developed     | factors assessment and   |
|    | Application    |                    | research plan:           |
|    | Formulated     |                    |                          |
|    |                |                    | \- Prioritize human      |
|    |                |                    | factors research issues  |
|    |                |                    |                          |
|    |                |                    | \- Identify the          |
|    |                |                    | activities, schedule,    |
|    |                |                    | and resources required   |
|    |                |                    | to resolve identified    |
|    |                |                    | human factors issues     |
+----+----------------+--------------------+--------------------------+
| 3  | Analytical/    | Conceptual         | Address human factors    |
|    |                | prototype of tool  | issues identified in TRL |
|    | Experimental   | developed and      | 2                        |
|    | Critical       | evaluated; initial |                          |
|    | Function or    | feasibility report |                          |
|    | Characteristic | developed          |                          |
|    | Pr             |                    |                          |
|    | oof-of-Concept |                    |                          |
+----+----------------+--------------------+--------------------------+
| 4  | Component or   | Research prototype | Assess human factors     |
|    | Integrated     | developed and      | issues associated with   |
|    | Components     | evaluated; initial | the concept to show how  |
|    | Tested in a    | FAA Operational    | they have been resolved; |
|    | Laboratory     | Concept of Use     | document human factors   |
|    | Environment    | developed and      | research; update human   |
|    |                | baselined          | factors plan             |
+----+----------------+--------------------+--------------------------+
| 5  | Components/    | Pre-development    | Resolve human factors    |
|    | Subsystems     | prototype of tool  | issues from the TRL 4    |
|    | Verified in a  | developed and      | update                   |
|    | Relevant       | evaluated          |                          |
|    | Environment    |                    |                          |
+----+----------------+--------------------+--------------------------+
| 6  | System         | Operational        | Collect human factors    |
|    | Demonstrated/  | demonstration of   | data to show that all    |
|    | Validated in a | the pre-production | issues have been         |
|    | Relevant       | prototype system   | addressed and that       |
|    | Environment    |                    | operations are           |
|    |                |                    | practicable in nominal   |
|    |                |                    | and off-nominal          |
|    |                |                    | conditions               |
+----+----------------+--------------------+--------------------------+

For this research, it was the goal of the JRPD 12 research team to
assess the relevance and importance of specific human factors attributes
for each of the TRLs. To begin, a set of human factors attributes was
adapted from the FAA Human Factors Job Aid (FAA, 1999b). Developed by
AAR-100, the goal of the Job Aid was to identify and define important
human factors concerns for the acquisition management process. These
attributes denote areas where human factors issues might be identified
and be reasonably clustered together. The list of attributes was refined
for this survey to include areas that have been identified through
discussions of lessons learned from prior human factors meetings (Krois
& Rehmann, 2002) and human-system integration assessments of enhanced
capabilities (Cardosi, 2003; Della Rocco, Panjwani, Friedman-Berg,
Kopardekar, & Hah, 2003). The alphabetical listing and definition of
these human factors attributes are shown below. An asterisk (\*)
indicates those attributes that were added to the Human Factors Job Aid
list for purposes of this study.

1.  Allocation of Function\* -- System design reflecting assignment of
    those roles/functions/tasks for which the human performs better, or
    assignment to the equipment that it performs better while
    maintaining the human's awareness of the operational situation.

2.  Anthropometrics -- System design accommodation of personnel (e.g.,
    from the 1st through the 99^th^ percentile levels of human physical
    characteristics) represented in the user population.

3.  Communications and Teamwork -- System design considerations to
    enhance required user communication and teamwork.

4.  Computer Human Interface -- Standardization of computer-human
    interface to access and utilize common functions employing similar
    and effective user dialogues, interfaces, and procedures.

5.  Displays and Controls -- Design and arrangement of displays and
    controls to be consistent with the operator's and maintainer's
    natural sequence of operational actions and provide easily
    understandable supporting information.

6.  Documentation -- Preparation of user documentation and technical
    manuals in a suitable format of information presentation, at the
    appropriate reading level, easily accessible, and with the required
    degree of technical sophistication and clarity.

7.  Environment -- Accommodation of environmental factors (including
    extremes) to which equipment will be subjected and the effects of
    environmental factors on human-system performance.

8.  Functional Design -- Use of a human-centered design process to
    achieve usability objectives and compatibility of equipment design
    with operation and maintenance concepts.

9.  Human Error\* -- Examination of unsafe acts, contextual conditions,
    and supervisory and organization influences as causal factors
    contributing to degradation in human performance, and consideration
    of error tolerance, resistance, and recovery in system operation.

10. Information Presentation -- Enhancement of operator and maintainer
    performance through the use of effective and consistent labels,
    symbols, colors, terms, acronyms, abbreviations, formats, and data
    fields.

11. Information Requirements -- Availability of information needed by
    the operator and maintainer for a specific task when it is needed
    and in the appropriate sequence.

12. Input/Output Devices -- Design of input and output devices and
    methods that support performing a task quickly and accurately,
    especially critical tasks.

13. Knowledge, Skills and Abilities (KSAs)\* -- Measurement of the
    knowledge, skills, and abilities required to perform job-related
    tasks. Necessary to determine appropriate selection requirements for
    operators.

14. Procedures -- Design of operational and maintenance procedures for
    simplicity and consistency with the desired human-system interface
    functions.

15. Safety and Health -- Reduction/prevention of operator and maintainer
    exposure to safety and health hazards.

16. Situation Awareness\* -- Consideration of the ability to detect,
    understand, and project the current and future operational
    situations.

17. Skills and Tools -- Considerations to minimize the need for unique
    operator or maintainer skills, abilities, or characteristics.

18. Staffing -- Accommodation of constraints and opportunities on
    staffing levels and organizational structures.

19. Subjective Workload -- The operator's or maintainer's perceived
    effort involved in managing the operational situation.

20. Task Load\* -- Objective determination of the numbers and types of
    tasks that an operator performs.

21. Training -- Consideration of the acquisition and decay of operator
    and maintainer skills in the system design and capability to train
    users easily, and design of the training regimen to result in
    effective training.

22. Visual/Auditory Alerts -- Design of visual and auditory alerts
    (including error messages) to invoke the necessary operator and
    maintainer response to adverse and emergency situations.

23. Work Space -- Adequacy of work space for personnel and their tools
    and equipment, and sufficient space for the movements and actions
    they perform during operational and maintenance tasks under normal,
    adverse, and emergency conditions.

6.0 Method

An assessment was conducted using a rating scale developed specifically
for this study. The rating scale was based upon the Cooper Harper Rating
Scale (Cooper & Harper, 1969), designed for pilot ratings of test
aircraft handling qualities, and the Controller Acceptance Rating Scale
(CARS) designed to measure controller acceptance in decision support
tool development and testing (Lee, Kerns, Bone, & Nickelson, 2001). A
four point, even-numbered scale was selected to more effectively
differentiate the importance of the human factors attributes being
addressed by not providing a neutral or "no opinion" choice.

The rating scale that was used is shown in Figure 2. The data obtained
from the survey was used as a basis for discussion at the IAIPT Human
Factors TIM.

Figure 2: TRL Rating Scale

![](media/image4.wmf){width="6.060416666666667in"
height="3.2194444444444446in"}

Respondents were five senior human factors researchers at FAA and NASA
who are actively involved in IAIPT research activities and are familiar
with the TRL research model. They represented a total in excess of 60
years of civil aviation human factors R&D experience.

7.0 Results of TRL Human Factors Survey

Mean ratings for each of the twenty-three attributes are shown in
Appendix A. The highest ratings by survey respondents for each of the
TRLs are shown in Figure 3, which shows that the value of the highest
ratings assigned to attributes increases as the TRLs progress.

Figure 3. Highest Ratings for Attributes

![](media/image5.wmf){width="5.104861111111111in"
height="3.0104166666666665in"}

Figure 4 shows the number of human factors attributes determined to be
important to critical within each TRL. The figure shows an increasing
number of salient human factors attributes as the research capability
matures across TRLs.

Figure 4. Number of Attributes Important or Critical to TRL

![](media/image6.wmf){width="5.115277777777778in"
height="3.0215277777777776in"}

Results for each of the six TRLs are summarized in Table 3, which shows
the categorization of the human factors attributes on the basis of a
Consideration (C), Important (I), or Critical (CR).

Table 3. Categorization of Human Factors Attributes for TRLs

  --------------------- -------- -------- -------- -------- -------- --------
  **Attributes**        **TRL    **TRL    **TRL    **TRL    **TRL    **TRL
                        1**      2**      3**      4**      5**      6**

  Allocation of         C        C        I        I        CR       CR
  Function                                                           

  Anthropometrics                C        C        I        CR       CR

  Communications and             C        C        I        I        CR
  Teamwork                                                           

  Computer Human                 C        I        I        CR       CR
  Interface                                                          

  Displays and Controls          C        I        I        CR       CR

  Documentation                           C        I        CR       CR

  Environment                             C        I        I        CR

  Functional Design              I        I        I        CR       CR

  Human Error                    C        C        I        CR       CR

  Information                    C        I        I        CR       CR
  Presentation                                                       

  Information           C        I        I        CR       CR       CR
  Requirements                                                       

  Input/Output Devices           C        C        I        CR       CR

  Knowledge, Skills and                            C        I        CR
  Abilities (KSAs)                                                   

  Procedures                              C        I        CR       CR

  Safety and Health                                C        C        I

  Situation Awareness            C        C        I        I        CR

  Skills and Tools               C        C        I        I        I

  Staffing                       C        C        I        I        CR

  Subjective Workload            C        C        I        CR       CR

  Task Load                      C        C        I        CR       CR

  Training                                C        I        I        CR

  Visual/Auditory                         I        I        CR       CR
  Alerts                                                             

  Work Space                              C        I        I        I
  --------------------- -------- -------- -------- -------- -------- --------

The ratings for TRL 1 suggest that only two of the 23 attributes may be
considered but failure to consider them is not critical among other exit
criteria. Respondents showed the highest average rating for Allocation
of Function at 1.8 and Information Requirements at 1.6. The higher
priority assigned to Allocation of Function denotes its fundamental
importance to distinguishing how the capability will support the roles
and responsibilities of the human operator in a context specific to the
research capability.

Thirteen of 23 human factors attributes were rated high enough to be
considered during TRL 2. Mean ratings were higher and more human factors
attributes were included as the research to prove concept feasibility
gets underway. Attributes rated highest started to approach a rating of
3, which indicates that some are important enough to be considered
although program impact should be manageable even if the human factors
attributes are not fully addressed. The highest rated attributes were
Information Requirements at 2.6, Functional Design at 2.5, and
Allocation of Function at 2.4. CHI, Communications and Teamwork, Skills
and Tools, Staffing, and Task Load achieved mean scores of 1.8.
Anthropometrics, Information Presentation, Displays and Controls, Human
Error, and Input/Output Devices were rated at 1.6.

For TRL 3, sixteen of 23 human factors attributes were rated 2.0 or
higher, and the highest rated attributes surpassed a rating of 3.
Information Requirements edged higher (3.4) compared to its rating at
TRL 2; Allocation of Function was rated second highest at 3.2. CHI,
Information Presentation, Displays and Controls, Functional Design, and
Visual/Auditory Alerts achieved mean ratings of 2.8. Input/Output
Devices and Task Load were rated 2.4; and Documentation, Human Error,
and Subjective Workload were rated 2.2. Procedures, Situation Awareness,
Skills and Tools, and Work Space achieved mean ratings of 2.0. More
importantly, the scores indicated that the criticality of considering
these attributes increases at higher TRLs.

Twenty-one of 23 human factors attributes were considered important
enough to consider during TRL 4. Information Requirements was rated
highest at 3.5. CHI, Information Presentation, Displays and Controls,
Documentation, Functional Design, I/O devices, Visual/Auditory Alerts,
Subjective Workload, Allocation of Function and Anthropometrics achieved
mean ratings of 3.2. Human Error, Procedures, and Task Load were rated
3.0. Communications and Teamwork were rated 2.8 and another grouping
occurred at the mean score of 2.6 including Environment, Situation
Awareness, Skills and Tools, Staffing, Training, and Workspace.

Twenty-one of 23 human factors attributes were considered important
during TRL 5. Functional Design achieved a mean score of 4.0.
Information Presentation, Displays and Controls, Information
Requirements, Input/Output Devices, Procedures, Visual/Auditory Alerts,
and Subjective Workload were rated an average of 3.8. Anthropometrics,
Documentation, Human Error, Task Load, and Allocation of Function were
rated at 3.6; CHI at 3.5; and Communication/Teamwork and Situation
Awareness at 3.4. Environment, Skills and Tools, Staffing, Training, and
Workspace were rated 3.2.

By TRL 6, all human factors attributes under consideration received an
average rating higher than 3.0. Anthropometrics, CHI, Displays and
Controls, Documentation, Human Error, Procedures, Situation Awareness,
Subjective Workload, and Task Load achieved a rating of 4.0. Raters
believe that failure to consider these attributes would have major
program implications.

Communication and Teamwork, Information Presentation, Functional Design,
Information Requirements, Input/Output Devices, Training,
Visual/Auditory Alerts, and Allocation of Function achieved a mean
rating of 3.8; Environment, KSAs, and Staffing were rated at 3.6, Skills
and Tools, Workspace was rated at 3.4 and Safety and Health at 3.2.

It is apparent that by TRL 5, human factors in general should be
thoroughly integrated in product R&D. At TRL 6 all 23 human factors
attributes were rated important enough that failure to consider them
would have major program implications such as related to cost,
usability, and controller acceptance.

8.0 Discussion

Participants at the TIM generally agreed with the results of the survey.
In particular, they agreed that Allocation of Function was an extremely
important consideration that should be addressed earlier rather than
later, and should be part of a capability's concept of operations.

Human factors attributes described in this paper provide a framework for
guidance on what should be considered during the research and
development continuum and when it should be considered as shown in Table
3. This guidance is based on the professional judgment of human factors
practitioners who participated in the survey and the TIM. This guidance
poses that attributes should be either considered, are important, or
critical to particular TRLs. An attribute that is deemed important or
critical should be addressed in a manner that recognizes its program
impact and how that will be managed. Inadequate addressing of critical
attributes could have major program implications.

Transitioning research concepts from exploration to development and
onward to acquisition products should be accompanied by increasingly
detailed assessments of information requirements, display
management/integration, and human-centered automation. Assessments
should include human performance measures that assess workload,
situation awareness, communications, and human error.

Indeed, a focus on allocation of function, display integration, and
interoperability of new capabilities with existing ones will encourage
developers to consider carefully how a new capability will be used and
integrated into the overall NAS. These findings suggest that key human
factors attributes should be integral to system engineering issues early
in evolution of the research capability. That is, by TRL 3 important
decisions about the capability may likely be firming up, and should be
accompanied by robust considerations of such issues as:

-   Information requirements

-   Allocation of function

-   Computer-human interface

-   Information presentation

-   Display and controls

-   Functional design

-   Visual/auditory alerts

In addition, these findings at TRL 3 argue that FAA implementing
organizations should engage with researchers at TRL 3 rather than TRL 4
as prescribed elsewhere. This earlier engagement is notionally depicted
in Figure 5. System engineering could more broadly begin to assess and
understand how the research capability would be integrated into the NAS
and identify potential impacts such as to functional and performance
characteristics as well as other human factors interactions.

![](media/image7.wmf){width="5.99375in"
height="3.8222222222222224in"}Figure 5. Revised TRL Overview Posing
Earlier Involvement

Increasingly, interoperability studies examining the impact of multiple
decision support tools conducted by FAA have revealed a potential for
conflicting information and also highlighted the need for clarity of
roles and responsibilities between members of a controller team, and
between controllers and pilots. Operational experience with fielded
systems has shown that capabilities with their intended concepts of use
do not always fully transition to the field as intended due to such
factors as differences in operational context, variability in controller
strategies and decision-making, and individual controller acceptance
(Cardosi, 2003). Integration of the 23 attributes should provide a basis
for assessing the operational suitability of a research capability.
Suitability is a major issue in the analysis, design, and test and
evaluation of new systems and components, and should be considered
during maturation of a research product.

From a human factors standpoint, exiting one TRL and progressing to the
next should ensure that the right human factors attributes have been
addressed at the right times. The process should include such
considerations as the following:

-   Identification of issues associated with each relevant attribute

-   Assessment of issue impacts on controller performance and risk to
    achieving intended benefits of the research capability

-   Definition of issue resolution strategies and priorities

-   Tracking of closed issues especially to garner lessons learned

9.0 Conclusion

The NAS Concepts of Operations and Vision for the Future of Aviation,
upon which FAA modernization efforts are based, reflects a shift in
operator roles and responsibilities. These changes in roles and
responsibilities can potentially impact human performance, error,
workload, situational awareness, communications, training, and
procedures as new capabilities are integrated into baseline systems and
shift or add demands upon operators. Therefore, the need for ongoing
human factors research throughout all phases of research and development
is more critical than ever.

The identification of key human factors attributes and their timely
consideration during the appropriate TRL phases will help assure
achieving maximum benefits through NAS modernization. It will also help
minimize the implementation delays and costs stemming from inadequately
considered or overlooked human factors issues. TRLs provide a conceptual
framework upon which progression of concepts can be tracked in a
consistent way and human factors issues considered throughout stages of
a research product. The TRL model is built upon guidance and experience
of researchers from NASA and FAA who collaborate to ensure their shared
research provides a pipeline of new technologies and procedures for the
NAS. Thanks to the efforts of IAIPT members, the pipeline results in
concepts and prototypes that are developed, discussed, and tested for
folding into the FAA acquisition management system.

10.0 Next Steps

This study describes a further effort to integrate human factors
activities with the TRL process. There are several possible steps that
could be taken next. The human factors attributes could be further
refined such as based upon a follow-on survey to include a wider range
of human factors practitioners having experience in addressing research
across the TRLs.

Also, human factors guidance could be developed in a more applied form
for use by researchers and acquisition offices. For example, more
detailed guidance could prescribe levels of maturation for the human
factors attributes relevant to each TRL. This could support determining
the TRL level associated with adapting an existing mature commercial
capability or categorizing an existing system capability to best fit
into the TRL model.

References

Cardosi, K. (2003). *Human factors integration challenges in the
terminal radar approach control (TRACON) environment*
(DOT/FAA/AR-02/127). Cambridge, MA: John A. Volpe National
Transportation Systems Center.

Cooper, G. & Harper, R. (1969). *The use of pilot ratings in the
evaluation of aircraft handling qualities* (NASA TN-D-5153). Moffett
Field, CA: NASA Ames Research Center.

Della Rocco, P., Panjwani, G., Friedman-Berg, F. Kopardekar, P., & Hah,
S. (2003). *Collocation of User Request Evaluation Tool, Traffic
Management Advisor, and Controller Pilot Data Link Communications: An
initial human factors evaluation* (DOT/FAA/CT-TN02). Atlantic City
International Airport, NJ: William J. Hughes Technical Center.

Federal Aviation Administration (1999a). *System Prototypes in
Operational Air Traffic Control Facilities*. Washington, DC: U.S.
Department of Transportation.

Federal Aviation Administration (1999b). *Human Factors Job Aid*.
Washington, DC: U.S. Department of Transportation.

Federal Aviation Administration (2000). *Free Flight Research Program
Plan, Version 2.3*. Washington, DC: U.S. Department of Transportation.

Federal Aviation Administration and National Aeronautics and Space
Administration (2002). *Integrated Plan for Air Traffic Management
Research and Technology Development, Version 6.0, Volume 1: Executive
Summary/Overall Plan*. Washington, DC: U.S. Department of
Transportation.

Krois, P. and Rehmann, J. (2002). Integrating Human Factors in Air
Traffic Control Research and Acquisition. *Proceedings of the 21^st^
Digital Avionics Systems Conference: Air Traffic Management for
Commercial and Military Systems*.

Lee, K., Kerns, K., Bone, R., & Nickelson, M. (2001). Development and
Validation of the Controller Acceptance Rating Scale (CARS): Results of
Empirical Research. *Proceedings of the 4^th^ USA/Europe Air Traffic
Management R&D Seminar*.

Radio Technical Communications Association (2000). *National Airspace
System Concept of Operations.* Washington, D. C. RTCA.

Appendix A. Attributes Mean Ratings

![](media/image8.wmf){width="6.027083333333334in" height="3.625in"}