REDAC / NAS Ops 

Dan Herschler, ANG-C1

March 9, 2016

Air Traffic Control / Technical Operations Human Factors

BLI Number:  8BA000 (Core Program)

Review of FY 2018 Proposed Portfolio

Federal Aviation

Administration

Federal Aviation

Administration

1

ATC / Tech Ops Human Factors – Core Program Overview

Purpose:

To provide technical sponsors with timely and appropriate R&D products and consultation services that improve safety and efficiency of complex ATC systems

Methods used:

Measuring individual and team performance of air traffic controllers and technical operations specialists. 

Recommending and testing improvements to design, procedures, training, selection and placement; and mitigations to address human performance shortfalls.

2

ATC/TO Human Factors

Federal Aviation

Administration

3

What are the benefits to the FAA:

Improving the safety and efficiency of complex ATC systems by application of R&D to address factors affecting human performance in air traffic control operations and ATC system maintenance.

Recommending and testing improvements to design, procedures, training, selection and placement; and mitigations to address human performance shortfalls.

What determines program success

R&D Sponsors and Stakeholders in the ATO are able to make important workforce policy and acquisition decisions based on the results of thorough, timely, and focused R&D efforts.

When programs embrace human factors processes and requirements during system acquisition, they reduce human factors risks.  This increases the likelihood for successful system implementation and operation, while reducing the likelihood for system design and engineering rework.

ATC / Tech Ops Human Factors Benefits

ATC/TO Human Factors

Federal Aviation

Administration

3

4

ATO Sponsors:

AJG – Management Services

AJI – Safety and Technical Training

AJM – Program Management Office

ANG-C1 Program Management:

PM - Dan Herschler

Human Factors Integration Lead - Bill Kaliardos

FAA Research Performers:

FAA Civil Aerospace Medical Institute (CAMI)

Aerospace Human Factors Research Division (AAM-500)

AAM-520 – Carol Manning, Manager

FAA William J. Hughes Technical Center

Aviation Research Division (ANG-E2), Human Factors Branch 

ANG-E25 – Kenneth Allendoerfer, Manager

ATC / Tech Ops Human Factors – Core Program Team

ATC/TO Human Factors

Federal Aviation

Administration

4

ATC / Tech Ops Human Factors Focus Areas

The program addresses R&D needs within five focus areas:

Human Factors Standards

Workforce Optimization – Human Factors Efforts

Improved Safety – Human Factors Efforts

Human Factors in NAS Technology Integration

Human Performance Enhancement

The program also supports Human Factors efforts for FAA acquisition programs through ISR Checklist human factors approval responsibility, and AMS Policy updates

5

ATC/TO Human Factors

Federal Aviation

Administration

6

6

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14 | FY15 | FY16 | FY17 | FY18 |
|------|------|------|------|------|
| $0   | $0   | $0   | $0   | $0   |

Promote human factors engagement in acquisition program planning and execution 

Provide advice and guidance to support incorporation of human factors principles and requirements, and the identification and mitigation of human factors issues throughout the AMS Lifecycle.

Serve as human factors subject matter expert with signature authority on in-service review (ISR) checklist prior to the In-Service Decision milestone.

Stakeholders: ATO Project Management Office (PMO)

Coordination on AMS HF across Lines of Business

Consensus-driven HF updates to draft AMS documents

Through ISR Checklist process: confirm human factors program adequacy and facilitate identification, tracking, mitigation, and resolution of human factors issues on acquisition programs.

Convened Human Factors Acquisition Working Group (Kaliardos) to discuss planned improvements to integrate human factors into 1) AMS guidance for the early phases, and 2)  the Safety Risk Management Guidance for System Acquisitions (SRMGSA) document.

Conducted semi-annual Human Factors Reviews to communicate the accomplishments, activities, and challenges for human factors R&D and F&E activities as they support ATO and AVS initiatives and research needs.

Updated FAA Human Factors Division website https://www.hf.faa.gov to facilitate access to key information.

Provide Human Factors Support to FAA Acquisition

ANG-C1 in-house resources:Bill Kaliardos, Dan Herschler

ATC/TO Human Factors

Federal Aviation

Administration

HF.1 Human Factors Standards

Develop and update human factors standards that convey:

Human factors design requirements for new and modified air traffic control systems.

Human factors practices and requirements for contractors who plan and execute human factors efforts to meet air traffic control acquisition program requirements.

Update AMS Policy and Guidance that PMO human factors specialists will reference and apply on FAA acquisition programs.

7

ATC/TO Human Factors

Federal Aviation

Administration

8

8

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14 | FY15 | FY16 | FY17 | FY18 |
|------|------|------|------|------|
| $0   | $78K | $0   | $0   | $0   |

The purpose of this effort is to update HF-STD-004 “Requirements for a Human Factors Program” that was published in June 2009. The current standard is based on MIL-STD-46855A and as such is more aligned to military procurement than with human factors acquisition contracting in the FAA. 

Updated material addresses lessons learned and best practices from acquisition programs’ human factors efforts. The updated standard will guide the PMO in defining SOW requirements for the system acquisition contractors’ human factors program activities and deliverables.

Sponsor: Human Factors Division (ANG-C1) 

Stakeholders: AJM (Acquisition Program offices); HFAWG (Bill Kaliardos)

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Product:

Revision A to FAA-HF-STD-004 (6/2016)

Note: This effort will help the PMO to define SOW requirements for the contractor’s human factors program activities and data for FAA system acquisition programs  

Develop new data item descriptions for human factors activities that are required during system development (e.g., Early User Involvement Events [EUIEs] and Heuristic Evaluations)

Develop a new section on HF support for system fielding and deployment that addresses HF in site adaptation, system acceptance testing, coordination, in-service support, and post implementation review.

Add text to clarify responsibilities FAA and acquisition contractor team human factors personnel regarding acquisition program human factors tasks and program support activities.

1.07 Update “Requirements for a Human Factors Program” (FAA-HF-STD-004)

Research Performer:

WJHTC Aviation Research Division, Human Factors Branch, ANG-E25

ATC/TO Human Factors

Federal Aviation

Administration

HF.2 Human Factors in Workforce Optimization

Develop recommendations and guidance for ATO and FAA Academy sponsors who are responsible for policies affecting controller and technician recruitment, training, placement, staffing, and performance evaluation.

Maintain reliable and valid job/task analysis data that support application of FAA policies, and that will withstand potential legal challenges.

Upon sponsor request, conduct research to identify and recommend alternative approaches where current policies and methods result in adverse impact for employee selection, as required by EEOC Uniform Guidelines in 29 Code of Federal Regulations.

9

ATC/TO Human Factors

Federal Aviation

Administration

10

10

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 

Assess a method for improving air traffic controller training certification rates for roll out across the NAS

Sponsor: Arlenne Gonzalez (ENE-N90)

POC: Stephanie Kreseen, Program Manager, ANG-C1 Nick Lento, Manager (A), ANG-C1

Product:

A complete and vetted collection of training standards

Research report providing data on the effectiveness of the training standards

Outcomes:

Increased certification rates

Reduced time to certification

Completed initial development of training standards with TRACON SMEs

Developed and delivered Phase 1 training workshops

OJTIs and FLMs received skill enhancement and best practices training

Developmentals received learning styles and study skills training

2.04 Training Certification Standards
| FY14 | FY15  | FY16 | FY17 | FY18 |
|------|-------|------|------|------|
| $0   | $310K | $61K | $0   | $0   |

Research Performer:

National Institute of Aerospace, Evans Inc.

ATC/TO Human Factors

Federal Aviation

Administration

HF.3 Human Factors in Improved Safety

Identify and characterize safety-related aspects of human capabilities and limitations that impact performance of air traffic control and technical operations personnel in the NAS

Through analysis and research, support major FAA safety initiatives such as the Runway Safety Call to Action, by developing human factors recommendations and guidance that will improve procedures and training in operational domains where air traffic controllers and pilots interact

11

ATC/TO Human Factors

Federal Aviation

Administration

12

12

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14 | FY15 | FY16 | FY17 | FY18 |
|------|------|------|------|------|
| $0   | $0   | $70K | $0   | $0   |

Numerous safety incidents have led ATO to declare Tower Controller Visual Scanning as a Top 5 Safety issues for FY2016. 

CAMI will review existing information to identify best practices based on what is currently known about visual scanning and what has proven effective for monitoring the airport environment from the air traffic control tower. These best practices  will be confirmed in discussion with FAA SMEs.  

Additional human performance data will be obtained in a low-fidelity laboratory simulation study of controller visual scanning patterns. 

The results of these efforts will be combined to produce recommendations and guidance by September 2016 to ATO Safety (AJI-1) for tower controller visual scanning best practices.

Sponsor: ATO Safety (AJI-155)

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Product:

Develop a research plan in conjunction with AJI-155 based on a workshop with field personnel, a review of existing ATC handbook policy, field practices, incident/accident observations and lessons learned.

Conduct low-fidelity research to test some identified techniques and provide recommendations that AJI-155 can offer to the field for improved scanning

Tower Scanning was included on AJI’s list of the Top 5 ATC Safety Hazards. 

AJI-155 (Jason Demagalski) identified a list of activities on visual scanning in the tower environment that he asked AAM-520 to support. 

This is a new task in FY16. 

3.02 Tower Controller Visual Scanning

Research Performer:

CAMI Aerospace Human Factors Research Division AAM-520

ATC/TO Human Factors

Federal Aviation

Administration

HF.4 Human Factors in NAS Technology Integration

Develop methods and tools to support air traffic control system acquisition program efforts to address human factors during concept development, including prototyping and scenario evaluations.

Improve human factors laboratory capabilities using state-of-the-art human performance and human factors measurement techniques, to support studies that more robustly evaluate and predict human performance with future NAS technologies and procedures.

13

ATC/TO Human Factors

Federal Aviation

Administration

14

14

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14 | FY15 | FY16 | FY17 | FY18 |
|------|------|------|------|------|
| $5K  | $31K | $0   | $0   | $0   |

ATO Project Management Office (AJM) requested assistance in developing methods to allow acquisition programs to identify and mitigate human factors concerns before fielding of new and upgraded NAS capabilities.

ATC technologies and procedures should be subjected to HF evaluations using standardized human performance metrics prior to deployment.  New systems should be evaluated against situations as close as possible to those likely to be encountered in the field and should include off-nominal events. 

To meet this need, CAMI is developing and obtaining SME concurrence on operational scenarios that will be used in human factors testing of new NAS technologies and procedures; they also are developing recommended measures of human performance that can be used in evaluations of new ATC capabilities.

Sponsor: ATO Program Management Office (AJM-352)

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Products:

Scenario requirements database

Validated performance measures

Technical report and briefing to sponsor

Completed sector analysis

Developed 9 example scenarios

Associated scenario elements with an existing ATC task analysis.

4.01 Standardized Scenario Development and Performance Metrics

Research Performer:

CAMI Aerospace Human Factors Research Division AAM-520

ATC/TO Human Factors

Federal Aviation

Administration

HF.5 Human Performance Enhancement

Develop methods of measurement and assessment criteria supporting evaluation of air traffic controller and technician performance and application in workforce improvement policies.

Conduct focused research to identify minimum qualifications for specific air traffic control and technical operations jobs and performance in initial training.

15

ATC/TO Human Factors

Federal Aviation

Administration

16

16

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14  | FY15 | FY16 | FY17 | FY18 |
|-------|------|------|------|------|
| $160K | $0   | $0   | $0   | $0   |

The Vice President of Technical Operations (AJW) would like to implement methods to identify trainees who are likely to be successful in training and certification as an ATSS.

To meet the AJW requirement under this task, CAMI will determine whether attachment of job jeopardy to ATSS performance in the Technical Operations Common Principles course is supported by content validation.

If there is a reasonable correspondence between course content and job knowledge and skill requirements, then attachment of job jeopardy to student performance is justified. If there is little correspondence between course content and job requirements, then job jeopardy would not be justified.

Sponsors: ATO Management Services (AJG-R42)

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Products:

Recommendation to AJW on whether to attach job jeopardy to Common Principles (April 2016)

Common Principles content validation technical report (June 2016)

American Institutes for Research previously completed subject matter expert (SME) panels and a joint management/labor review panel to identify technician job tasks, knowledge, skills, abilities, and other personal characteristics (KSAOs), and tools and equipment.

Prepared a Job analysis survey data to identify: (a) the most critical or important job duties/tasks, and (b) the most important KSAOs needed on Day 1 at an SSC. 

5.10 Content Validation of ATSS Common Principles Course to Support Employment Decisions

Research Performer:

CAMI Aerospace Human Factors Research Division AAM-520, American Institutes for Research (AIR)

ATC/TO Human Factors

Federal Aviation

Administration

Anticipated Research in FY16 and FY17

Planned Research Activities (with anticipated completion dates)

Human Factors Standards:

Develop a standard for the use of color on ATC displays using a defined palette of colors which are recognizable, legible, and discriminable by controllers (FY2017)

Develop and implement an air traffic control information display and control management design strategy that incorporates best practices and lessons learned from prior and current air traffic user team activities (2019)

Expand the work on the air traffic control information display and control management design strategy to achieve agile design characteristics (2020)

Workforce Optimization:

Conduct research to provide data and targeted analyses to support data-driven decision-making at the FAA Academy Air Traffic Division, including documenting and improving the reliability of the raters who evaluate ATC student performance at the end of their initial courses (FY2017)

Assess the Radar Vectoring Aptitude Test (RVAT) as a measure of vectoring aptitude and assist in determining its predictive validity, utility, and fairness use in the placement of newly hired air traffic developmental controllers (FY2017).

17

ATC/TO Human Factors

Federal Aviation

Administration

17

Anticipated Research in FY16 and FY17

Planned Research Activities (with anticipated completion dates)

Improved Safety:

Evaluate factors related to controller recognition of  hazards in the airport traffic area (e.g., via visual scanning) and develop recommendations to improve safety in the airport environment (FY2017).

Conduct an ATO fatigue mitigation effectiveness study to determine the extent to which fatigue associated with insufficient sleep remains an issue in ATC/TO work force and whether particular work schedules are associated with greater fatigue (FY2017).

Human Factors in NAS Technology Integration:

Develop a systematic analysis method for evaluating how well  the user community is adopting and taking advantage of Air Traffic automation systems' capabilities (FY2017). 

Human Performance Enhancement:

Deliver prototype training standards and training performance measures to decrease the attrition rate at large and complex ATC facilities (FY2017).

Develop recommendations for increasing the likelihood that controller trainees will succeed in field training to ensure that trainees are not lost due to factors other than their ability to control air traffic (FY2017).

18

ATC/TO Human Factors

Federal Aviation

Administration

18

Emerging FY18 Focal Areas

Human Factors Standards:

Complete development of an empirically-validated color palette for ATC displays to ensure that displayed information is recognizable, discriminable, and legible for personnel with normal color vision and color-vision deficient personnel

Workforce Optimization:

Develop and evaluate the suitability of objective assessments of electronics knowledge and skill levels of newly hired technical operations personnel and applicants (AJW funding)

Improved Safety:

Assess the effects of shift work on personnel fatigue and develop recommended mitigations for fatigue effects on performance

Human Factors in NAS Technology Integration:

Develop recommendations for a design strategy to optimize presentation of air traffic control information on controller workstation displays, to allow the controller to quickly and accurately process the information.

Human Performance Enhancement:

Develop training performance measures and standards to reduce attrition of controllers-in-training at ATC facilities.

19

ATC/TO Human Factors

Federal Aviation

Administration

19

BACKUP SLIDES

QUAD CHARTS FOR EACH PROJECTSORTED BY YEAR OF EXPECTED COMPLETION

20

ATC/TO Human Factors

Federal Aviation

Administration

PROJECTS ENDING IN FY2016

21

ATC/TO Human Factors

Federal Aviation

Administration

22

22

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14 | FY15  | FY16 | FY17 | FY18 |
|------|-------|------|------|------|
| $0   | $208K | $0   | $0   | $0   |

The purpose of this effort is to produce revision B to the Human Factors Design Standard (FAA HF-STD-001A). The original standard was released in 2003 and needs to be revised so that its requirements address the current landscape of technologies that are likely to be used in air traffic control systems. This document is the main source of human engineering design requirements for new and modified systems that are developed or purchased COTS products.  

The updated standard will guide the PMO in defining SOW design requirements for air traffic control system acquisitions. 

Sponsor: Human Factors Division (ANG-C1) 

Stakeholders: AJM (Acquisition Program offices); HFAWG (Bill Kaliardos)

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Product:

Revision B to HFDS (6/2016)

Note: This document is the main source of human engineering design requirements for new and modified systems that are developed or purchased COTS products.  

Published interim update FAA HF-STD-001A on the FAA Technical Center Library website

Identified references that were outdated or superseded by new material

Compared automation, controls and visual indicators, and displays and printers chapters from HFDS-001 (2003) to HF-STD-001A and identified new material

1.06 Update Human Factors Design Standard (FAA-HF-STD-001A)

Research Performer:

WJHTC Aviation Research Division, Human Factors Branch, ANG-E25

ATC/TO Human Factors

Federal Aviation

Administration

23

23

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14 | FY15 | FY16 | FY17 | FY18 |
|------|------|------|------|------|
| $0   | $78K | $0   | $0   | $0   |

The purpose of this effort is to update HF-STD-004 “Requirements for a Human Factors Program” that was published in June 2009. The current standard is based on MIL-STD-46855A and as such is more aligned to military procurement than with human factors acquisition contracting in the FAA. 

Updated material addresses lessons learned and best practices from acquisition programs’ human factors efforts. The updated standard will guide the PMO in defining SOW requirements for the system acquisition contractors’ human factors program activities and deliverables.

Sponsor: Human Factors Division (ANG-C1) 

Stakeholders: AJM (Acquisition Program offices); HFAWG (Bill Kaliardos)

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Product:

Revision A to FAA-HF-STD-004 (6/2016)

Note: This effort will help the PMO to define SOW requirements for the contractor’s human factors program activities and data for FAA system acquisition programs  

Develop new data item descriptions for human factors activities that are required during system development (e.g., Early User Involvement Events [EUIEs] and Heuristic Evaluations)

Develop a new section on HF support for system fielding and deployment that addresses HF in site adaptation, system acceptance testing, coordination, in-service support, and post implementation review.

Add text to clarify responsibilities FAA and acquisition contractor team human factors personnel regarding acquisition program human factors tasks and program support activities.

1.07 Update “Requirements for a Human Factors Program” (FAA-HF-STD-004)

Research Performer:

WJHTC Aviation Research Division, Human Factors Branch, ANG-E25

ATC/TO Human Factors

Federal Aviation

Administration

24

24

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14 | FY15 | FY16 | FY17 | FY18 |
|------|------|------|------|------|
| $0   | $56K | $0   | $0   | $0   |

ATO Safety and Technical Training (AJI-1 and AJI-2) are concerned about the efficacy of controller on-the-job training at field facilities.  CAMI will evaluate two Academy courses:  Initial  On the Job Instructor (OJTI) training (course 55049) and Cadre Training (course 50331). 

CAMI will develop course improvements and a supplemental resource, The Air Traffic Instructor’s Handbook, for use in all OJTI courses and for use by OJTIs and FLMs at ATC field facilities.  Then, CAMI will evaluate the improvements and make recommendations to AJI sponsors concerning demonstrated improvements in CPC and OJTI training. 

Initial OJTI training helps Certified Professional Controllers (CPCs) learn skills for effectively conducting OJT, with emphasis on safety standards.

Sponsors: ATO Technical Training (AJI-231, 212), ATO Safety (AJI-155)

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Product:

Deliver course improvements and a supplemental resource, The Air Traffic Instructor’s Handbook, for use in all OJTI courses and for use by OJTIs and FLMs at ATC field facilities.

Developed and administered surveys to OJTIs and FLMs about topics they thought should be added to the initial OJTI training course. Reported results to sponsors. 

Collected and analyzed evaluative data from participants of OJTI Training Operational Tryout. 

The evaluation will help to determine whether the courses meet the stated objectives and demonstrate improvements in OJTI training. 

Management/NATCA disagreements may delay implementation of OJTI and OJTI Cadre training.

2.02 OJTI Training Support

Research Performer:

CAMI Aerospace Human Factors Research Division AAM-520

ATC/TO Human Factors

Federal Aviation

Administration

25

25

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14  | FY15  | FY16 | FY17 | FY18 |
|-------|-------|------|------|------|
| $668K | $626K | $0   | $0   | $0   |

AJI – Safety has identified several areas in which human factors R&D and engineering support are needed to address urgent safety needs, including defining human factors taxonomies and providing training in their application as part of formalized risk analysis processes, and identification of visual performance requirements for  air traffic control operations in the airport environment. 

Specific tasks  will support FAA efforts including :

Service Integrity and Air Traffic Control Airborne Risk Analysis Processes (SIRAP and Airborne RAP) taxonomy development, harmonization, and analyst training.

Provide guidance based on human factors analyses for virtual ATC towers, facility operations, and RNAV/RNP operations.

Sponsor: ATO Safety (AJI-121, AJI-155, AJI-1112, ANG-C5)

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Products:

Report on 2014 and 2015 SI-RAP  to-date analysis

Report summarizing ATCT Scenarios for remote tower requirements development

Report on Human Performance Assessment for Remote Towers

Completed delivery of SI-RAP training materials and course delivery to AJI and AJW personnel

Developed reporting template for SI-RAP annual aggregate analysis report

Developed scenarios based on FAA Order JO7110.65 to support development of performance requirements for remote (virtual) towers.

3.03 Human Factors and Safety Considerations in ATC Operations

Research Performer:

Fort Hill Group, LLC

ATC/TO Human Factors

Federal Aviation

Administration

26

26

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14 | FY15 | FY16 | FY17 | FY18 |
|------|------|------|------|------|
| $5K  | $31K | $0   | $0   | $0   |

ATO Project Management Office (AJM) requested assistance in developing methods to allow acquisition programs to identify and mitigate human factors concerns before fielding of new and upgraded NAS capabilities.

ATC technologies and procedures should be subjected to HF evaluations using standardized human performance metrics prior to deployment.  New systems should be evaluated against situations as close as possible to those likely to be encountered in the field and should include off-nominal events. 

To meet this need, CAMI is developing and obtaining SME concurrence on operational scenarios that will be used in human factors testing of new NAS technologies and procedures; they also are developing recommended measures of human performance that can be used in evaluations of new ATC capabilities.

Sponsor: ATO Program Management Office (AJM-352)

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Products:

Scenario requirements database

Validated performance measures

Technical report and briefing to sponsor

Completed sector analysis

Developed 9 example scenarios

Associated scenario elements with an existing ATC task analysis.

4.01 Standardized Scenario Development and Performance Metrics

Research Performer:

CAMI Aerospace Human Factors Research Division AAM-520

ATC/TO Human Factors

Federal Aviation

Administration

27

27

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14 | FY15 | FY16 | FY17 | FY18 |
|------|------|------|------|------|
| $0   | $0   | $0   | $0   | $0   |

Controller workload measurement has been a perennial challenge for the human factors community.  It is important because it has potential safety implications as well as potential impacts on airspace design, procedures, and controller scheduling and staffing.

The project is intended to improve our ability to measure workload when conducting human factors studies of new ATC technologies and procedures. Researchers are developing and testing software to efficiently screen, process, and analyze electroencephalograph (EEG) data. If the data reliably correlate with and predict controller workload and performance, such data could be used to support improvements in controller training and staffing.  

The project will culminate in a demonstration and transition to the ANG-E25 Human Factors Laboratory at the WJHTC in April 2016.

Sponsor: to be determined

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Products:

EEG processing software tool

Tutorial

Demonstration using actual study data

4.03 Applied and Translational Human Factors Research: EEG for Controller Workload Measurement

$269K in FY13

Issue:

Looking to ANG-E25 for a plan for implementation and maintenance of the EEG analysis tool as part of the laboratory infrastructure.

Research Performers:

CAMI Aerospace Human Factors Research Division AAM-520, Drexel University, University of Oklahoma

ATC/TO Human Factors

Federal Aviation

Administration

28

28

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14  | FY15 | FY16 | FY17 | FY18 |
|-------|------|------|------|------|
| $160K | $0   | $0   | $0   | $0   |

The Vice President of Technical Operations (AJW) would like to implement methods to identify trainees who are likely to be successful in training and certification as an ATSS.

To meet the AJW requirement under this task, CAMI will determine whether attachment of job jeopardy to ATSS performance in the Technical Operations Common Principles course is supported by content validation.

If there is a reasonable correspondence between course content and job knowledge and skill requirements, then attachment of job jeopardy to student performance is justified. If there is little correspondence between course content and job requirements, then job jeopardy would not be justified.

Sponsors: ATO Management Services (AJG-R42)

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Products:

Recommendation to AJW on whether to attach job jeopardy to Common Principles (April 2016)

Common Principles content validation technical report (June 2016)

American Institutes for Research previously completed subject matter expert (SME) panels and a joint management/labor review panel to identify technician job tasks, knowledge, skills, abilities, and other personal characteristics (KSAOs), and tools and equipment.

Prepared a Job analysis survey data to identify: (a) the most critical or important job duties/tasks, and (b) the most important KSAOs needed on Day 1 at an SSC. 

5.10 Content Validation of ATSS Common Principles Course to Support Employment Decisions

Research Performer:

CAMI Aerospace Human Factors Research Division AAM-520, American Institutes for Research (AIR)

ATC/TO Human Factors

Federal Aviation

Administration

PROJECTS ENDING IN FY2017

29

ATC/TO Human Factors

Federal Aviation

Administration

30

30

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14 | FY15  | FY16  | FY17 | FY18 |
|------|-------|-------|------|------|
| $0   | $138K | $145K | $0   | $0   |

The new standard for ATC Display Colors replaces FAA-HF-STD-002 “Baseline requirements for color use in air traffic control displays” that we determined was insufficiently detailed and did not address display requirements to address the perceptual limitations of Color Vision Deficient (CVD) air traffic controllers. 

Under this task, CAMI is to develop a color palette by applying color science practices, to ensure that ATC display colors are recognizable, discriminable, and legible for color-vision normal and CVD air traffic controllers. 

CAMI will update the draft ATC display color standard (completed December 2015) to include the new color palette.

Sponsor: ATO Program Management Office (AJM-352) 

Stakeholder: HFAWG (Bill Kaliardos)

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Product:

FAA-HF-STD-010, replacing FAA-HF-STD-002 (4/26/2007)

Note: This effort will help the PMO to define SOW requirements for the contractor’s human factors program regarding color display requirements for ATC system acquisition programs  

Completed draft FAA HF-STD-010.  Awaiting guidance from FAA General Counsel on intellectual property considerations.

FAA technical experts have evaluated color palette work completed at Embry Riddle Aeronautical University (ERAU) in 2015, and determined that ERAU tested an inadequate number CVD individuals.  There were other problems noted: 1) we have little confidence in the chromaticities that ERAU claimed they used and 2) the ERAU color palette was less than optimal for CVDs.  

1.05 Develop ATC Display Color Standard and Color Palette (FAA-HF-STD-010)

Research Performer:

CAMI Aerospace Human Factors Research Division AAM-520 and consultants

ATC/TO Human Factors

Federal Aviation

Administration

31

31

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 

Assess a method for improving air traffic controller training certification rates for roll out across the NAS

Sponsor: Arlenne Gonzalez (ENE-N90)

POC: Stephanie Kreseen, Program Manager, ANG-C1 Nick Lento, Manager (A), ANG-C1

Product:

A complete and vetted collection of training standards

Research report providing data on the effectiveness of the training standards

Outcomes:

Increased certification rates

Reduced time to certification

Completed initial development of training standards with TRACON SMEs

Developed and delivered Phase 1 training workshops

OJTIs and FLMs received skill enhancement and best practices training

Developmentals received learning styles and study skills training

2.04 Training Certification Standards
| FY14 | FY15  | FY16 | FY17 | FY18 |
|------|-------|------|------|------|
| $0   | $310K | $61K | $0   | $0   |

Research Performer:

National Institute of Aerospace, Evans Inc.

ATC/TO Human Factors

Federal Aviation

Administration

32

32

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14 | FY15 | FY16 | FY17 | FY18 |
|------|------|------|------|------|
| $0   | $0   | $0   | $0   | $0   |

Success in as a developmental air traffic controller requires aptitudes that match the requirements of the job. To improve use of limited training resources, ATO Technical Training would like to institute a radar vectoring aptitude test for new-hire controllers.

CAMI will be assessing the validity, utility, and fairness of a recently developed Radar Vectoring Aptitude Test (RVAT) to evaluate it for potential use in the job placement of newly hired air traffic developmental controllers.

The RVAT tool was initially designed by MITRE for AJI, to train radar vectoring skills, however on this project, CAMI researchers  are evaluating its potential use for evaluating aptitudes as part of job placement decisions. This year, the research project will be planned in detail.

Sponsor: ATO Technical Training (AJI-2)

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Product:

Develop a research plan in conjunction with MITRE, AJI, and FAA Academy.

The RVAT tool was initially designed by MITRE for AJI, to train radar vectoring skills. 

This is a new task in FY16. 

In FY2017, CAMI’s evaluation will address the tool’s predictive validity, utility, and fairness , allowing CAMI to make a recommendation to AJI-2 regarding its use as a job placement tool.

2.08 Evaluation of the MITRE Radar Vectoring Aptitude Test (RVAT) as a Job Placement Method

Research Performer:

CAMI Aerospace Human Factors Research Division AAM-520, collaborating with MITRE CAASD

ATC/TO Human Factors

Federal Aviation

Administration

33

33

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14 | FY15 | FY16 | FY17 | FY18 |
|------|------|------|------|------|
| $0   | $0   | $70K | $0   | $0   |

Numerous safety incidents have led ATO to declare Tower Controller Visual Scanning as a Top 5 Safety issues for FY2016. 

CAMI will review existing information to identify best practices based on what is currently known about visual scanning and what has proven effective for monitoring the airport environment from the air traffic control tower. These best practices  will be confirmed in discussion with FAA SMEs.  

Additional human performance data will be obtained in a low-fidelity laboratory simulation study of controller visual scanning patterns. 

The results of these efforts will be combined to produce recommendations and guidance by September 2016 to ATO Safety (AJI-1) for tower controller visual scanning best practices.

Sponsor: ATO Safety (AJI-155)

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Product:

Develop a research plan in conjunction with AJI-155 based on a workshop with field personnel, a review of existing ATC handbook policy, field practices, incident/accident observations and lessons learned.

Conduct low-fidelity research to test some identified techniques and provide recommendations that AJI-155 can offer to the field for improved scanning

Tower Scanning was included on AJI’s list of the Top 5 ATC Safety Hazards. 

AJI-155 (Jason Demagalski) identified a list of activities on visual scanning in the tower environment that he asked AAM-520 to support. 

This is a new task in FY16. 

3.02 Tower Controller Visual Scanning

Research Performer:

CAMI Aerospace Human Factors Research Division AAM-520

ATC/TO Human Factors

Federal Aviation

Administration

34

34

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14 | FY15 | FY16 | FY17 | FY18 |
|------|------|------|------|------|
| $0   | $3K  | $0   | $0   | $0   |

ATO Management Services has requested research to better understand why air traffic control specialists (ATCSs) fail field qualification training.   Developmentals, recently in training, will be surveyed.  

CAMI is administering surveys to unsuccessful developmentals who were reassigned to a lower level facility after failing training at the first facility.  Eventually, other developmentals, both those who failed and left the FAA  and those who passed, will be surveyed. 

The National Employee Services Team (NEST, http://nwp.natca.net/nest.php) receives the outputs and will use the results to determine if interventions or policy changes to improve facility success rates are needed.

Sponsor: AJG-R41

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Products:

Preliminary assessment of the data collected through 2016 for the successful and unsuccessful developmentals

The project focus is on non-cognitive factors  (e.g., management, training program, facility location, and/or facility culture) as perceived barriers to success. 

Researchers administer the Controller Transfer Questionnaire (CTQ) to trainees who come to the Academy to retrain in tower after failing field training at en route and higher level TRACON facilities. 

NATCA coordination is in process for giving a similar questionnaire to developmentals who fail and leave the FAA and developmentals who pass and become CPCs. 

5.07 Understanding Why Some Developmental Controllers Fail to Succeed

Research Performer:

CAMI Aerospace Human Factors Research Division AAM-520

ATC/TO Human Factors

Federal Aviation

Administration

35

35

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14 | FY15 | FY16 | FY17 | FY18 |
|------|------|------|------|------|
| $0   | $0   | $25K | $0   | $0   |

FAA has invested substantial time and money in conducting job/task analyses (JTAs) for mission-critical occupations such as Air Traffic Control Specialist, but the JTAs were printed (and sometimes electronic documents) with limited distribution. The web-based JAIdB will ensure that the JTA data are accessible to FAA stakeholders who use the data.

CAMI will develop a web-based database for capturing JTA data, providing a single point of access to those data for the FAA’s HF and training communities of interest within the FAA. 

The intent is to transition management of the JAIdB to ATO Technical Training (AJI-2).

Sponsors: Human Resource Services (AHF-10), ATO Technical Training (AJI-213)

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Products:

Cloud-based job analysis database (August 2016)

Includes controller (ATCS), technician (ATSS), and operations research analyst (ORA) job analysis data 

American Institutes for Research (AIR) hosted a workshop on cognitive task analysis in FY2015 and delivered recommendations on how to incorporate cognitive task analysis information into the Job Analysis Information Database (JAIdB).

AIR Version 2.0 of the JAIdB in FY2015.

Work is underway to populate the database with job analysis information.

JAIdB will be installed  and configured on the FAA Cloud in FY2016. This will provide easy access to all potential users.

5.09 Job Analysis Information Database (JAIdB) Enhancement

$353K in FY13

Research Performer:

CAMI Aerospace Human Factors Research Division AAM-520

ATC/TO Human Factors

Federal Aviation

Administration

PROJECTS ENDING AFTER FY2017

36

ATC/TO Human Factors

Federal Aviation

Administration

37

37

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14 | FY15 | FY16 | FY17 | FY18 |
|------|------|------|------|------|
| $46K | $67K | $85K | $90  | $95  |

This is an infrastructure project in support of the FAA Academy and ATO Management Services (AJG-R41) that provides a data repository for tracking ATC student training performance.

The database allows CAMI human factors scientists to answer research questions posed by ATO sponsors using data such as: individual attributes (aptitude and experience); training performance criteria; and other outcome measures. 

This research supports evaluation of proposed strategies, tests, or interventions in training, placement, and/or promotion of ATCSs, and supports data-driven decision-making by the ATO.

Sponsor: ATO Management Services (AJG-R41)

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Product:

This research supports evaluation of proposed strategies, tests, or interventions in training, placement, and/or promotion of ATCSs.  

Established Memoranda of Understanding in previous years with multiple offices to allow CAMI to collect and match various types of data for ATCS developmentals.

Software and procedures were developed to process, review, and match the data. 

Used data from database to assess training outcomes when controllers who were unsuccessful in training at their first assigned facility were allowed to transfer to a lower-level ATC facility. 

2.03 Longitudinal Database of ATCS Training and Performance: Predictors of Training Success

Research Performer:

CAMI Aerospace Human Factors Research Division AAM-520

ATC/TO Human Factors

Federal Aviation

Administration

38

38

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14 | FY15 | FY16 | FY17 | FY18 |
|------|------|------|------|------|
| $0   | $0   | $0   | $0   | $0   |

FAA Academy Air Traffic Division requires approaches for improving the rating of developmental air traffic control specialists (ATC students) that are based on scientific data and state-of-the-art (best) practices.

FY2016 task: Help the Academy document and improve the reliability of the raters who evaluate ATC student performance at the end of their Initial courses. 

Provide expertise and training on the psychological biases and errors that can impact performance ratings. Incorporate best practices in this area. 

Analyze practice ratings made during the training program to document their reliability. 

Sponsor: ATO Technical Training (AJI-2)

Stakeholders: FAA Academy (AMA-1, AMA-500)

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Product:

Help the Academy document and improve the reliability of the raters who evaluate ATC student performance at the end of their Initial courses. 

New project in FY2016

2.07 Data-Driven Decision Support for FAA Academy Air Traffic Division

Research Performer:

CAMI Aerospace Human Factors Research Division AAM-520

ATC/TO Human Factors

Federal Aviation

Administration

39

39

FY 14/15 Accomplishment / Issues

Outputs/Outcomes

Research Requirement

Out Year Funding Requirements 
| FY14 | FY15 | FY16 | FY17          | FY18 |
|------|------|------|---------------|------|
| $0   | $0   | $0   | $500K AJW Ops | $0   |

The Technical Operations organization (AJW) is seeking methods to improve recruitment and selection of qualified applicants for the job of Airway Transportation Systems Specialist (ATSS).

Develop an objective assessment of the electronics knowledge and skills that are required at the time of hire and are essential to the performance of critical and/or important job duties and tasks. 

The proposed assessment (a test battery delivered via the Internet) would provide objective verification of the applicant’s claimed experience and expertise in electronics.

In this first project phase, CAMI will define the requirements for a contractor to develop a suitable test battery, culminating in a contract SOW.

Sponsor: ATO Management Services (AJG-R42)

POC: Dan Herschler, ATC/TO HF BLI Program Manager, ANG-C1 

Products:

Concept of Operations for pre-hire testing of electronics knowledge and skill

ATSS knowledge and skill test feasibility evaluation report and briefing

SOW for test battery development (for procurement)

5.05 Develop a Pre-hire Assessment of ATSS Applicants’ Electronics Technical Knowledge and Skills

$269K in FY13

New research task – no previous activity

Funding issue:  

The initial estimate for test battery development is $500K and the BLI contract funding level is much less than this ($0K in FY16 and $230K in FY17 for all requirements). AJW may fund this with Ops $.

Contracting plan for test battery:

identifying existing and available assessments of the electronics Knowledge and Skills (K&S) identified through job analysis as essential to the performance of critical and/or important job tasks and required at the time of hire, 

adapting those tests for internet delivery (which might require as much as a year), and 

pilot- and stress-testing the test battery. 

Research Performers:

CAMI Aerospace Human Factors Research Division AAM-520

ATC/TO Human Factors

Federal Aviation

Administration

# image2.png

# image3.jpeg

# image4.png

# image5.png

# image6.jpeg

# image7.jpeg

# image8.jpeg

# image9.jpeg

# image10.png

# image11.jpeg

# image12.jpeg

# image13.png

# image14.png

# image15.jpeg

# image16.jpeg

# image17.jpeg

# image18.jpeg

# image19.jpeg

# image20.png

# image1.jpeg