Subcommittee on Human Factors WINTER/SPRING 2016 | MINUTES

Meeting date | March 29-31, 2016

Meeting location | Mike Monroney Aeronautical Center (HQ-Building 1 Rm 352)- 6500 S. 

MacArthur Blvd. Oklahoma City, OK 73169

Purpose: Review the R&D portfolio developed based on their strategic guidance from fall 

meeting and UAS focus.

Tuesday, March 29, 2016 (Subcommittee Meeting on Human Factors)

Jack Blackhurst and Sherry Chappell welcomed the Subcommittee members and thanked CAMI 

for hosting.

Jack briefed that this meeting will be UAS-focused. Need to have assessments and not 

necessarily findings on UAS. 

Introductions around the room and on the phone.

Shelley Yak, Director of William J. Hughes Technical Center and REDAC Co-Chair since 

January 2016. 

• Participating in all the REDAC Subcommittee meetings to see what is working or not.

• Use NARP as communication to congress. NARP will put the vision forward.

Jack briefed that HF is fragmented across FAA so the story isn’t getting to higher levels. People 

have trouble advocating for HF. The subcommittee is here to be advocates to tell the story to the 

FAA, OMB, etc.

Presentation HF Budget Impact and Implications | Presenter Mike Gallivan, RE&D Financial 

Manager & Mark Orr, AVS R&D Manager

Mike Gallivan, Manager of RE&D Financial Management, reviewed the FY16-FY18 RE&D 

budget status and portfolio and out year targets. 

• Current Authorization extended through Mar 31, 2016 - expect another extension

• Budget future – funding levels after FY17 are unclear, sequestration may resume.

Mark Orr, AVS R&D Manager, presented A11G HF funding history FY13-FY16.

• In FY 15 congress got in the mix and flight deck core research went from 5.2 to 2.1 & in 

FY16 from 6.1 to 1.0 ($967K)

• Congress came in FY15 & FY16. Spread A11L for UAS A11G control account to fill in 

the gap

• Have high priority programs that are not being funded because the total already hit the 

Mendoza line.

Presentation Review of HF activities/Emerging Issues | Presenter Nick Lento, ANG-C1 Acting 

Division Manager

Nick Lento introduced himself. 

Human Factors REDAC Subcommittee March 28-31, 2016 1
Working with Jason Demagalski and Mark Orr to get their tasks completed in a timely manner 

and also get them funded appropriately.

• Reviewed current programs against subcommittee list of priorities

• There are currently four BLIs (8BA000, 8AA000, 1A07A0 and 111110). FY17 has only 

three (8BA000, 8AA000, and 111110)

Presentation Human Factors Aspects of Manned Commercial Space Transportation | Presenter

Dr. Melchor Antunano, Director of CAMI

Dr. Antunano presented on the Aerospace Transportation Technologies and Human Support 

Technologies.

• The FAA is licensing the operation and not the spacecraft.

• Commercial space is pushing flexibility of space access

• We want to promote the development of space travel and the safety of space travel

Presentation FY18 Requirements ATC/Tech Ops & ATC/Tech Ops Strategic R&D Plan| 

Presenter Jason Demagalski, AJI Human Performance

Jason Demagalski reviewed Human Performance definition and four process areas:

1. ATO Human Performance Strategic Activities

2. External Human Performance Leadership

3. ATO Operations Optimization

4. Operational Human Performance Education & Training

Jason presented response to F&R Winter 2015_#5.

Held roundtable with active sponsors – pilot meeting on Feb 16, 2016

The subcommittee agreed that Jason give update at next meeting and address the system 

engineering aspect. Close current F&R and open a new action item.

Presentation HF work focusing on design, adaption, and implementation of PBN procedures for 

PBN - F&R Winter 2015 | Presenter Stephanie Kreseen, ANG-C1

• Discussion about reduction in HF flight deck core money – need to explore why we’ve 

been ‘downgraded’ by congress – we’re taking the biggest hit in the shift to UAS

• We are using antiquated training methods and approaches for measuring the effectiveness 

of the training – no one is addressing the human performance aspect 

Subcommittee agreed to close F&R #4 but the recommendation has not been satisfied.

Subcommittee agreed to close Action #7 - Summer 2015_05 and #8 - Summer 2015_06.

Presentation Updates on NASA’s Program for Data Mining of Operational Data and 

Identification of unique anomalies | Presenter Jessica Nowinski, NASA Ames

Action Item #4 - Summer 2015_02

Human Factors REDAC Subcommittee March 28-31, 2016 2
NASA Data Mining Efforts:

• Anomaly detection

o Current methods of identifying issues rely on exceedance-based methods. You 

don’t need to have background knowledge of standard operations to be able to 

identify the anomalies. Can’t identify unknown anomalies.

o Have low false positive rate (miss anomalies), but also high false negative rate 

(not really of interest)

• Precursor identification

o E.g., precursors of go-around. The unstable approach that leads to go-around. 

Precursor index to predict go-around.

o Look at combination of variables for a number of anomalous flights

o By comparing many records of go-around, the most common preceding events 

/precursors are unique. They make sure it is not common to many flights. Looking 

at precursor, you need a number of records. 

• Text mining

Wrap up – Homework Assignments – Review of Action Items

Reviewed presented briefings and asked for subcommittee further feedback.

Meeting adjourned 4:13pm

Wednesday, March 30, 2016

Review of Previous Day – Findings and Recommendations Discussion

The Subcommittee reviewed the findings and recommendations from the previous day’s 

presentation.

Jack reviewed drafted F&Rs:

• HF priorities and research and application investment

• HF research budget changes in FY16 and FY17, specific impacts of FY17 budget 

reductions

Alan reviewed draft F&R and action item:

• Progress of Human Performance efforts and roundtable process at next meeting

Drafted F&R hard copies were distributed to attendees for further review and edits.

Presentation Review of FY18 Flight Deck Core & NextGen| Presenter Kathy Abbott, Chief 

Scientific and Technical Scientific Advisor, Flight Deck Human Factors

Reviewed the FY18 NextGen Flight Deck Research Program

Human Factors REDAC Subcommittee March 28-31, 2016 3
• NextGen HF Guidelines for Advanced Instrument Procedure Design and Use 

(A12B.HFNG.2):

• NextGen Procedures, tasks, Skills and Training for NextGen Air Carrier Pilots and 

Dispatchers (A12B.HFNG.3):

• NextGen FD Systems – Flight crew Interfaces, Installation, Integration, and Operations 

(A12B.HFNG.4):

• NextGen: DataComm HF R&D (A12B.HFNG.5)

• NextGen Human Error and Complex Systems (A12B.HFNG.6)

Reviewed the Flight Deck Core FY18 Plan

• Avionics & New Technologies (A11G.HF.2)

• Advanced Vision Systems – EFVS, EVS, SVS, and DVS, HUD-Certification and 

Operational Approval Criteria (A11G.HF.4)

• Maintenance HF to Support Risk-Based Decision Making (RBDM) and Maintenance 

Safety Culture (A11G.HF.10)

• FY15-FY18 Core FD Research Requirements Overview

Discussion Remote Towers / Q&A | Presenter Jerome Lard 

• Look at organization and facilities and take into account set of requirements. 

• Recognize a research need on a broad spectrum. 

Presentation Air Force Human Factors Research Program-How it relates to UAS & Operational

FAA Issues | Presenter Mark Draper & Jessica Bartik, Air Force Research Lab

Action Item #5

Four technical competencies

1. Decision Making

2. Training

3. Bioeffects

4. Human-Centered Intelligence, Surveillance, and Reconnaissance

Presentation FAA UAS HF Program Research Requirements | Presenter Stephan Plishka, 

Human Factors Engineer

Stephen Plishka presented UAS Human Factors Research Requirements for FY18.

• UAS HF Control Station Design Standards (A11L.UAS.24)

• High Visual Contrast for UAS (A11L.UAS.31)

• NextGen Minimum Detect and Avoid (DAA) Display and Flight Path Information 

(A12B.UASNG.1)

• NextGen UAS Automation/Autonomy (A12B.UASNG.2)

FY16-FY19 UAS HF Research Requirements overview

• A11L – Core

• A12B – NextGen – funding not yet determined 

Human Factors REDAC Subcommittee March 28-31, 2016 4
Presentation UAS HF Research Roadmap Activities | Presenter Carla Hackworth, Division 

Manager

• Driven by resources and urgent need.

• Get SMEs together regarding UAS HF work and develop research roadmap near term 

and long term. Various groups represented in November 2015 at CAMI.

• Purpose – engage stakeholders from science, industry, and regulatory agency, review by 

the unique stakeholders.

• Created breakout groups with categories to come up with the issues that are presented. 

Each individual responsible for a category was asked for a description of what was 

discussed.

• Post-summit assignments – gather the issues and all participants were asked to provide 

feedback on the urgency and impact and importance of the issues.

• Top research issues – In the process or developing a proceeding report and findings in 

the breakout groups. Provide overall summary of items rated most important. In 

addition, look at breakout groups of their top issues.

• Provide report to Chris Swider and all attending members. This will become a way for 

his office and TCRG to know what work is priority that needs to be done.

Presentation FAA UAS Research Expo | Presenter Tom Prevot, NASA and Sheryl Chappell, 

DFO

• The concept from the expo is to bring all the REDAC subcommittee members to spend 

time going through what’s going on. The new structure was introduced with Earl 

Lawrence and their vision. Talked about working with other efforts like RTCA and how 

things are going with Section 333, and the pathfinder programs were reviewed. John 

Hansman brought out a lot of good concerns for REDAC in general. 

Presentation UAS Detect & Avoid Research | Presenter Kevin Williams, CAMI and Carolina 

Zingale, Tech Center

Kevin Williams – AAM-510 presented Detect and Avoid (DAA) Traffic Display Minimum 

Information Requirements:

• Looking at the following independent variables

o Display Symbology

o Control Station Interface

o Pilot Experience

 UAS experience

 Instrument-rated manned aircraft experience only

• There are 3 types of DAA information: suggestive, directive, & basic traffic info

• MOPS will cover flight in Class D transiting to Class A

o Because of the flight in Class D the MOPS have to cover non-cooperative 

intruders

• Traffic avoidance – divided into two diff phases. Close up– collision avoidance and 

separation – includes negotiations between ATC, UAS pilot and other traffic to 

coordinate a maneuver.

Human Factors REDAC Subcommittee March 28-31, 2016 5
• Research is part of RTCA SC-228 – work on the separation phase, no interest in 

collision avoidance. And interested in negotiation with ATC. 

• Looked at four different traffic displays. There are different types of info to the pilots. 

Suggestive or Basic information. Directive is a particular maneuver. Suggestive is a 

range of maneuvers. Also, could be in between suggestive or directive. There has to be a 

pilot thought process. 

Presentation UAS Center of Excellence HF Research | Presenter Ellen Bass, Drexel University

Ellen Bass is the HF lead and on the ASSURE COE executive board. In this COE, the HF work 

is well represented.

• Studying function allocation between the automation and the pilot

• Studying control station design

• Studying pilot training and certification requirements

• Studying visual observer training and certification requirements.

Presentation NASA UAS HF Research | Presenter Jay Shively & Lisa Fern, NASA Ames

Jay Shively – UAS in the NAS. Presented series of UAS DAA experiments:

• Goal – provide data on the effect of various detect and avoid (DAA) display features with 

respect to pilot performance of the self-separation function.

• Three types of displays – informative, suggestive, directive

• Approach - Conduct a series of iterative human-in-the-loop (HITL) experiments, in a 

representative simulation environment, with different display configurations to 

objectively measure pilot performance on maintaining well clear.

o Key metrics: pilot response time, losses of well clear, severity of losses of well 

clear

o Three stimulations have been conducted

o Two Mini-HITLs have been conducted

• Draft RTCA MOPS Alerting Structure

• Consistent advantage between basic & advanced info with remaining well clear

• Saw effect of control station ease of entering maneuver info

UAS Discussion – What the subcommittee has learned about the program & what we 

recommend for future research

• Jack – Will draft the UAS F&R

Findings and Recommendations and Action Item Discussion

The Subcommittee reviewed the findings and recommendations from the day’s presentations.

Meeting adjourned 5:00pm

Human Factors REDAC Subcommittee March 28-31, 2016 6
Thursday, March 31, 2016

Presentation Review of Homework Assignments from Previous Day/F&R Discussion | 

Presenter All

Reviewed last meeting Fall/Summer 2015 F&R FAA responses.

Reviewed draft F&Rs from Wednesday’s meeting.

Thanks were given for the time and support of all the members of the Subcommittee and 

participants attending the meeting.

The FAA DFO adjourned the meeting at 10:02 am on Thursday, March 31, 2016.

A tour of the CAMI research facilities followed.

The next meeting date is September 07-09, 2016 in Washington, DC.

Committee Operations Discussion

• Fall/Summer 2016 - The Subcommittee identified week of September 6-9, 2016 for next

the Human Factors Subcommittee meeting. The location TBA (Washington DC). 

• Winter/Spring 2017 – The Subcommittee identified week of March 28-30, 2017 for next 

year’s Human Factors Subcommittee meeting. The location is NASA Ames Research 

Center (Moffett Field, CA).

Human Factors REDAC Subcommittee March 28-31, 2016 7
Provide a more detailed description of the research 

plan for maintenance human factors to the REDAC 

Human Factors Subcommittee at the next meeting and 

invite the CSTA for Maintenance Human Factors to 

the meeting to discuss the overall plan.

TBD Fall/Summer 2016 

Provide a status briefing to the REDAC Human 

Factors Subcommittee on progress of the Air Traffic 

Organization’s Human Performance efforts and 

specifically on the roundtable process at the next 

meeting. 

Jason Demagalski Fall/Summer 2016

Provide a status briefing to the REDAC Human 

Factors Subcommittee on the status and progress of 

UAS recommendations at the next subcommittee 

meeting. 

TBD Fall/Summer 2016

There are two areas that the REDAC Human Factors 

subcommittee feels the FAA should initiate research 

on. First, distance learning is already in the 

requirements, but funding has not been initiated. The 

FAA should provide a briefing to the REDAC Human 

Factors Subcommittee at the next meeting on how 

funding for this research will be incorporated in the 

portfolio. 

TBD Fall/Summer 2016

The subcommittee recommends the FAA initiate 

research on training methodologies (including required 

knowledge and skills) required for current and future 

NextGen systems. Provide a briefing to the REDAC 

Human Factors Subcommittee at the next meeting on 

their progress and how this research will be 

incorporated in the portfolio and funded. 

TBD Fall/Summer 2016 

Provide a status briefing on the Administrator’s 

initiative on risk based decision making.

TBD Fall/Summer 2016 

Action items Person responsible Deadline

Human Factors REDAC Subcommittee March 28-31, 2016 8 
Meeting Attendance

Members

Jack Blackhurst (Subcommittee Chair)

Chris Desenti

Alan Jacobsen

David McKenney

Tom Prevot 

Phil Smith

Other Attendees 

Shelley Yak, FAA Director of WJHTC 

REDAC Co-chair

Chinita Roundtree-Coleman, FAA 

REDAC 

Sheryl Chappell, FAA 

HF REDAC Subcommittee DFO

Ashley Awwad, FAA

Carla Hackworth, FAA-CAMI

Charlie Tang, FAA

Kathy Abbott, FAA 

Mark Orr, FAA

Michelle Yeh, FAA

Dan Herschler, FAA

Dan Brock, FAA 

Jason Demagalski, FAA

Jay Shively – NASA AMES

Jessica Bartik, Air Force Research Lab

Joey Jaworski, Engility Corp. 

Katrina Avers, FAA-CAMI

Kenneth Allendoerfer, FAA 

Kevin Williams, FAA-CAMI

Mark Draper, Air Force Research Lab

Melchor Antunano, FAA-CAMI

Regina Bolinger, FAA

Phi Anh Nguyen, JMA 

Stella Mollman, FAA

Stephanie Kreseen, FAA

Stephen Plishka, FAA 

Phone Attendees 

Bill Kaliardos, FAA

Carolina Zingale, FAA

Devon Kelley, FAA 

Chris Swider, FAA Ellen Bass, Drexel

Jimmy Bruno, FAA

Jessica Knowinski, NASA 

John Crowley 

Lisa Fern, NASA AMES

Mike Gallivan, FAA 

Kerin Olson

Rachel Seely, FAA

Sehchang Hah, FAA

Todd Truitt, FAA 

Vicki Ahlstrom, FAA

Human Factors REDAC Subcommittee March 28-31, 2016 9 
DAY 1 – Tuesday 29, March 2016

Time Topic Presenter

8:00am-8:30am Pick up visitor badge/register vehicle at CAMI facility

8:30am-9:00am Welcome Shelley Yak, FAA Director of William 

J. Hughes Technical Center, REDAC 

Co-Chair

9:00 am – 9:15 am Welcome / Opening comments / Introductions Sherry Chappell, DFO

Jack Blackhurst, HF Subcommittee 

Chair

9:15 am – 9:45 am Budget Update Mike Gallivan, Manager, R,E&D 

Financial Management

9:45 am – 10:00 am HF Budget Impact/Implications Mike Gallivan, Manager RE&D 

Financial Management

Mark Orr, AVS R&D Manager

10:00 am – 10:30 am Review of HF activities/Emerging Issues Nick Lento, ANG-C1 Acting Division 

Manager

10:30 am – 10:45 am Morning Break

10:45 am – 12:00 pm Human Factors Aspects of Manned Commercial Space 

Transportation Melchor Antunano, Director of CAMI

12:00pm-1:00pm Lunch

1:00pm-1:30pm FY18 Requirements ATC/TechOps Jason Demagalski, AJI Human 

Performance

1:30pm-2:00pm ATC/Tech Ops Strategic R&D Plan 

F&R Winter 2015 Finding #5

Jason Demagalski, AJI Human 

Performance

Jerome Lard, ANG-C1 Program 

Manager

2:00pm-2:30pm Findings and Recommendations Discussion All

2:30pm-2:45pm Afternoon Break

2:45pm-3:15pm HF work focusing on the design, adoption, and 

implementation of PBN procedures for PBN

F&R Winter 2015 Finding #4 Action Item #7 & 8

Stephanie Kreseen, ANG-C1 Program 

Manager

Jerome Lard, ANG-C1 Program

Manager

3:15pm-3:45pm Updates on NASA’s Program for Data Mining of 

Operational Data and Identification of unique anomalies

Action Item #4

Jessica Nowinski, NASA Ames

3:45pm-4:45pm Wrap up – Homework Assignments – Review of Action 

Items

All

Evening Networking with AP-15/Group Dinner- Toby Keith’s Bar & 

Grill

10
DAY 2 - Wednesday 30, March 2016

Time Topic Presenter

8:30 am – 9:00 am Review of Homework Assignments from 

Previous Day / Findings and Recommendations 

Discussion

All

9:00 am – 10:00 am FY18 Flight Deck Core & NextGen Kathy Abbott, Chief 

Scientific and Technical 

Scientific Advisor, Flight 

Deck Human Factors

10:00 am – 10:30 am Q&A/Finding and Recommendation Discussion All

10:30 am – 10:45 am Morning Break

10:45 am – 11:45 pm Air Force Human Factors Research ProgramHow it relates to UAS & operational FAA issues

Action Item #5

Mark Draper, Air Force 

Research Lab

11:45 am – 12:45 pm Lunch

12:45 pm-1:15pm FAA UAS HF Program Research Requirements Stephen Plishka, Human 

Factors Engineer

1:15pm-1:30pm UAS HF Research Roadmap Activities Carla Hackworth, Division 

Manager

1:30pm-1:45pm FAA UAS Research Expo Tom Prevot, NASA

Sherry Chappell, DFO

1:45 pm -2:15pm UAS Detect & Avoid Research Kevin Williams, CAMI

Carolina Zingale, Tech 

Center

2:15pm-2:45pm UAS Center of Excellence HF Research Ellen Bass, Drexel 

University

2:45pm-3:00pm Afternoon Break

3:00pm-3:30pm NASA UAS HF Research Jay Shively & Lisa Fern, 

NASA Ames

3:30pm-4:30pm UAS Discussion. Prepare a summary of UAS HF 

Research – What the subcommittee has learned 

about the program & what we recommend for 

further research

All

4:30pm-4:45pm Q&A/Findings and Recommendations 

Discussion/Wrap up-Homework AssignmentsReview of action Items

All

EVENING Dinner-TBD

Human Factors REDAC Subcommittee March 28-31, 2016 11
Time Topic Presenter

8:00am-9:45am Review of Homework Assignments from Previous Day 

/Findings and Recommendations Discussion

All

9:45am-10:00am Morning Break

10:00am-10:15am AAM-003 Meet & Greet Melchor J. Antunano, Director 

of CAMI

10:15am-1030am AAM-500 VLJ Kevin Williams, AAM-510 

Research Psychologists

10:30am-1045am AAM-500 AGARS/BGARS Kevin Williams, AAM-510 

Research Psychologists

10:45am-11:00am AAM-500 ATCARS Jerry Crutchfield, AAM-520 

Research Psychologists

11:00am-11:20am AAM-600 FlexSim David Weed, AAM-600 HF 

Research Specialist

11:20am-11:40am AAM-600 Impact Sled Rick DeWeese, AAM-630 

Research Biomedical Engineer

11:40am-12:00pm NVG Demonstration Eric Simson, AAM-400 

Training Inst (AV Physiol & 

Survival)

12:00pm AAM-003 Closing Melchor J. Antunano, Director 

of CAMI

DAY 3 - Thursday 31, March 2016

Human Factors REDAC Subcommittee March 28-31, 2016 12