1

Subcommittee on Human Factors SUMMER/FALL 2017 | MINUTES

Meeting date | August 29-30, 2017

Meeting location | MITRE, McLean, VA

Purpose: Review the R&D accomplishments for fiscal year 2017.

Note Taker: Dr. Sherry Chappell

All releasable briefings can be found at: https://redacdb.faa.gov/browse.cfm. 

Tuesday, August 29th, 2017

Welcome / Opening Comments / Introductions | Presenters Dr. Barbara Holder, HF Subcommittee Chair, 

Dr. Sherry Chappell, DFO

 Drs. Holder and Chappell welcomed the subcommittee and started introductions

 Introductions around the room and on the phone

 Special thanks to MITRE for hosting this event

 Review of comments from the full meeting: what the FAA wants to be the focus

 Cybersecurity: Emerging issue how to protect against threats, all subcommittees have been asked to 

review

 Strategic work: need for a longer-term perspective

 Integration of UAS into the NAS: working to redefine what a “partnership” means

 Identify enabling technologies that the FAA needs to know about and how to bring those about

 In this meeting, we will be reviewing the cyber security plan as well as reviewing FY17 work

Presentation REDAC and Subcommittee: Roles and Responsibilities | Presenter Chinita Roundtree-Coleman,

William J. Hughes Technical Center 

 Reminder: want to provide an overview of the purpose of the subcommittees (legislative overview)

 Advisory committee oversight allows us (FAA) to leverage combined knowledge as it pertains to 

aviation research

 The FAA Act took place around 1958

 Federal Aviation Act of 1958 – was an act of the US Congress that was signed by President 

Eisenhower that created the Federal Aviation Agency which eventually became the Federal Aviation 

Administration (FAA)

 More legislation in the 80s and 90s

 Public law 100591 November 1988, amended the FAA ACT

 http://uscode.house.gov/statutes/pl/100/591.pdf

 Public law 101508 in 1990

 Omnibus Budget Reconciliation Act of 1990 - OBRA-90; Pub.L. 101–508, 104 Stat. 1388, enacted 

November 5, 1990) is a United States statute enacted pursuant to the budget reconciliation process to 

reduce the United States federal budget deficit.

 FAA Reauthorization Bill of 1996: Re-scoped the mandates and requirements (Public Law: 104264)

 Now: 49 USC 44508- Research Advisory Committee

 Provide advice and guidance through findings and recommendations to the administrator

 Allows for a global perspective

 Looks at needs, objectives, plans, content, etc. of the agency to give feedback on strengths and 

weaknesses

 We need things that are actionable: craft language that allows the FAA to respond directly
2

 When you identify something good through a finding, an action can be to continue doing what is 

being done (positive findings and recommendations)

 When findings are negative, recommendations should be actionable and concrete

 Specificity and granularity

 Look at COEs (Center of Excellence)

 Annually review allocation of funds from an overview perspective and weigh in if necessary

 We are part of a subordinate committee to the FAA Administrator

 Subcommittees include:

 Human Factors (HF)

 Aircraft Safety (SAS)

 NAS Operations (NAS Ops)

 Environment and Energy (EE)

 Airports (ARPTS)

 REDAC is a public forum, all are welcome to come and observe in any REDAC subcommittee meeting

 Obligated by law to provide presentations, notes, etc. to the public if requested

 (Nothing will ever go out blindly, requests will be made and approved before release)

 Meetings are twice a year

 First meeting, Summer/Fall

 Second meeting, Winter/Spring

 Summer/Fall: Strategic guidance for the upcoming year through FY+3 (up to 2021 this year)

 Winter/Spring: Review portfolio developed based on strategic guidance from Fall meeting for current 

through FY+2 (through 2020 this year)

 Mark Orr: we released strategic guidance for the 2020 portfolio in Spring 17

 Guidance due September 22 for final

 We’re shifted slightly so for the next two days we will mainly cover the FY17 review

 Human Factors is a small portion of the total R&D portfolio

 Purpose of the subcommittee: we have an opportunity to provide a deep dive, and identify critical and 

overarching issues, help to develop the portfolio

 Each subcommittee has a designated federal official (DFO) (Dr. Sherry Chappell: DFO for Human Factors)

 Responsibilities include, but limited to:

 Collecting information

 Developing the output (reports)

 Meet throughout the year

 Work with the subcommittee chair

Presentation Budget Update and Brief on Recommendation 3 (assessment of priority and funding of HF 

NextGen research) | Presenter Mike Gallivan, RE&D Financial Manager

 Mike started with some history

 FY17 appropriation signed May 5, 2017

 Overview of requests vs rewarded appropriations

 He then moved on to FY18 requests and the current state of the FY18 appropriations

 House and Senate appropriations committees have passed

 NOT the full House or Senate

 Subcommittee Member: How do committees come up with a number higher than asked for?

 Answer: Review our submission, RE&D, Operations, Facilities & Equipment submission, compare to other 

programs within their jurisdiction, sometimes based on different justifications than ours

 Mike expects an Omnibus Bill, which means there’s a greater chance that we will not get between 170M 

and 179M as would be expected otherwise

 We expect that we may start the year under a continuing resolution (CR)

 We hope that the continuing resolution goes through December
3

 Having many CRs make planning more difficult

 He gave an overview of the FY18 RE&D request with draft House and Senate marks

 Overviews of the House and Senate Bills’ language by research type and deliverables

 Slightly more direction from the Senate than from the House (but pretty different language and numbers 

throughout)

 At present, we’re still working under sequestration (so the budget can’t exceed certain levels, or else a 

percentage of all Budget Line Items (BLIs) will be cut), there is currently no break down to the agency level

 FY19 level for RE&D is 150M (Mike reviewed the schedule for submission, it has already gone to OST)

 All targets through FY23 are currently at 150M (they are expected to change, with revisions expected 

sometime in January)

 FAA Reauthorization

 Currently only extends through the end of FY17

 We are awaiting congressional action, but there will likely be another CR

 Budget future:

 Uncertain (Sequestration? Reauthorization?)

Second presentation: NextGen RE&D Prioritization Process | Presenter Mike Gallivan, RE&D Financial 

Manager

 In-brief: we pay our people (salaries) first (take that out of the 150M and the rest goes to work contracts)

 We will give advice to program planning teams

 Mark Orr: in the January timeframe is when Mike will come down with the 2020 targets

 The AVS requirements are being collected

 Ranked in October

 Cost estimates will be provided by PM/Performers

 These will then go through REDAC reviews in March 2018

 To then write budget narratives in May 2018

 Mike continued the in-brief process overview

 RE&D executive board are reps for the JRC

 Unique characteristics of RE&D

 We use program planning teams (PPTs)

 Program planning teams are self-governed

 Look at what is important, what we can afford to do, what we can’t afford, what requirements exist

 Essentially the same process is completed every time (some level of continuity is preserved)

 ANG-C reviews all NextGen programs

 Ensure there isn’t any duplication of efforts (maintain synergy)

 RE&D Executive board

 10 years of history (predecessor was the G7)

 Representatives from all associations, and the administrator

 They use majority rules, but it always boils down to dollars and cents

 Subcommittee Member: implications of the fact the HF doesn’t have a PPT

 Answer: people from each subcommittee are available to represent their interests

Dr. Sherry Chappell: read related recommendation (brief on prioritization process), asked subcommittee 

to determine whether we have satisfied this recommendation (Discussion).

 Shelley Yak: Prioritization schemes are not available in a simplified way. But there are conversations that 

are had across organizations (but the management of 4 BLIs can become a little complicated).

 We have the NARP (National Aviation Research Plan) redesign, and there is work on new strategic plan 

for Human Factors

 Paul Krois: Proceeding with FY18 budget, in the next few weeks working to develop that plan for both ATC 

(Bill Kaliardos) and Flight Deck (Gina Bolinger)
4

 We do have budget profiles at the BLI level for how that work will take shape in FY18 working with 

Mike/Mark for what out years will look like.

 ANG-C1 does have multi-year plans for internal programs (NextGen has built-in requirements for this)

 Question from Subcommittee Member: all recommendations and findings go to the administrator, but do the 

folks in congress get any of these recommendations or insights when they are creating/modifying these 

budgets?

 Mike Gallivan: yes, because they’re included as part of the NARP which is available to them

 Jamie Figueroa: at budget submission time we get a request from the committee for the latest set of 

REDAC recommendations

 Follow up question: do all decision makers get REDAC recommendations?

 Paul Krois: yes.

 The subcommittee decided recommendation was satisfied

Presentation Welcome & Director’s perspective | Presenter Shelley Yak, Director of William J. Hughes 

Technical Center and REDAC Co-Chair since January 2016. 

 (No Slides)

 Will be around to answer questions, etc

Morning Break

Presentation Review of FY17 Core and NextGen Flight Deck Research Accomplishments| Presenters Dr. 

Sherry Chappell, Scientific & Technical Advisor for Human Factors, Dr. Katrina Avers, Manager, Flight Deck 

Human Factors Research, Regina Bolinger, Scientific & Technical Advisor for Human Factors, Ashley Awwad, 

Scientific & Technical Advisor for Human Factors, Carl Berntsen, Booz Allen

 (Merged presentations)

 Start with briefing of Flight Deck Maintenance System Integration Human Factors (A11G): Core BLI

 New but not too far in the future, or current issues that are ongoing that need more research

 Not currently much research in GA

 Sherry requests feedback on the format

 Mark Orr interjected to explain the genesis of this format 

 Everyone expressed dissatisfaction (or grievance) about quad charts and we’ve been trying to find a 

different way to show inform the subcommittee members

 One of the issues the Aircraft Safety Subcommittee (SAS) has (who review a ton of work, 10/12 

more BLIs)

 Last spring, we up-leveled to program areas (which roughly align to the BLI, Some BLIs don’t make 

any sense, so we separated to program areas)

 REDAC is up-front in the planning for this process to update formatting

 This format is centered on the research outputs: shows outcome, the task area, the research output, 

and the status of the current research (table form)

 The output is a research output that informs the sponsors in their work

 We’ve gone to this format in order to try to nail down the right format (more of a narrative, to make 

this easier for review)

 Extra plug for this format: in talking about the research outputs, this version aligns with the NARP

 Sherry continued the overview of all task areas from the table with help from Katrina, Gina, Ashley, and 

Carl 

 Reviewed resources: people, facilities, partnerships, and highlights
5

 Moved on to briefing of NextGen Air Ground Integration (A12B) with overview of all task areas from the 

table with help from Katrina, Gina, Ashley, and Carl 

 Reviewed resources: people, facilities, partnerships, and highlights

 Work with FedEx simulators to look at enhanced flight vision systems (EFVS) issues

 Critical elements for data-driven charts

 Look at bigger pictures, within Core and NextGen (funding covers 3-year periods)

 For the flight deck, there can be anywhere from 24 to 144 projects going at a time

 Highlights are designed to help show the big picture

 Briefing on UAS Human Factors Research (A11I)

 Overview of all task areas from the table (Ashley Awwad) not funded by ANG-C1, but managed by 

ANG-C1

 All conducted by the ASSURE COE

 All only for UAS larger than 55 lbs

 Requests for various reports from Core and NextGen outputs

Feedback on Format

 We can see that there are reports, but where are they and how do we get them?

 Mark: We’re getting more comments next week from SAS

 All of the plans are in our system somewhere (to some extent)

 Subcommittee Member: It’s difficult to get the big picture, across everything that is going on (to evaluate 

and identify gaps)

Discussion

 Obligation is to look at the current year and 3 years out

 Mark gets a couple months jump ahead so the discussion will incorporate whether there are additions, but 

basically, we need the findings and recommendations for the FY20 portfolio (but we’re out of sync now, so 

it’s actually just looking at what we’re doing anyway).

 Mark Orr:

 Last spring there was potential to input for the FY20 guidance

 Sponsors are more involved in this subcommittee than others

 SAS is very involved

 Relative to the portfolio it seems like we’re missing some of the implementation connections with the 

intent being that we could be commenting on next steps that we could see being useful

 Findings and Recommendations (F&Rs) identify areas you think we need other work, and areas that 

may be lacking

 Are more things missing? Are there next steps that aren’t included in the requirements?

 One of the issues is the big picture, and we depend on the continuity of membership on the 

subcommittees

 Shelley has proposed that the NARP redesign is the big picture that we need

 Shelley Yak:

 Challenge (questions to put forth)

 As we’re going through FY17, is that the right priority? Should we keep doing what we’re doing or 

is there something that’s a higher priority?

 Look forward, looking to 2020, what areas are we not examining that we should be?

 What do you see coming on the horizon that would require our RE&D efforts?

 Is there anything immediately that we should be doing differently?

 Paul Krois:

 Complicated picture

 In following the template that Mark and AVS provided, it does boil the project information up to a 

pretty high level, if you want to stay at that level looking across projects, that could work
6

 For the flight deck we do also have quad charts, if you need them as read-ahead materials

 Quad charts can be made available

 Maintain them on a quarterly basis

 Reviewed by Paul Fontaine, needs to have an understanding of how our work flows across time

 There are different facets to all of this, particularly because it’s one office with 4 programs, all of which 

have slightly different processes (have to really rely on the program managers to maintain a good mental 

picture of the processes and what needs to be gone through on an annual basis).

 How do we portray the information that’s unique year by year?

 We are criticized for “doing the same work every year”, how do we show the evolution and the 

added value?

 Subcommittee Member:

 We started with the briefings from the program managers, and the rest of the presentations today should 

be bigger picture things, where we can show how these projects are feeding into recommendations, 

deliverables, etc

 Support the big picture

 Mark Orr:

 Twice a year, you meet for two days to plan everything for years out

 This is part of why we’re struggling here (pieces over time, to try to make a big picture)

 Subcommittee Member:

 Specific gaps might be helpful: touch displays, function allocation, etc

 So, one is latency issues

 There’s no consensus on latency issues

 There just isn’t sufficient data (there’s a gap but it doesn’t seem to be addressed)

 Latency for UAS Comms, there are some assumptions about what the latency of comm links is in 

order for the operation to be done the way that they are seeing it, we don’t understand the latency 

issues as far as being able to control a vehicle under those circumstances (Ashley: someone is 

actually working on that, but because it’s not considered human factors we don’t have research on 

that)

 We shouldn’t just be shelving reports, even if Congress doesn’t care about them anymore

 The main point made was moving towards a big picture, “emerging issues” to drive the REDAC content

 Need continuous input from industry

Lunch

Discussion F&R | All

 Discussion of Findings & Recommendations

Presentation FAA UAS Integration Research Plan 2017-2022 | Presenters Sabrina Saunders-Hodge, Division 

Manager, UAS Research, and Kerin Olson, Aerospace Engineer AUS-300

 Section 2211: unmanned aircraft systems research and development roadmap

 We have a lot of things that get labelled “roadmap”

 Plan that Sabrina has been working on with a cross section of FAA people to prep it for congress

 Sabrina covered major sections of the plan including how the plan aligns with the NARP

 A major part of the plan includes the research approach, particularly in the area of applied research

 Sabrina gave an overview of applied research at the FAA
7

 The FAA has many partners for this plan including MITRE CAASD, CAMI, WJHTC, DoT Volpe, 

ASSURE COE, NASA, EXCOM, etc.

 Integration areas and domains cover the key challenges for safe and effective UAS ops in the NAS, 

specifically relevant for congressional staffers/bills

 The plan specifically tries to highlight ongoing research versus gaps for each of the operational capabilities 

throughout

 Would appreciate HF research for things like GUI and display standards

 Needed research is identified in the timeline in the plan, they need input as to who could do the work, what 

should be done, and whether some of it is already in progress

 Because of the nature of the FAA’s funding cycles, we rely heavily on partnerships

 Working on adjudicating comments internally, has not been publicly released yet, hoping to have it for 

Winter/Spring 2018 REDAC

 Action Item: Provide a briefing on the influence of the HF UAS research summit report on the FAA’s 

overall UAS Research Plan (TCRG-approved) and the integration with the UAS Implementation Plan. 

(Carla Hackworth/Kenneth Allendoerfer)

 Closed

Presentation Analysis on UAS ASRS Study | Presenter Dr. Kim Cardosi, Principal Technical Advisor in 

Human Factors, Volpe

 Overview of number of UAS reports submitted to ASRS by year 

 UAS also means unanticipated/undesired aircraft state so reports retrieved needed to be filtered

 Classification of a conflict 

 It had to have been described by a pilot as a near miss, near collision, closer than 500 feet, or a pilot had 

to take evasive action

 This covers all UAS (small, large, military)

 Most reports are filed by pilots of manned aircraft

 Pilot concerns, often distracting or hard to see

 Birds versus drones (birds try to get out of the way, are lighter, and don’t carry lithium batteries)

 Recommendation: People need to know more about what to expect and need to know what to do when 

things do go awry

 Require that drones have a transponder (so that they can see them on their TCAS display)

 Controller concerns are often different from the concerns voiced by pilots

 Controller recommendations were also provided

 Many of the mistakes that are made by UAS pilots are the same as are made by manned aircraft pilots

 Kim gave a final summary of recommendations for UAS operations

 Then Bill Kaliardos provided the following link to a video recording of a human factors review focused on 

UAS issues and research: https://www.faa.gov/tv/ search on “human factors review.”

Discussion: possible action item or F&R on improving UAS reporting to NASA.

Afternoon Break
8

Presentation RECs 1&2 (Technical Competencies) | Presenter Dr. Carla Hackworth, Manager, Aerospace 

Human Factors Division, AAM-500

 Populated a table to respond to the recommendation: to establish a process to define and assess the FAA’s

technical human factors competencies

 Operationally defined Lead, Leverage, and Watch approach to each competency

 The goal is to create partnerships in various research areas if we need more than what we are able to 

provide in house

 Soliciting feedback

 Work in progress, but want to know what could be better or changed

 Did we leave anything off?

 Should it be higher level?

 Comments:

 Subcommittee member: Good list, like the approach. But now that we have this list, have we identified 

gaps? Using the Watch or Leverage columns as gap identifications to fill out if we don’t have them 

already filled?

 Some might be gaps; some might be just sections that we don’t know about for sure yet

 Sherry: it would be ok to not be lead in everything, but there should be relationships to ensure 

expertise coverage

 Subcommittee member: could offer MITRE as additional leverage

 Subcommittee member: do you feel that this will be useful within the FAA to guide recruiting?

 Carla: I think that’s a tough question but maybe create partnerships where we can’t hire (ensure KSA 

coverage)

 Sherry: could be a good communication tool for within the FAA

 Subcommittee member: categories in the first column, would have expected to see something along the 

lines of expertise in automation...what was your criteria for defining the competency categories?

 Needs a complementary document with the methodology for how this was derived: they used the 

congressional definition from the reauthorization, but there’s also the Human Factors Plan (Kenneth 

and Carla), essentially describing what is Human Factors, and HF Research within the FAA, and 

what are the competencies for within the FAA

 We might need to refine more; it is by no means a final document; it’s in progress

 Action to pass this around to solicit feedback on categories, and who falls under each category

Presentation Discussion on the Cyber Security Research and Development Plan | Presenter John Lapointe, 

Portfolio Manager, Aviation Research Division

 John provided a quick overview of the Cybersecurity R&D plan with the intent of soliciting feedback

 The request was mandated by congress, as part of Section 2111 Aviation Cybersecurity

 The general objectives have all been completed

 We are mandated to collaborate with international partners

 Budget was FY18-FY22 (established July 14th of this year)

 Currently on schedule, have released V 1.0

 Wanted to make sure that the plan aligns to the NARP

 And specifically aligns to the Cybersecurity Committee (CSC) strategic plan

 Gave an overview of the CSC steering committee and 5 strategic goals

 John gave a brief description of the Cybersecurity R&D Plan Partnerships in the FAA, Government, 

Industry and Academia, and International Community

 Defined FAA Goals, research areas and FAA domains (regulatory side of the house is embedded there)

 They have attempted to align the requirements to the research areas and the goals (the plan framework)
9

 John gave a more detailed overview of the research requirements

 Security and Resiliency

 Data analytics

 Human Behavior/Human Factors

 System-wide safety assurance

 Overview of research needs: topic areas that have been brought up, but that didn’t have enough in them to 

be classified as a research area (sort of gap areas)

 Used a table to map the requirements to the FAA domains

 And gave a short overview of the planed and requested budget

 What are the next steps?

 Address comments (and continue to solicit) from recent briefings

 Would like to have an interim update by the end of the calendar year

 Might need to scrub some requirements

 Questions:

 Subcommittee member: for the Human factors work, it’s all looking like it’s reactive, and I’m 

wondering about educating the systems designers who are developing the newer systems. Education on 

how to incorporate cybersecurity into the actual system design. Is there information about what that kind 

of education or training should look like?

 DARPA (Defense Advanced Research Project Agency) is developing software development 

platforms in order to ensure that the design and beyond include the ability to build in cyber 

resiliency

 What will people do when they have bad data or a persistent attack on them?

 The best we can do so far is to survey and ask what their responses would be, would like to improve

behavioral research on risk assessment in the future

 Subcommittee member: still seems reactive, should be including proactive interventions

 Dan Herschler: TechOps insider threat? What kind of procedures, training, hiring, etc. can we 

incorporate to mitigate cybersecurity threats?

 Going to target insider data, but from a data flow as opposed to from a human behavioral approach 

 Subcommittee member: why are you looking at what pilots do now, as opposed to what they should be 

doing?

 What is the difference between misinformation and a cyber-attack, as far as how we’re researching?

 Data that might be indicative of malfunction vs maliciousness

 Subcommittee member: is there already an established set of scenarios ranked by subtlety etc., so we can 

refine research?

 They’re working on that right now

 How much impact can various types of attack actually have on the aircraft?

 Trying to tease that out, trying to elicit responses that would be appropriate in the case of a cyberattack.

 Subcommittee member: have some documentation that can be sent around, from pilots’ perspective as 

far as what might be expected with cybersecurity

 A lot of the research is defined by the technology (sometimes restricted by it as well) and a lot of the 

time, the weakest link is the pilot (hard to program people’s responses)

 Action: Provide feedback (60 days)- Feedback will go to Barbara Holder to compile
10

Presentation Fatigue Mitigation in Flight Operations | Presenter Dr. Carla Hackworth, Manager, Aerospace 

Human Factors Division, AAM-500, Dr. Katrina Avers, Manager, Flight Deck Human Factors Research, and 

Dr. Tom Nesthus, Principal Investigator

 This presentation will also be given at the Subcommittee on Aircraft Safety (SAS) REDAC meeting next 

week

 CAMI has been doing fatigue research for the last few decades

 Fatigue research for ATC, pilots, flight attendants, maintenance, and other.

 Some basic research, some applied, some behavioral, some mitigation

 Short overview of fatigue research historically from CAMI (1991-present)

 Napping isn’t the worst thing ever anymore, but you have to call it a recuperative break

 Shiftwork, etc.

 Goal is to move towards the release ATC fatigue risk management systems (FRMS) recommendations 

and become part of rulemaking instead of just research

 Tom: is working on FAR 117 (FRMS 117.7)

 Provides authorization procedures for exceeding regulations

 Carla gave a short overview of the benefits of the FRMS research

 Created a scientifically based rule

 More flexibility, more knowledge, better safety

 Lots of assessment and planning up front in the phases of FRMS authorization process

 Summary:

 CAMI fatigue research products list is broken out by the research area

 In addition to the tech reports, people like Tom are providing real-time consultation for addressing 

approval for carriers, so there is immediate impact on the NAS

 Other work, addressing issues, and creating guidance and education to ensure knowledge is readily 

available and easy to incorporate

 In September, another tech report that summarizes findings for maintenance

 Questions

 Subcommittee member: are you looking into effective real-time assessment of fatigue, and/or individual 

differences in fatigue (susceptibility and mitigation)

 Modelling: trying, but hard when it’s based on aggregate data, some movement by modelers to try to 

refine their algorithms to personalize the outcomes where possible. Not sure how far forward they 

are with that right now.

 Real time: haven’t really been collecting that kind of data, probably could, but haven’t been so far as 

part of any of the ongoing research projects, not sure where that would lead if people were trying to 

do that on their own (wearables), we’re not to the point where we could predict fatigue on a realtime basis

• There is some medical research looking at DNA markers for fatigue

• The research is more in the collaborative world than in just one place

• If they get better at algorithms, CAMI is in a good position to do more of that type of research

• Fatigue management working group (federal level)

o NHTSA, NIH, NTSB, etc.

o Next meeting focusing on countermeasures

 Subcommittee member summarizes: so CAMI is watching what’s going on but not doing any of that 

research actively right now (Tom, only Biomarker research)

 Subcommittee member: make sure you include not just number of legs, but time between, time 

sitting on the ground between flights, etc.

Wrap Up F&R Homework Assignments – Review of Action Items

 Meet here tomorrow
11

 Homework:

 Send cyber comments and thoughts to Barbara

 For FY20 the HF research plan, thoughts on emerging issues, gaps, etc.

 Doesn’t have to be driven by only what we discussed today, could be just things you already know

 Thoughts on F&Rs

End of Day 1
12

Wednesday, August 30th, 2017

Presentation: MITRE Interval Management (IM) Time-based Sequencing and Spacing (TSAS) Briefing and 

Demo @ MITRE 2 | Presenter: Randy Bone, MITRE

 Abbreviated preliminary results prior to the lab demo

 Randy gave a simulation overview

 HITL with pilots and controllers, aimed at informing the further maturation of flight deck and ground 

requirements

 Data collection was finished in May, final deliverable in January

 TSAS vs IM (absolute vs relative spacing)

 Randy gave an overview of both TSAS and IM procedures

 Precursor activities involve evaluation of three separate concepts

 Brought in controllers for feedback

 Methods

 Wanted to make sure that these tools are both feasible and acceptable

 Bill Kaliardos asked Randy to clarify absolute versus relative spacing

 TSAS uses slot markers, ATC has to get aircraft into slot markers, so absolute time is just make sure 

you’re getting to a certain spot at a certain point

 Whereas IM is just make sure you’re a certain amount of time behind a certain craft

 Independent variables for ATC

 IM awareness (situation awareness)

 Make sure controllers have sufficient information to be able to appropriately use IM

 Independent variables for Flight Deck

 CDTI IM feature set

 Make sure flight deck can use IM

 Randy quickly ran through the past flight deck IM lit review

 Need crews to trust the algorithm, need to comply

 Not knowing that the speed was limited using IM to +/- 15% of the profile speed

 Might need some sort of predictive tool to mitigate confusion

 That confusion is still present (issues with minimum)

 Overview of the study participants and procedure

 Because they used a couple different combinations of IM operations there were a lot of possible situations

 Environment overview: ATC involved feeder and final controllers, flight crew flew in the sim, 

communicating over the common frequency, and multiple pseudo-pilots

 Used IM sample algorithms, and made sure IM retrofit adhered to minimum operational performance 

standards (MOPS)

 Used prototypes for the IM tools for the flight deck

 IM Feasibility check 

 Progress indicator

 Developed a graphic progress indicator

 Used the MOPS, but was very high level so had to integrate multiple standards

 Situation awareness tool

 If flight crew doesn’t want to use IM, they can report that they can’t do IM to ATC and it will stop and go 

back to normal metering operations

 Dan Herschler wants to know whether one aircraft not using IM, would mess up the daisy chain of 

aircraft

 Randy says unless it’s really far off, it shouldn’t be a problem
13

 Randy showed some prototype TSAS tools on the STARS display, which is what they used at the time 

because it was the most up to date as well as IM displays

 Gave an overview for slot markers, speed, sequence number, advisory, runway assignment, etc.

 IM aircraft can be lead and trail

 ATC had 5 days, trained for 2 days, then pilots came in for one day only with the controllers

 Randy finished with thoughts on the preliminary results

 Controllers seem OK with using IM but aren’t totally comfortable and want info from the flight crew to 

know whether they need to intervene

 The minimum might be a little challenging, but it’s getting better 

 Working with NASA to close on the standards

 Dan: how does this tie into the equivalent visual operations function

 Randy: that’s a very controversial topic with the unions, paired-approach topic has been rewritten

 Controllers didn’t want any transfer of separation responsibility

 Subcommittee member: ongoing topic of discussion, what do we know about the complexity of the mixed 

environment?

 Randy please comment on this pairwise comparison of IM and TSAS, how difficult was it to use them 

together, how will we let these merge operationally

 Randy: this was not a TSAS experiment, we could have some recommendations, but that wasn’t what 

this was supposed to be, what we saw here was the same as before regardless of IM (we didn’t really see 

anything that would make them change TSAS)

 Highlight that a challenge in exploratory mode...without the blue slot marker it seems like this might be 

a little more complicated .... might make the slot markers behave differently in IM than in TSAS

 Subcommittee member: should this be a little more longitudinal?

 Randy: yes

 If this is implemented, it does seem to decrease workload

 They did have a few pseudo pilots accept IM, but then just maintain a high speed not actually use the IM

 This was hard to detect until the very end for the controller

 Can also save fuel

 Having just more help is a benefit

 They are collecting data on the number of speed changes that are required (environmental impact?)

 Gina Bolinger: if you look at the surface ops movement (true gate-to-gate) process that NextGen is scoping, 

if we could get to where it’s predictable everywhere, could solve the issue of getting the crews to the gates 

on time, maybe

 Theoretically dramatically increase efficiency

 As long as things go well

IM TSAS Demo in IDEA Lab

Morning Break

Presentation Human Factors in NextGen Data Communications | Presenter Dr. Kim Cardosi, Principal 

Technical Advisor in Aviation Human Factors, Volpe, Dr. Tracy Lennertz, Engineering Research Psychologist

 Kim kept the overview brief in order to address more pressing issues

 Looked at “free-text” to determine what is said so often that it needs to be included into the message set

 Recommended: Those that are needed to reduce pilot errors (some messages lead pilots into errors)

 Looked at some other best practices for flight standards

 Read-aloud method had issues, recommended that both pilots needed to read the clearance silently 

before responding
14

 Task 3 – Loss of communications

 If pilot is told to monitor a frequency, vs. contact the controller: Expect benefits in terms of workload

 Eventually these data need to be compared to Data Comm performance

 Will there be an increase in loss of communication events?

 Looked at mandatory occurrence reports

 Found that we don’t have good data on loss of communication events

 Wasn’t worth the amount of money it would cost to actually go look at the duration

 Data Comm will help fix some things

 Bottom line is that even with Data Comm, ATC can still send a contact request as backup

 Most common clearances were frequency changes and altitude (which can both be covered by Data 

Comm)

 Kim gave an overview of the current issues

 No ICAO recommended practice for reading and writing in aviation English (major issue, 

internationally)

 Continual need to identify the messages that should be used for NextGen operations

 Oftentimes we’re asked to identify what the messages should be before the concept of operations are 

figured out

 Data Comm isn’t pre-implementation anymore so NextGen money can’t go to it

 Data Comm is focused on today’s implementation

 Redefine the way that we do research in a way that ensures that it will be used and implemented after it 

is carried out

 Getting back to what we were talking about yesterday, can’t stress enough that the implementation of Data 

Comm is not going to solve the similarity of call signs problem

 More of a factor within airlines, more with four-digit flight numbers versus three, more with how we tell 

controllers to say numbers (we say seventy-two thirty-two, instead of seven two three two).

 We had a lot of progress and traction in this area, we need to educate controllers about how they say 

numbers/call signs and how that affects the way the numbers are perceived

 Dan: there are some guidelines but they don’t cover all of the issues

 Kim: a large part of the problem is self-inflicted (built-in similarity based on marketing assigning the 

call signs)

 Questions:

 One of the issues years ago was the printing of the messages...brought up because now what we’re 

doing on the ground, there is no printing in the cockpit

 There are issues with the message being on multiple pages

 Kim; fixed the problem with the messages being split over multiple pages

 Printers are not certified like the displays, they can mix things up, jam, etc.

 Is there a procedure on how these should be read (departure clearances), how does that work (read at 

same time, read one at a time, etc.)?

 Cut in interest of time

Presentation Review of FY17 ATC Core Research Accomplishments | Presenter Dan Herschler, Scientific & 

Technical Advisor for Human Factors

 Overview of research, has some supplemental information

 Improve human performance, recommend and test various improvements in ATC

 Overview of all sponsors and performers (primary orgs are CAMI and WJHTC)

 Focus areas are:

 HF Standards

 Workforce optimization

 Improved safety

 HF in NAS Technology integration
15

 Human performance enhancement

 Support the joint resources council with ISR HF checklist (can create action plans to improve HF 

integration)

 Dan gave a brief description of each of the focus areas and the projects that have been accomplished

 Subcommittee member: does the GUI style guide inform new displays?

 Mainly for TechOps as of right now, but want to have examples etc. so that it can be used broadly

 Dan needs multi-year plans in place to make sure that research is completed in a series of steps to make sure 

all nooks and crannies are done

 Subcommittee member: design standard?

 Dan: it’s in a weird gray zone, where it’s hard to make sure things are applied across programs

 Subcommittee member: is part of this work going to be the articulation of cases where this has all 

passed?

 Dan: want to fix things that have the biggest potential for issues, most critical, not just low hanging fruit

 Gave a brief overview of the FY17 requirements for ATC

 Some were withdrawn

 Resources were limited, or some sponsors changed their minds

 Some of the work he included in the supplements is ongoing

 Jason Demagalski’s group is now an external sponsor/coordinator, created the R&D Requirements 

Roundtable, but they are no longer able to support that

 Now need a different ATO sponsor that can help out

 Dan gave recommended readings

Presentation Review of FY17 ATC NextGen Research Accomplishments | Presenter Dr. Bill Kaliardos,

Human Factors Integration Lead

 Inherited this from Stephanie Kreseen

 Bill gave a short intro and background on his BLI, as well as an overview of current and recent projects

 There were requests for reports on each of the projects

 Subcommittee member: have you gone to any of the stakeholders to ask about what could be done to make 

sure that our requirements end up being used?

 Bill: that’s a huge issue, but it is part of our duties, and we do do that as much as possible

 Lastly the 4 items that are going on right now

 PBN Human Performance Metrics 

 Established on RNP (EoR) Human Factors Implementation Guidance

 Human Factors Integration Considerations of Time, Speed, and Spacing Tools

 Human Performance Considerations for Contingency Operations in a Degraded NextGen Environment

Discussion Review of Homework Assignments from Previous Day / Findings and Recommendations | All

 Discussion of Findings & Recommendations

Lunch

MITRE IDEA Lab demo (3T in the tower & RTA/Terminal in the flight deck) @ MITRE 2
16

Presentation Human Factors Research for Technical Operations | Presenter Dr. Dana Broach, Personnel 

Research Psychologist, Aerospace Human Factors Division, AAM-500

 Get to know your Tech Ops people because without them controllers are blind, deaf, and dumb

 Specialists have a legal responsibility to certify the facilities, services, & engineering (FSE)

 Threshold to get in isn’t particularly high, usually hire about 300 a year

 Common Principles is the training gateway course

 No one fails, if they did, about 20% people would be lost 

 Tech Ops president wanted to attach job jeopardy and content validation for whether it’s actually teaching 

knowledge, skills, & abilities (KSAs) used on the job

 Started with job/task analysis

 Strong legal considerations

 Dana reviewed the common principles validation plan

 Job analysis survey, mostly knowledge questions

 Data broke into two clusters:

 Core and essential, vs administrative/as required

 And went into a deep dive for the most important

 Most important:

 Integrity, these need to be trustworthy people

 Lots of most important things were not technical things

 Did a content review to assess whether to attach job jeopardy to the course (decided not to)

 Observations:

 Troubleshooting is becoming more complex, and some issues might be different based on the location or 

airport setup

 Distinction between doing the job and learning the job

 Dana covered other uses for and potential for the job analysis data

 Job Analysis Tools are not great right now, but can use the job analysis data to standardize

 Looked at future Tech Ops CONOPS

 At CAMI:

 Planning to look at how the network will impact jobs in the future

 Router errors can propagate across facilities and make them go dark

 Might see redesign of the hiring process

 Change the KSAs

 Questions:

 How do you identify the non-technical skills?

 Some are actually really hard to measure, but some are already standardized

 Dan: cybersecurity? Reliability of personnel and not repeating the Chicago event...do you have any 

quick opinions?

 Measuring some of the stuff we pointed out from the job task analysis but there’s only so much that 

we can do without being overly invasive

Presentation Analysis of potential risks with NextGen mid- term drivers | Presenter Dr. Kelley Krokos, 

American Institutes for Research

 Kelley introduced herself then launched into a background

 The goal of the project was to conduct a strategic job assessment on the job of the controller

 Needed a single word to describe the “stuff” that is NextGen

 Strategic job analysis is a future-oriented job analysis

 Talked to NextGen to identify drivers

 Talked to incumbents to identify the current job
17

 Talked to researchers

 10 unique drivers were evaluated with 19 impacts (independently or as a group)

 Impacts

 Decision making is collaborative

 Control is dynamic

 Need to know what kind of plane and what capabilities the plane has, as well as what the crew has been 

trained to do

 Change not only persists but increases

 Decision making and support are all improving

 Independence and automation

 Cumulative effects over time

 Impacts of drivers on the controller jobs

 Need to understand that it’s not everybody all the time in every facility

 Some tweaking to individual situations

 Properties of the tasks and KSAs will change significantly

 What, when, where, how do they see information, is it shared?

 Effects of tools and equipment are pretty large

 Driver induced human factors risks (potential)

 Identified 12 risks, Kelley gave an overview of each risk

 Overview of table coding for driver results

 Subcommittee member: Nice dashboards! For each of those they need to go down to the task level, can you 

drill down with those? Then can that be used to help determine how to mitigate the risk?

 Kelley: yes, this was a rational exercise not with participants but that could be done

 Dan: this should all be coming out soon, and hopefully will have follow up work

Afternoon Break

Presentation Update on NASA’s program for data mining of operational data and identification of unique 

anomalies | Presenter Dr. Jessica Nowinski, NASA Technical Advisor for Safety, Airspace Operations and 

Safety Program

 Cancelled

Presentation Brief on the ATO Human Performance Center of Excellence | Presenter Karen Callihan, 

Program Manager, Center of Excellence for Technical Training and Human Performance

 Big picture view

 Use grants and IDIQ contract vehicles

 “SOAR” is the newest COE

 On August 12th they celebrated their first birthday

 Karen gave an overview of SOAR versus other COEs, including SOAR’s mission

 SOAR general in that the specific topic it covers is broadly including across all aspects of the FAA

 16 core universities (as well as affiliate schools and industry)

 Overview of some of the research facilities etc. (majority of the university members are classified at the 

highest research level)

 Lastly, Karen presented statistics on the current funding and ongoing research areas and professions

 Subcommittee member: can you use this COE to fill some gaps?

 Dan: they have their own source of funding, but they can take any source of funds through them to reach 

any of those institutions or partners
18

Q&A/F&R Discussion / Wrap up – Homework Assignments – Review of Action Items

 Schedule for next Winter/Spring 2018 REDAC (February 27- March 1, need to decide if it’s 2 or 3 days)

End of Day 2

Action Items Status

No. ID Action Item Meeting Fall 2017 

Status

1 Summer 

2015_01 

Every meeting present updates on all FAA UAS human 

factors research projects. (Plishka/Awwad, Chappell) 

Summer/Fall 

2015 

Open 

2 Summer 

2015_02 

Every meeting present updates on NASA’s program for data 

mining of operational data and identification of unique 

anomalies. (Jessica Nowinski) 

Summer/Fall 

2015 

Closed

3 Summer 

2015_04 

FAA to brief on the Human Performance Center of 

Excellence. (Karen Callihan)

Summer/Fall 

2015 

Closed

4 Winter 2016_04 There are two areas that the REDAC Human Factors 

subcommittee feels the FAA should initiate research on. 

First, distance learning is already in the requirements, but 

funding has not been initiated. The FAA should provide a 

briefing to the REDAC Human Factors Subcommittee at the 

next meeting on how funding for this research will be 

incorporated in the portfolio. (Kathy Abbott)

Winter/Spring 

2016 

Open 

5 Winter 2016_05 The subcommittee recommends the FAA initiate research on 

training methodologies (including required knowledge and 

skills) required for current and future NextGen systems. 

Provide a briefing to the REDAC Human Factors 

Subcommittee at the next meeting on their progress and how 

this research will be incorporated in the portfolio and funded.

(Kathy Abbott)

Winter/Spring 

2016 

Open 

6 Summer 

2016_01 

Provide a briefing on the influence of the HF UAS research 

summit report on the FAA’s overall UAS Research Plan 

(TCRG-approved) and the integration with the UAS 

Implementation Plan. (Carla Hackworth/Kenneth 

Allendoerfer) 

Summer/Fall 

2017

Closed
Federal Aviation Administration

REDAC Human Factors Subcommittee

McLean, VA

Meeting Agenda, August 29-30, 2017

Location: MITRE 3, 7594 Colshire Drive, McLean, VA 22102

McLean Metro

DAY 1 – Tuesday, August 29, 2017

Please join meeting from your computer, tablet or smartphone.

https://global.gotomeeting.com/join/477921797

Time Topic Presenter

8:00 am – 8:20 am Check in @ MITRE 3

8:20 am – 8:50 am Welcome / Opening Comments / 

Introductions

Dr. Barbara Holder, HF Subcommittee 

Chair, Dr. Sherry Chappell, DFO

8:50 am – 9:00 am Welcome Shelley Yak, FAA Director of William J. 

Hughes Technical Center, REDAC Co-Chair

9:00 am – 9:10 am Meeting Legislature Requirements Chinita A. Roundtree-Coleman, REDAC 

Team Lead/Program Manager, Research 

and Development Management Division 

(E41)

9:10 am – 10:00 am Budget Update and Brief on REC 3 

(Assessment of priority and funding 

of HF NextGen research)

Mike Gallivan, Manager, RE&D Financial 

Management

10:00 am – 10:15 am Morning Break

10:15 am – 11:05 am Review of FY17 Core Flight Deck 

Research Accomplishments

Dr. Sherry Chappell, Scientific & 

Technical Advisor for Human Factors

Dr. Katrina Avers, Manager, Flight Deck

Human Factors Research

11:05 am – 11:55 am Review of FY17 NextGen Flight Deck 

Research Accomplishments

Regina Bolinger, Scientific & Technical 

Advisor for Human Factors 

Ashley Awwad, Scientific & Technical 

Advisor for Human Factors

11:55 am – 12:55 pm Lunch

12:55 pm – 1:25 pm F&R Discussion All

1:25 pm – 1:50 pm FAA UAS Integration Research Plan 

2017-2022

Sabrina Saunders-Hodge, Division 

Manager, UAS Research, and Kerin Olson, 

Aerospace Engineer AUS-300

1:50 pm – 2:15 pm Analysis on UAS ASRS study Dr. Kim Cardosi, Principal Technical 

Advisor in Aviation Human Factors, Volpe

2:15 pm – 2:35 pm RECs 1 & 2 (Technical 

Competencies)

Dr. Carla Hackworth, Manager, 

Aerospace Human Factors Division, AAM500

2:35 pm – 2:50 pm Afternoon Break

You can also dial in using your phone. Joining from a video-conferencing room or 

United States: +1 (786) 535-3211 Access system? 

Code: 477-921-797 Dial: 67.217.95.2##477921797 

Cisco devices: 477921797@67.217.95.2 
Federal Aviation Administration

REDAC Human Factors Subcommittee

McLean, VA

Meeting Agenda, August 29-30, 2017

2:50 pm – 3:20 pm Fatigue Mitigation in Flight 

Operations

Dr. Carla Hackworth, Manager, 

Aerospace Human Factors Division, AAM500

Dr. Katrina Avers, Manager, Flight Deck 

Human Factors Research

Time Topic Presenter

3:20 pm – 4:05 pm Discussion on the Cyber Security 

Research and Development Plan

John Lapointe, Portfolio Manager, 

Aviation Research Division

4:05 pm – 4:30 pm Human Factors in NextGen Data 

Communications

Dr. Kim Cardosi, Principal Technical 

Advisor in Aviation Human Factors, Volpe

Dr. Tracy Lennertz, Engineering 

Research Psychologist

4:30 pm – 5:00 pm Wrap up – F&R Homework 

Assignments – Review of Action 

Items

All

EVENING Group Dinner

DAY 2 – Wednesday, August 30, 2017

Please join meeting from your computer, tablet or smartphone.

https://global.gotomeeting.com/join/477921797

You can also dial in using your phone.

United States: +1 (786) 535-3211 

Access Code: 477-921-797

Joining from a video-conferencing room or 

system?

Dial: 67.217.95.2##477921797 

Cisco devices: 477921797@67.217.95.2

Time Topic Presenter

7:45 am – 8:00 am Check in @ MITRE 2

8:00 am – 10:00 am MITRE IM TSAS Briefing and Demo @ 

MITRE 2

Randy Bone, MITRE

10:00 am – 10:10 am Morning Break

10:10 am – 10:30 am Review of Homework Assignments from 

Previous Day / Findings and 

Recommendations Discussion

All

10:30 am – 11:00 am Review of FY17 ATC Core Research 

Accomplishments

Dan Herschler, Scientific & 

Technical Advisor for Human Factors

11:00 am – 11:30 am Review of FY17 ATC NextGen Research 

Accomplishments

Dr. Bill Kaliardos,

Human Factors Integration Lead

11:30 am – 12:00 pm F&R Discussion All

12:00 pm – 1:00 pm Lunch

1:00 pm – 2:00 pm MITRE IDEA Lab demo (3T in the tower & 

RTA/Terminal in the flight deck) @ 

MITRE 2

TBD

2:00 pm – 2:30 pm Human Factors Research for Technical 

Operations

Dr. Dana Broach, Personnel 

Research Psychologist, Aerospace 

Human Factors Division, AAM-500
Federal Aviation Administration

REDAC Human Factors Subcommittee

McLean, VA

Meeting Agenda, August 29-30, 2017

Time Topic Presenter

2:30 pm – 3:00 pm Analysis of potential risks with NextGen 

mid- term drivers

Dr. Kelley Krokos, American 

Institutes for Research

3:00 pm – 3:15 pm Afternoon Break

3:15 pm – 3:40 pm Update on NASA’s program for data 

mining of operational data and 

identification of unique anomalies

Dr. Jessica Nowinski, NASA 

Technical Advisor for Safety, Airspace 

Operations and Safety Program

3:40 pm – 4:05 pm Brief on the ATO Human Performance 

Center of Excellence

Karen Callihan, Program Manager, 

Center of Excellence for Technical 

Training and Human Performance

4:05 pm – 4:40 pm Q&A/F&R Discussion / Wrap up –

Homework Assignments – Review of 

Action Items

All

EVENING Group Happy Hour
Research, Engineering and Development Advisory Committee

Human Factors Subcommittee

August 29-30, 2017

Attendees List

August 29, 2017

Name Affiliation

Barbara Holder Honeywell

Alan Jacobsen Boeing

Paul Krois FAA

Phil Smith OSU

Kim Cardosi DOT/VOLPE

Mark S. Orr FAA/AVS

Richard Mendell FAA

Mike Gallivar FAA

Bridget Lewis Humanproof

Jon Jonsson Hi-Tec Systems

Jerry Craft FAA

Regina Bolinger FAA

Dan Herscher FAA

Kevin Comstock ALPA

Bill Kaliardos FAA

Tim Evans FAA

Jimmy Bruno FAA

Kenneth Allendoerfer FAA

Edward Austrian Fort Hill Group

Carl Bearnsten Booz Allen

Ryan Blondino FAA

Monique Moore FAA

Evan Harvey Humanproof

Ashley Awwad FAA

Sherry Chappell FAA

Shelley Yak FAA

Jaime Figueroa FAA

Chinita Roundtree-Coleman FAA

Teleconference – August, 29th, 2017

Name Affiliation

Carla Hackworth FAA/CAMI

Gregory Wu FAA

Eric Neiderman FAA

John LaPointe FAA

Chuck Agava FAA

Isidore Venetos FAA
Research, Engineering and Development Advisory Committee

Human Factors Subcommittee

August 29-30, 2017

Attendees List

August, 30, 2017

Name Affiliation

Paul Krois FAA

Mark S. Orr FAA

Kim Cardosi Volpe

Phil Smith OSU

Chris Desenti MITRE

Alan Jacobson Boeing

Richard Mendell FAA

Bill Kaliardos FAA

Jimmy Bruno FAA

Tim Evans FAA

Kevin Comstock ALPA

Randy Bone MITRE

Kenneth Allendoerfer FAA

Ryan Blondino FAA

Edward Austrian Fort Hill Group

Carl Bearnsten BAH

Dana Broach FAA

Ashley Awwad FAA

Jon Jonsson Hi-Tec Systems

Bridget Lewis Humanproof

Maura Lohenz DOT VOLPE

Dana Bromen FAA/CAMI

Kelley Krokos AIR

Barbara Holder Honeywell

Chinita Roundtree-Coleman FAA