Richard J. Ranaudo, Borja Martos, and Bill W. Norton

University of Tennessee Space Institute, Tullahoma, Tennessee 

David R. Gingras and Billy P. Barnhart

Bihrle Applied Research, Inc., Hampton, Virginia

Thomas P. Ratvasky

Glenn Research Center, Cleveland, Ohio

Eugene Morelli

Langley Research Center, Hampton, Virginia 

Piloted Simulation to Evaluate the Utility of a 

Real Time Envelope Protection System for 

Mitigating In-Flight Icing Hazards

NASA/TM—2011-216951

October 2011

AIAA–2010–7987
NASA STI Program . . . in Profile

Since its founding, NASA has been dedicated to the 

advancement of aeronautics and space science. The 

NASA Scientific and Technical Information (STI) 

program plays a key part in helping NASA maintain 

this important role.

The NASA STI Program operates under the auspices 

of the Agency Chief Information Officer. It collects, 

organizes, provides for archiving, and disseminates 

NASA’s STI. The NASA STI program provides access 

to the NASA Aeronautics and Space Database and 

its public interface, the NASA Technical Reports 

Server, thus providing one of the largest collections 

of aeronautical and space science STI in the world. 

Results are published in both non-NASA channels 

and by NASA in the NASA STI Report Series, which 

includes the following report types:

• TECHNICAL PUBLICATION. Reports of 

completed research or a major significant phase 

of research that present the results of NASA 

programs and include extensive data or theoretical 

analysis. Includes compilations of significant 

scientific and technical data and information 

deemed to be of continuing reference value. 

NASA counterpart of peer-reviewed formal 

professional papers but has less stringent 

limitations on manuscript length and extent of 

graphic presentations.

• TECHNICAL MEMORANDUM. Scientific 

and technical findings that are preliminary or 

of specialized interest, e.g., quick release 

reports, working papers, and bibliographies that 

contain minimal annotation. Does not contain 

extensive analysis.

• CONTRACTOR REPORT. Scientific and 

technical findings by NASA-sponsored 

contractors and grantees.

• CONFERENCE PUBLICATION. Collected 

papers from scientific and technical 

conferences, symposia, seminars, or other 

meetings sponsored or cosponsored by NASA.

• SPECIAL PUBLICATION. Scientific, 

technical, or historical information from 

NASA programs, projects, and missions, often 

concerned with subjects having substantial 

public interest.

• TECHNICAL TRANSLATION. Englishlanguage translations of foreign scientific and 

technical material pertinent to NASA’s mission.

Specialized services also include creating custom 

thesauri, building customized databases, organizing 

and publishing research results.

For more information about the NASA STI 

program, see the following:

• Access the NASA STI program home page at 

http://www.sti.nasa.gov

• E-mail your question via the Internet to help@

sti.nasa.gov

• Fax your question to the NASA STI Help Desk 

at 443–757–5803

• Telephone the NASA STI Help Desk at

443–757–5802

• Write to:

 NASA Center for AeroSpace Information (CASI)

 7115 Standard Drive

 Hanover, MD 21076–1320
Richard J. Ranaudo, Borja Martos, and Bill W. Norton

University of Tennessee Space Institute, Tullahoma, Tennessee 

David R. Gingras and Billy P. Barnhart

Bihrle Applied Research, Inc., Hampton, Virginia

Thomas P. Ratvasky

Glenn Research Center, Cleveland, Ohio

Eugene Morelli

Langley Research Center, Hampton, Virginia 

Piloted Simulation to Evaluate the Utility of a 

Real Time Envelope Protection System for 

Mitigating In-Flight Icing Hazards

NASA/TM—2011-216951

October 2011

AIAA–2010–7987

National Aeronautics and

Space Administration

Glenn Research Center

Cleveland, Ohio 44135

Prepared for the

Atmospheric and Space Environments Conference

sponsored by the American Institute of Aeronautics and Astronautics

Toronto, Ontario, Canada, August 2–5, 2010
Acknowledgments

The development and testing of ICEPro was funded under a federal grant, which was sponsored three-year cooperative research 

effort with NASA under the NASA Research Announcement NNH06ZEA001N, Appendix B, of the Aviation Safety Program. The 

objectives of the NRA were defined under the Integrated Vehicle Health Management Project, topic IVHM 3.1, Environmental 

Hazards, which are caused by the “Effects of Icing on Aircraft State”. The successful development, testing, and evaluation of ICEPro 

would not have been possible without the contributions of key individuals and organizations, which made up the research team. 

The author wishes to recognize Mr. Thomas Ratvasky, NASA Glenn Research Center, and Dr. Eugene Morelli, NASA Langley 

Research Center for their guidance and support throughout this project. Mr. Ratvasky’s experience and technical expertise with 

icing related problems were essential during the development and testing phases of ICEPro. Dr. Eugene Morelli provided critically 

important technical assistance during the integration and application of the Real Time Parameter Identification algorithm, which 

he and others from his organization had developed over a period of several years (see Refs. 12, 13, 14, and 16). In addition, Mr. 

David Gingras and Mr. Billy Barnhart, Bihrle Applied Research, who are experts in the field modeling and simulation, successfully 

designed, developed, integrated and tested the ICEPro system in NASA’s ICEFTD. Mr. Barnhart was directly involved during 

the test and evaluation phase of ICEPro, and along with Mr. Gingras, provided important simulation functionality, and solved 

numerous technical and integration problems necessary for a successful evaluation. Mr. Borja Martos, UTSI worked diligently 

with BAR to identify and address issues with ICEPro during its development, and was an invaluable test conductor during the 

pilot evaluation at ERAU. Mr. Billy Norton, an engineering management doctoral candidate at UTSI developed and conducted 

the statistical data handling methods, which were essential to the analysis of pilot performance metrics. Lastly, the authors wish 

to express their sincere appreciation to the faculty and staff of the Embry Riddle Aeronautical University for their support of this 

testing at their Eagle Works facility in Daytona Beach, Florida.

Available from

NASA Center for Aerospace Information

7115 Standard Drive

Hanover, MD 21076–1320

National Technical Information Service

5301 Shawnee Road

Alexandria, VA 22312

Available electronically at http://www.sti.nasa.gov

Trade names and trademarks are used in this report for identification 

only. Their usage does not constitute an official endorsement, 

either expressed or implied, by the National Aeronautics and 

Space Administration.

Level of Review: This material has been technically reviewed by technical management. 
NASA/TM—2011-216951 1

Piloted Simulation to Evaluate the Utility of a Real Time Envelope 

Protection System for Mitigating In-Flight Icing Hazards

Richard J. Ranaudo, Borja Martos, and Bill W. Norton

University of Tennessee Space Institute

Tullahoma, Tennessee 37388–9700

David R. Gingras and Billy P. Barnhart

Bihrle Applied Research, Inc.

Hampton, Virginia 23666 

Thomas P. Ratvasky

National Aeronautics and Space Administration

Glenn Research Center

Cleveland, Ohio 44135

Eugene Morelli

National Aeronautics and Space Administration

Langley Research Center

Hampton, Virginia 23681–2199

Abstract

The utility of the Icing Contamination Envelope Protection (ICEPro) system for mitigating a 

potentially hazardous icing condition was evaluated by 29 pilots using the NASA Ice Contamination 

Effects Flight Training Device (ICEFTD). ICEPro provides real time envelope protection cues and 

alerting messages on pilot displays. The pilots participating in this test were divided into two groups; a 

control group using baseline displays without ICEPro, and an experimental group using ICEPro driven 

display cueing. Each group flew identical precision approach and missed approach procedures with a 

simulated failure case icing condition. Pilot performance, workload, and survey questionnaires were 

collected for both groups of pilots. Results showed that real time assessment cues were effective in 

reducing the number of potentially hazardous upset events and in lessening exposure to loss of control 

following an incipient upset condition. Pilot workload with the added ICEPro displays was not 

measurably affected, but pilot opinion surveys showed that real time cueing greatly improved their 

situation awareness of a hazardous aircraft state.

Nomenclature

α Alternate Hypothesis

AOA Angle of Attack

D-ICES Dynamic Inversion Control Evaluation System

ICEFTD Ice Contamination Effects Flight Training Device

ICEPro Icing Contamination Envelope Protection System

ISP Icing Severity Parameter

H0 Null Hypothesis

PCE Pilot Coupling Event

RTPID Real Time Parameter Identification
NASA/TM—2011-216951 2

1.0 Introduction

The University of Tennessee Space Institute (UTSI) in partnership with Bihrle Applied Research 

(BAR) completed a 3-year cooperative research effort with NASA under NASA Research Announcement 

(NRA) NNH06ZEA001N, Appendix B, of the Aviation Safety Program. The objectives of the NRA were 

defined under the Integrated Vehicle Health Management (IVHM) Project, topic IVHM 3.1, 

Environmental Hazards, which are caused by the “Effects of Icing on Aircraft State”. This effort 

culminated in the development of a real time vehicle state assessment system, which was described by 

Gingras (Ref. 1) and is referred to as the Icing Contamination Envelope Protection system (ICEPro). The 

algorithms in ICEPro were initially developed and tested in a desktop simulation environment, and then 

integrated and further developed in the NASA Ice Contamination Effects Flight Training Device 

(ICEFTD). The two flight models used in simulation to develop ICEPro for testing and evaluation were 

based upon the characteristics of the NASA Twin Otter icing research aircraft in a no-ice baseline 

condition, and those representing a failure of the ice protection system (IPS). The failure case was defined 

by the ice accretions that would form during a 22.5 min FAR 25 Appendix C icing encounter as 

calculated by LEWICE version 2.0. Most of the data for the simulator came from testing a 6.5 percent

scale Twin Otter model in the Wichita State University 7X10 Low Speed wind tunnel and BAR’s Large 

Amplitude Multi-Purpose (LAMP) rotary balance wind tunnel using scaled ice shapes (Ref. 2). Flight data 

from aircraft testing with failure case ice shapes was also gathered to refine the sub-scale model data so it 

would match full scale flight test results (Ref. 3).

After development and testing in a desktop environment, ICEPro was integrated with the NASA

ICEFTD (Refs. 4 and 5). Researchers from BAR and UTSI completed final development and integration 

testing to ensure readiness for the planned pilot evaluations. A final check of test readiness was then 

performed by having two experienced NASA test pilots complete the evaluation profile, and all required 

pre-test preparation and training. This included a review of: the test plan, NASA web based icing training, 

standard operating procedures (SOP) for the Twin Otter, the NASA Task Load Index or TLX workload 

method (Ref. 6), a tutorial on flying with ICEPro displays (only for the pilot using the experimental 

displays), and post-test survey questionnaire. One NASA pilot was selected to represent the control group 

and flew with baseline displays. The other was selected to represent the experimental group and flew with 

the ICEPro modified displays. Before the test began, each pilot was briefed on the entire test process. The 

test protocol at the time called for minimal training in aircraft handling with hazardous ice formations, 

somewhat similar to that which a typical pilot would receive in an aircraft transition course. However, 

once testing began, it became apparent that the flight task was so difficult that without practice, there was 

a high probability of a random “crash” regardless of the displays used. It was apparent that a different 

approach would have to be taken. After assessing the pilot’s comments from the debriefings, it was 

therefore decided to modify the test protocol and train the pilots in both groups to proficiency for 

executing the hazardous approach task using their respective flight displays. Utility benefits could then be 

assessed by comparing pilot performance, workload, and opinions between control and experimental 

groups who were flying the same task with the same training and preparation, but with different flight 

displays. This adjustment in the test plan was an essential change, and would not have been discovered 

without this pre-test. The NASA pilots also completed the NASA TLX forms and answered all the survey 

questions during their respective evaluations. A few small changes in the survey questions were also 

required to eliminate ambiguity problems. After these adjustments were made to the test plan, pilot-inthe- loop (PIL) evaluations of ICEPro were scheduled at the Embry Riddle Aeronautical University 

(ERAU), Daytona Beach, Florida.

2.0 Background

Ice formations on aircraft can negatively affect performance, stability and control characteristics, and 

handling qualities. Accidents that have occurred due to these issues have been documented by the 

National Transportation Safety Board (NTSB) and the Federal Aviation Administration (FAA) over many 
NASA/TM—2011-216951 3

years. In a recent study (Ref. 7) of aircraft icing accidents and incidents reported between 1978 and 2002, 

loss of control events were usually preceded by stall and more likely to occur during approach and 

landing. This is the busiest phase of flight where historically over 50 percent of all accidents occur, and 

where human error is responsible for approximately 75 percent of those accidents (Ref. 8). Ice 

contamination, which accretes on a critical airframe surface such as the wings and tail, due to an Ice 

Protection System (IPS) failure, encountering an exceedance condition, or operator error can have a 

detrimental effect on aircraft performance, stability and control, and handling qualities. This situation is 

further exacerbated by the required changes in airspeed and configuration as the pilot slows and 

configures during the descent, approach, and landing phases. A lateral upset resulted in the fatal crash of 

American Eagle 4184 after encountering what appeared to be an exceedance icing condition. The NTSB 

listed the probable cause of the accident as “...the loss of control, attributed to a sudden and unexpected 

aileron hinge moment reversal that occurred after a ridge of ice accreted beyond the deice boot and 

ultimately lead to an upset condition” (Ref. 9). The pilots of this aircraft had no forewarning of the 

impending upset condition and furthermore, were flying on auto pilot at the time of the upset, which 

masked the rapid build-up in lateral control force as the flaps were raised while descending in the holding 

pattern. As the upset occurred, the autopilot disengaged and the aircraft rolled sharply to the right. The 

pilots then attempted to recover the aircraft, but in so doing entered a secondary stall upset, which caused 

the aircraft to roll inverted and enter a steep dive and crash. In its Tailplane Icing Program (TIP) NASA 

employed various artificial ice shapes, which were attached to the horizontal tail plane of a research 

aircraft (Ref. 10). Numerous test maneuvers were conducted where wing flap settings, engine power, and 

flight speed were varied. The results showed a considerable decrease in longitudinal stability and elevator 

control effectiveness as a function of increasing wing flap angle, power setting, and airspeed. In the 

extreme, a tail stall upset would occur, which resulted in an uncontrollable nose down pitching motion.

Airframe ice contamination also increases drag, reducing climb performance and is most evident in 

propeller driven aircraft. Although not a handling issue per se, it can greatly reduce engine out ceiling

(Ref. 11), which in multi-engine aircraft and could ultimately result in controlled flight into terrain or 

CFIT. This is especially a concern if an engine should fail during a missed approach procedure. In cases 

such as these, wing flap settings must be reduced and speeds increased to ensure a positive rate of climb 

while avoiding a stall upset. Unfortunately, pilots cannot always determine when the state of the aircraft is 

adversely degrading, nor are they always able to predict how a configuration change such as raising or 

lowering the wing flaps, would affect aircraft stability and control. The ICEPro system was therefore 

developed with the theory that a knowledge-based system, which compares predicted aircraft stability and 

control characteristics with those existing in real time, could effectively provide envelope protection cues 

and alerting messages to help the pilot keep the aircraft in a safe condition of flight. The knowledge base 

consists of the baseline or clean aircraft model, and an ice contamination model with known safe flight 

envelope limits as demonstrated by either wind tunnel or flight test. Angle of attack (AOA) cues, which 

are driven by ICEPro, show the pilot safe limits that will protect against either a wing stall or tail stall 

upset event. Messages, on the other hand, alert the pilot of loss in stability and control as a function of 

flight configuration and condition, and of configuration limits as a function of icing condition and wing 

flap setting.

3.0 ICEPro System Description

3.1 System Design and Architecture

During the initial system design stage of the program, the research team decided upon a real time 

state assessment system architecture, which would incorporate two state estimation methodologies; a 

Dynamic Inversion Control Evaluation System (D-ICES), which assesses control position from measured 

aircraft response, and a Real Time Parameter Identification (RTPID) (Refs. 12, 13, and 14) system, which 

estimates stability and control parameters from manual or automatic flight control system inputs and the 

resulting aircraft response. ICEPro is controlled by executive logic, which 
NASA/TM—2011-216951 4

Figure 1.—ICEPro system architecture.

uses D-ICES to compare existing control deflections with those predicted from a knowledge base of the 

“clean” or un-iced aircraft. As shown in Figure 1, the system architecture consists of three modes of 

operation, defined as Monitor, Identify or ID, and Report. In the “MONITOR” mode, measured control 

surface deflections are compared to predicted control surface deflections from the baseline aircraft model 

in the D-ICES algorithm. If the deflections of the real-time aircraft differ from those of the baseline model 

by a defined threshold value, ICEPro switches to the “ID” mode. The system logic executes this mode 

change by using a normalized statistical metric called a Theil (Ref. 15) coefficient, which is calculated 

from control deflection samples taken in a moving time window. This metric indicates the level of 

agreement between the actual control deflection and the D-ICES prediction. In this mode, ICEPro

diagnoses the degraded vehicle condition by executing a series of optimally-designed control excitations 

(Ref. 16) for 10 sec to provide information for the RTPID algorithm to estimate stability and control 

derivatives from measured control surface deflections and vehicle response. An “Icing Severity 

Parameter” or ISP is calculated based upon the time averaged differences between the a priori clean and 

iced values. When the ISP exceeds a threshold amount, ”Report” mode is entered and flyable cues and 

alerting messages relating to the severity of the condition are provided on pilot displays. These displays 

provide a prognosis of the condition and envelope protection limits. This process is iterative, and 

continues throughout the duration of the flight. If at any time the ISP drops below a predetermined 

threshold, cueing and messages will disappear and the system returns to the monitor mode. The system 

also consists of latching and de-latching timers, not shown in the figure, which prevent nuisance cueing 

and message toggling. It is beyond the scope of this paper to provide specific system design details. These 

details however, will be discussed in a companion paper by David Gingras, which will be presented at the 

AIAA Modeling and Simulation Conference, Toronto, Ontario, Canada, August 2 to 5, 2010.

3.2 Functionality Description and Flight Displays

The fundamental function of ICEPro is to perform real time state assessment and provide cues and 

messages to the flight deck that assist the pilot in keeping the aircraft in a safe operating envelope at all 

times. Figure 2 shows how those cues and messages are implemented on flight displays.

Airspeed carets provide high and low airspeed limits for the current aircraft configuration and 

estimated state due to icing. In general, the high speed limit is based upon longitudinal stability and 

control considerations, and the low speed limit is based on wing stall speed. The carets appear next to the 

airspeed tape and are marked with a snowflake symbol to indicate that they are estimated by ICEPro.

Their position on the tape depends upon wing flap setting and the estimated aircraft state due to icing.

Figure 2 shows a notional display of the high speed limit (rectangular wedge caret) corresponding to tail 
NASA/TM—2011-216951 5

Figure 2.—ICEPro flight displays.

stall upset speed, and the low speed limit (triangular caret) corresponding to the stall speed with wingflaps in the landing position. When the wing flaps are not in the landing position, the triangular stall speed 

caret is amber and provides stall warning corresponding to 107 percent of the stall speed.

AOA brackets provide maneuvering limits in pitch for the pilot. These brackets are superimposed on the 

pitch ladder of the attitude indicator display. There is an upper bracket that indicates the predicted stall AOA 

for the current configuration and ice contamination state, and a lower bracket that indicates the minimum safe 

AOA to provide tail stall and/or negative-g protection. The brackets are nominally white when the aircraft is 

being flown at a safe mid range AOA between the brackets. When the AOA approaches either the upper or 

lower bracket, both brackets turn amber to indicate caution, and when the AOA is at or beyond the bracket 

levels, the brackets turn red to warn of an unsafe condition. In addition, whenever the brackets turn amber or 

red, they flash for several seconds to ensure that the pilot notices the change. The relationship between upper 

and lower bracket depend upon the AOA limits for the current wing flap configuration. 

ICEPro provides a flap position indication that advises the pilot as to the state of the vehicle at the 

current flap deflection. As flaps are deployed during icing conditions and the system detects degradations 

in stability and control and/or in the single-engine climb capability, the indicator will change to an amber 

color, cautioning further deployment of the flaps. As the condition worsens, the indicator turns red, 

warning the pilot to stop deployment and retract the flaps until the condition is cleared.

Stability and control alerting messages are provided to inform the pilot of an adverse change in the 

stability and or control in the pitch, roll, or yaw axes. These messages on the Primary Flight Display 

(PFD) are repeated on the Flight Control Status display, which is shown to the right of the PFD on 

Figure 2. Thresholds for displaying the alerting messages are activated when pre-established reductions 

are determined from real time parameter estimates. An amber PITCH DGRD message for example, 

means that pitch stability or controllability has degraded by 50 percent or more than that of the nominal 

un-iced aircraft. The message turns red when degradation is 75 percent or greater than nominal. To 

enhance pilot awareness, the color of the primary or secondary control surface in the affected control axis 

is also changed to that of the message. This functionality prompts the pilot to return to a previous 

configuration and flight condition that provided safer flight characteristics.
NASA/TM—2011-216951 6

A climb limit (CLIMB LIM) caution and warning message is a performance-based cue to provide the 

pilot with an indication of single-engine climb potential given the current state of the aircraft. An amber 

message indicates that should an engine fail the airplane climb rate for the current flap setting will be less 

than 100 ft per minute. If the message is red, this indicates that the airplane will be unable to climb or will 

descend in the event of an engine failure. 

Later in development, it was also decided to incorporate a “stick shaker” for stall warning, a STALL 

message, stick pusher and aural warning for a stall upset and a de-cluttering mode for the primary flight 

display when flight attitudes became excessive during extreme upset conditions. De-cluttering removes 

all but primary recovery cues such as AOA and airspeed limits.

4.0 Test Methodology

4.1 Experiment Design 

The purpose of the test and evaluation with pilots in the loop was to determine if ICEPro had utility 

for mitigating a potentially hazardous icing encounter. The independent parameters were the two displays 

(baseline, and ICEPro with envelope protection), and the dependent parameters were technical 

performance, workload, and pilot opinion. A non-parametric, independent measures experiment was 

therefore selected where differences in the dependent variables were analyzed using the Mann-Whitney 

“U” test (Ref. 17). A critical “U” was determined, based on a one-tailed test where a 95 percent

confidence criteria or better was required to correctly reject the null hypothesis (H0). H0 assumed that 

there was no difference between the two groups for the tested variable, therefore the level of significance 

or alpha level required to reject the null hypothesis was chosen to be α = 0.05. The experiment was 

planned for an evaluation by 30 pilots with relatively similar flight experience. All were ERAU instructor 

pilots with a median flight experience of approximately 1300 flight hours. Each possessed commercial, 

multi-engine, and instrument ratings, but none had any specific experience in actual icing conditions or 

any prior training in aircraft handling and upset recovery due to a hazardous icing condition. The 30 pilots 

were randomly divided into a control group (baseline flight displays and no ICEPro system) and an 

experimental group (displays modified with ICEPro cues and messages). A scheduling problem resulted 

in a total of 29 pilot evaluations and unequal sample sizes between the control and experimental groups

(nc = 14 and ne = 15). This disparity was accounted for by the MWU methodology as provided for in 

Reference 17.

4.2 ICEFTD Simulator Description

The aircraft model and sub-system components for the ICEFTD simulator were assembled in BAR’s 

commercial off the shelf (COTS) proprietary simulator development environment, D-Six (Bihrle Applied 

Research, Inc.). The simulator provides highly representative flight characteristics of the Twin Otter icing 

research aircraft. Although originally designed as a pilot training device, the ICEFTD’s ability to 

accurately represent aircraft icing characteristics made it a very useful research tool with several 

advantageous features. One is its capability for accurately representing flight characteristics beyond stall 

AOA, which permits pilots to experience post-stall gyrations and tail stall hard-over’s that occur during 

upset conditions. Another is the pitch control system, which can provide control force gradients and 

deflections that closely match those recorded from flight tests of the NASA Twin Otter during upset 

events. Lastly, an “aircraft emulator” was designed and integrated in order to represent the same sensor 

feedback and output response as that of the aircraft. Much of the functionality testing during development 

focused on refining the system logic to fine tune thresholds and minimize nuisance messaging. Figure 3 is 

an overview of the ICEFTD as modified for this test. The items in red font are those which support 

ICEPro operation.
NASA/TM—2011-216951 7

Figure 3.—ICEFTD configuration for test and evaluation of ICEPro.

The ICEFTD simulator also provided excellent three-screen graphics of a visual scene, which was an 

essential element of the final approach and missed approach task in this evaluation. A video recording 

camera system was also employed for each pilot evaluation run in the event a replay of the pilot’s actions 

needed to be observed. During testing, the evaluation pilot wore a headset and was completely enclosed in 

the simulator by a system of curtains (not shown in the figure). This isolated the pilot from any outside 

disturbances or distractions, and enhanced the fidelity of the simulation environment. A member of the test 

team operated the simulator and performed data acquisition and processing tasks from a remote console. A 

second member communicated with the pilot while acting as test conductor and air traffic control (ATC).

4.3 Pre-Test Pilot Proficiency Training

Before performing the evaluation tasks for data collection, each pilot was subjected to approximately 

1.5 hr of flight training. Pilots were trained for the evaluation task using the respective flight displays 

depending on whether they were in the control or experimental group. Training began by having each 

pilot execute the evaluation approach task profile as shown in Figure 4 in a no-ice or baseline 

configuration under instrument meteorological conditions (IMC). The IMC condition for training and 

testing was set at 400 ft and 1 nautical mile (NM) visibility. The white boxes in the figure describe the 

flight condition and events during the task and the green shaded boxes show the commands given by the 

test conductor, who acted as ATC at various stages during its execution. The missed approach procedure 

(MAP) with an engine failure was initiated by the test conductor at the approximate position as shown in 

red font. Pilots practiced the approach task according to the SOP, which specified speeds, configurations, 

and performance standards throughout the profile. This phase of training was important in order for the 

test team to determine that each evaluation pilot possessed the requisite instrument flight skills to 

satisfactorily perform the task. The next phase of training was conducted up and away under visual 

meteorological conditions (VMC) to familiarize pilots and develop their skill in dealing with aircraft 

characteristics due to the failure case icing condition. This training emphasized recognition of impending 

upset conditions and the effects of wing flap configuration and flight speed on aircraft handling and 

control characteristics. Pilots in the control group were taught how to most effectively use their baseline 

flight displays to control the aircraft and recognize and recover from upset conditions. Pilots in the 

experimental group received the same training emphasis, but were taught how to integrate their scan and 
NASA/TM—2011-216951 8

Figure 4.—Evaluation profile approach task flown by each pilot group during ICEPro testing and evaluation.

control strategy with envelope protection cues to facilitate safe flight control and upset recoveries. During 

this training, evaluation pilots were also instructed to fly with both hands on the yoke. For some pilots 

this was a learning experience because nearly all of them were used to one handed operation in aircraft 

with relatively light control forces. Once the up and away training was completed, each of the pilots 

practiced the approach procedure with the hazardous failure case icing condition as would be performed 

in the actual evaluation. A certain amount of time was also spent having the pilots practice the missed 

approach procedure under iced conditions. This was the most difficult task to perform because of high 

pitch control forces and the requirement to null the asymmetric thrust with high rudder forces when an 

engine was failed. Pilots had to practice this maneuver several times before they could consistently 

accomplish it in a proficient manner and not lose control. Evaluation pilots were continually reminded to 

fly according to the SOP, which was intentionally designed to place the aircraft near the edges of the 

flight envelope at all times. This protocol forced considerable attention to precise aircraft control 

throughout the approach task, which was essential for making a good utility assessment of ICEPro. The 

evaluation profile approach task for both groups required considerable mental concentration, physical 

strength and very aggressive control inputs to prevent or recover from upset conditions. All pilots, 

regardless of size or gender, were able to effectively perform the approach task where pitch control forces 

reached 90 to 100 lbf at times. In general, most pilots only required one practice approach before they 

were ready to begin the evaluation phase. In addition to the flight training, pilots were briefed on and 
NASA/TM—2011-216951 9

practiced completing the NASA TLX workload form. Following this training, pilots were critiqued by the

test team, and then given a short break before they began their evaluation runs. 

4.4 Flight Evaluations

The evaluation phase required that a total of three satisfactory approach tasks be performed by each 

pilot. The test conductor sat behind the simulator cab and acted as ATC and was able to view the evaluation 

pilot through a small opening in the rear curtain. During each evaluation run, the pilots were completely 

enclosed in the simulator cab, and no coaching or extraneous discussions were permitted. As shown in 

Figure 4, the approach task consisted of “radar vectors” to intercept a precision approach procedure. The 

pilot would execute the approach according the SOP, and “break out” of the clouds at 400 ft above ground 

(AGL), and continue the descent to the runway on a visual glide path. At approximately 100 ft AGL, “ATC” 

would instruct a missed approach procedure, which the pilot would acknowledge and then initiate. As soon 

as both power levers were fully advanced, the test conductor would fail an engine, and the pilot would 

continue the procedure while raising the wing flaps from full down to 10°. The test would be terminated 

when the evaluation pilot was stabilized on the missed approach heading. Once complete, the evaluation 

pilot would fill out an electronic TLX form, which automatically computed the numerical workload score, 

and receive a short debriefing from the test team. The pilot would then re-enter the simulator for the next 

evaluation run. The entire evaluation process for each pilot including the pre-test training exercises and three 

evaluation runs took approximately 3 hr, which seemed to be the limit before fatigue became a factor 

affecting pilot performance. After testing was completed for each pilot, they were asked to fill out a post-test 

survey questionnaire, included in Appendix A. This questionnaire consisted of four sections using a Likert

(Ref. 18) style format: Part I provided pilot demographic and experience information; Part II asked about 

situational awareness; Part III asked for the experimental group’s assessment of ICEPro integration; and 

Part IV asked general questions about workload factors.

4.5 Data Acquisition and Handling

Pilot flight performance data parameters, which were available through a flight data file that was 

resident in D-Six, were sampled at 10 Hz. MATLAB (The MathWorks, Inc.) routines were used to collect 

and reduce these data for further analysis. For the final approach segment of the task, the Theil coefficient 

was used to evaluate how closely the pilot’s flight path and airspeed agreed with that of a perfectly flown 

approach. A Theil coefficient was calculated for measured lateral and vertical deviations from the 

centerline of the approach path, and the airspeed error from the required 75 kn indicated airspeed (KIAS) 

approach speed. MATLAB routines were also used to record the number of times the stick shaker and 

pusher fired, and number and depth of tail stall AOA exceedance events over the duration of the entire 

flight task. In addition to these performance data, other parameters were recorded during each run to 

provide a more complete picture of the pilot’s control strategy which was also useful when training and 

de-briefing the evaluation pilots. These included time histories of pressure altitude (HP), pitch attitude (θ), 

pitch rate (q), normal acceleration (nZ), elevator position (δe), elevator control force (Fe), and wing flap 

setting (δF). Pilots completed the NASA TLX immediately after each data run, and then completed the 

survey questionnaire at the termination of the evaluation session.

The pilot performance data quantified how well the pilot flew the approach task, how many stick shaker 

and pusher events occurred, and the number of times and extent to which the tail stall AOA was exceeded 

by 5° or more. It should be noted that reaching a tail stall AOA, which was one of the measured parameters,

did not always result in an upset as long as the pilot immediately recognized the condition and promptly 

arrested the negative pitch rate. The added 5° margin took into account pilot reaction time and aircraft 

dynamic response to the controls. Therefore, tail stall upsets were defined by the number of times the pilot 

reached or exceeded the 5° margin, and the maximum negative AOA achieved in the event. This latter 

metric was also used to define “upset risk”, since the deeper the stall, the greater the chance of an upset from 

which recovery was less probable. In addition to pilot performance data, the numerical scores of the TLX 
NASA/TM—2011-216951 10

workload assessments and the numerical results from the Likert-type post-test questionnaires were analyzed 

by the MWU process. The TLX analysis was of interest to understand if the added information of the 

ICEPro displays adversely affected workload. The post-test survey analysis was used to assess if there was a 

real and positive difference in hazardous aircraft state awareness, and perception of workload between the 

two groups. Additionally, a section of the survey provided non-comparative information on how well the 

experimental group felt the ICEPro was integrated with flight displays.

The utility evaluation of ICEPro was based upon assessing real differences between the control and 

experimental groups. The MWU analysis process began by rank ordering the scores of each group, and 

computing a “U” value based on the rank sums. In this one directional test, the “U” for the experimental 

group had to be greater than 144 to ensure a 95 percent probability for correctly rejecting H0. A cumulative 

probability p was calculated from the z score and was compared to α = 0.05, the requirement for accepting 

the alternate hypothesis. If this probability was less than 0.05, the null hypothesis was rejected. A negative z

score (number of standard deviations) of more than magnitude 1.69 (i.e., less than –1.69) was required to 

produce the cumulative probability which met this requirement. The parameters which were assessed 

included all quantified measures of pilot performance, TLX workload, and subjective opinions from the 

survey questionnaires.

An assessment of recovery from tail stall upsets was also performed. In many cases, when a tail stall 

event was encountered, a series of pitch oscillations would ensue from which the pilot would have to 

recover. An analysis of repeated events was performed to determine if the experimental group could 

recover from these oscillations using ICEPro cueing sooner than the control group, and thereby be less 

prone to entering an out of control condition. A pilot coupling event (PCE) represents a hazardous 

situation, which if not arrested quickly, could progress to a loss of control. In this context, the risk of an 

unrecoverable upset was associated with the number of repeated tail stalls in a PCE scenario, and the 

depth of each stall beyond the 5° margin previously discussed. A time window analysis was therefore 

used to analyze the number of tail stalls occurring within a continuous 5, 10, 15, and 20 sec span, while 

the pilot was attempting a recovery. Search scripts in MATLAB were used to analyze the beginning and 

ending of each PCE, and total the number and severity of each tail stall within the PCE. The results were 

then reported in a tabular format for both groups, and analyzed for significance by the MWU method. An 

important outcome of this analysis was to assess the utility of ICEPro AOA cues for minimizing the 

recovery time from a PCE, and thereby reduce risk of a loss of control situation.

5.0 Results and Discussion

5.1 Pilot Demographics

All evaluation pilots were instructor pilots in the ERAU professional pilot training program. Thirty 

pilot volunteers were broken down into two groups—a control group who flew baseline displays, and an 

experimental group who flew with ICEPro modified displays. The 29 ERAU instructor pilots held FAA 

commercial licenses with multi-engine and instrument ratings. The demographic data was collected in 

Part I of the post-test survey questionnaire, and is discussed in Section 5.6. Due to a scheduling problem, 

the control group consisted of one less pilot that the experimental group. The demographics of the groups 

were as follows:

• Subjects and gender

– 14 control group (12 male, 2 female)

– 15 experimental group (12 male, 3 female)

• Median flight hours

– Control group Experimental group

 Total 1350 Total 1250

 Multi-engine 122 Multi-engine 100
NASA/TM—2011-216951 11

Icing related experience and training as related from Part I of the post-test survey questionnaire:

• 75 percent had no prior in-flight icing experience

• 89 percent did not feel that their prior icing related knowledge/experience would have prepared 

them to perform the test scenario (training was required in order for pilots to execute the flight 

task)

• 72 percent agreed that the NASA videos/web based training gave more information about icing 

than they had known before (viewing NASA web based training videos was a pre-test 

requirement)

• Mixed responses were given for icing related flight training being focused on IPS operation (this 

depended on the types and classes of aircraft in which pilots had been previously trained)

5.2 Pilot Control Performance 

Pilot control performance assessment from the control and experimental groups are summarized in 

Figure 5. Results highlighted in green indicate that the significance or real difference criterion for the 

assessed parameter was met, and the performance of the experimental group was better than that of the 

control group. Results highlighted in blue indicate that the assessed parameter showed better performance 

by the experimental group, but the result did not meet the real difference criterion. Results highlighted in 

orange, indicate that there was no difference between the performance of the two groups, and results 

highlighted in magenta indicate that the experimental group did not perform as well as the control group.

In general the cumulative pilot performance for the two most critical events, pusher activation and tail 

stall, met the real difference criterion. These events were considered most important from a safety of 

flight standpoint because a wing or tail stall upset had occurred, and potential for loss of control was very 

high unless the pilot promptly applied the correct sequence of recovery controls. The shaker activation 

event indicated that the control group had slightly better performance than the experimental group, but 

this finding can be misleading without considering important aspects of shaker implementation and pilot 

use of this functionality in a human factors context. This is explained in the following discussion of the 

results, which were tabulated in Figure 5. The first column of the figure identifies the parameter or event 

that was evaluated during the approach and missed approach task, the second column indicates the 

number of standard deviations (z) required in meeting the real difference criterion, the third column 

indicated the cumulative probability (p), and the last column provides a performance assessment between 

the experimental and control group based upon the analysis.

5.2.1 Shaker Events—Stall Warning

The results of the pilot performance for avoiding stick shaker events as shown in Figure 5 indicate 

that, cumulatively, the control group had a slightly better performance than the experimental group. 

Shaker firings typically occurred in banked turns if speed got too low, during positive pitch overshoots 

incited by a tail stall induced PCE, and when pilots made an overly aggressive stick pusher recovery. The 

following discussion places these results into perspective and discusses factors that may have affected 

their outcome. The control group flew with basic aircraft displays and a simple stick shaker functionality, 

which always provided a 7 percent stall speed margin based upon a clean wing; i.e., no-ice. Therefore, 

when the shaker fired during the approach task in the icing condition, the actual margin to stall was less 

than 7 percent. On the other hand, the experimental group flew with ICEPro generated displays, which

provided real time AOA cues and shaker functionality that maintained a 7 percent stall speed margin for 

the icing condition. Experimental group pilots could avoid shaker events if they remained below the upper 

alpha limit as shown in Figure 2. The control group had no AOA information on their basic flight 

displays. In order to avoid shaker activation, they had to closely maintain the SOP required airspeeds 

throughout the entire task. 
NASA/TM—2011-216951 12

Pilot control 

performance by run 

number and 

cumulative

z score:

z < -1.69 for 95% 

confidence in H0

rejection

Probability:

p < 0.05 required to satisfy α 

(alternate hypothesis)

Group performance: Exp. versus

control

Positive real difference (PRD), 

Better (B), Worse (W), Same (S)

Shaker events

Run 1 1.0474 0.8575 W 

Run 2 –0.2837 0.3799 B 

Run 3 –0.5892 0.2705 B 

Cumulative runs 0.0873 0.5434 S

Pusher events

Run 1 –0.0655 0.4652 S

Run 2 –1.2875 0.0952 B

Run 3 –2.1822 0.0137 PRD

Cumulative runs –1.7457 0.0385 PRD

Stall Messages

No stall messages appeared for the experimental group

Tail stall events

Run 1 –1.2657 0.0989 B

Run 2 –1.8330 0.0318 PRD

Run 3 –1.7894 0.0350 PRD

Cumulative runs –2.1822 0.0137 PRD

Legend

Meets 95% confidence for rejecting H0 

Experimental group better over control group but < 95% confidence for rejecting H0

No difference between experimental and control group

Experimental group performance worse than control group

Figure 5.—Summary of pilot control performance

Notably, the control group was technically at greater risk when the shaker fired because their stall 

speed margin was less than 7 percent. A study of the frequency of shaker events is helpful in order to 

better understand the results of Figure 5. Table 1 provides the number of recorded shaker events per run 

for each group of pilots.

TABLE 1.—NORMALIZED SHAKER EVENTS

Control group 

(14 pilots)

Experimental group 

(15 pilots)

Run Shaker events Normalized Run Shaker events Normalized

1 87 6.21 1 113 7.53

2 92 6.57 2 86 5.73

3 87 6.21 3 72 4.80

Cum 266 6.33 Cum 271 6.02

During run one, 14 control group pilots experienced 87 shaker events, whereas 15 experimental group 

pilots experienced 113 shaker events. On average, the control group experienced 6.21 events per pilot, 

and the experimental group experienced 7.53 events per pilot. The corresponding MWU analysis for run 

one in Figure 5 clearly shows better (but not significant) performance from the control group. During the 

subsequent runs, however, the experimental group’s performance got better, and this is reflected in the 

analysis for runs two and three in Figure 5. If all three runs are considered cumulatively, the normalized 

ratio between the two groups is slightly better for the experimental group, but the cumulative results of 

the statistical analysis shown in Figure 5 slightly favors the performance of the control group. Regardless, 
NASA/TM—2011-216951 13

the differences between control and experimental groups as shown in Figure 5 and Table 1 are very small. 

What is important to observe here is that the experimental group did a much better job of avoiding shaker 

events during subsequent runs, while the control group’s performance remained nearly constant. This was 

likely due to increased proficiency in using ICEPro AOA cues, which was not fully realized during pretest training. Unfortunately, practical limits on pre-test training were necessary in order to conduct all 

required evaluations within time and program cost constraints.

5.2.2 Pusher Events—Wing Stall Upsets

Table 2 summarizes the number of pusher events per run for each group. Pusher events were 

generally precipitated by the same factors that incited shaker events and occurred when the pilot could not 

arrest an angle of attack increase, which resulted in a stall upset. This was particularly evident when the 

missed approach procedure was executed. Here, pilots immediately advanced the throttles to full power, 

raised the wing flaps to 10° and the nose to the takeoff attitude in one coordinated motion. As soon the 

power levers reached the full power stop and the evaluation pilot reached for the wing flap lever, the left 

engine was failed by the test conductor, and the pilot then had to null the yaw and adjust pitch so as to 

maintain a 76 KIAS climb speed. This was a very dynamic maneuver, and in many cases it precipitated a 

PCE. As in the case of the shaker events, the experimental group of pilots improved with each evaluation 

run, whereas the control group actually got worse. The raw data in Table 2 supports the statistical 

assessment that the ICEPro displays resulted in better pilot control performance and lower risk of an upset 

due to either wing or tail stall.

TABLE 2.—NORMALIZED PUSHER EVENTS

Control group 

(14 pilots)

Experimental group 

(15 pilots)

Run Pusher events Normalized Run Pusher events Normalized

1 12 0.86 1 13 0.87

2 16 1.14 2 10 .67

3 19 1.16 3 6 .40

Cum 47 1.05 Cum 29 .64

5.2.3 Stall Messages

A “STALL” message was incorporated into the PFD in the event the pilot attempted to override the 

pusher and thereby remain in a stall upset condition. This was a functionality associated with ICEPro 

displays, and appeared to be unnecessary as all the experimental pilots promptly applied recovery controls 

to reduce the AOA below the upper bar when the pusher fired. No stall messages appeared during any of 

the runs flown by the experimental group.

5.2.4 Tail-Stall Upsets and Recoveries

Tail stall upsets resulted when pilots flew too fast, exceeding the speeds specified at various segments 

of the flight task, or when they allowed the angle of attack to get too low. This condition was greatly 

aggravated with increasing wing flap deflections, and was most sensitive during the final approach and 

missed approach procedure when the flaps were 30o down. As described previously, a tail stall was 

defined as reaching a negative angle of attack that was 5° below that where the tail had technically 

reached a stall condition. This margin was determined experimentally, as it appeared that if a pilot 

immediately applied recovery controls before reaching this negative angle of attack, recovery would be 

prompt and the chance of a PCE minimized. When a tail stall occurs in an aircraft equipped with 

mechanical controls, such as the one simulated in this test, the pilot feels an immediate increase in pull 

force and notes a rapid build-up in negative pitch rate. In order to arrest the condition, the pilot must 

quickly exert very high pull forces (on the order of 90 to 100 lbf) to reverse the downward motion of the 

nose. If successful, the nose starts back up; but, as it does, pitch forces immediately lighten and a positive 

pitch overshoot can easily occur. This can precipitate a PCE, which can develop into a loss of control 
NASA/TM—2011-216951 14

situation should a negative angle of attack be reached, which the pilot is not able to physically overcome.

Pre-test pilot training acquainted both control and experimental group pilots with these characteristics, 

and the basic methods for recovery. The control group, of course, did not have the AOA displays of the 

experimental group and therefore did not know their margins to either a wing or tail stall upset. They 

practiced tail stall upset recoveries by keeping the pitch divergence to a minimum. The experimental 

group, on the other hand, was taught to use the ICEPro displays to facilitate recovery by adjusting pitch 

during a PCE to remain within the safe envelope defined by the AOA bars. When pitch rate was negative, 

the pilot would attempt to pull the nose up but remain below the high AOA bar. If the pitch rate was 

positive, the pilot would push the nose down, but remain above the low AOA bar. If done properly, this 

would result is a very expeditious recovery—normally within one or two cycles. Of note, this was also the 

reason why there were fewer pusher events as the experimental group tended to be better at avoiding stall 

angle of attack than the control group. The learning bias evident in the shaker and pusher data seem to 

indicate that had more time been available for the experimental group to practice with the ICEPro

displays, their performance during these events would have been better. Runs 2, 3, and cumulative of the 

tail stall event in Figure 5 clearly show positive real difference in the performance of the experimental 

group and provide better than 95 percent confidence for correctly rejecting H0.

5.3 Pilot Precision Approach Performance

The final approach segment required the pilots to intercept and accurately fly a precision approach to 

the minimum descent altitude of 200 ft. As shown in Figure 2 the flight displays for both groups did not 

provide computed steering or a flight director format. Therefore, the pilots had to fly what is termed “raw 

data” and null localizer and glide slope errors by controlling heading and pitch attitude. As described 

earlier, this task was also performed in IMC until reaching the imposed ceiling of 400 ft above ground 

level. When the pilots “broke out” of the cloud at this altitude, they continued their descent to 100 ft and 

were directed to execute a missed approach. The difficulty in flying the approach task was largely due to 

control aspects discussed earlier. Some pilots were very smooth and precise throughout the task, and were 

able to execute it without too much difficulty. This was because they did not allow the attitude of the 

aircraft to vary much beyond that required to maintain a given flight condition. Generally, these were 

pilots who had very efficient instrument scans, and who made very small but immediate corrections when 

required. Pilots whose instrument scans were less efficient and allowed large pitch or roll variations to 

occur were more prone to get into a pitch axis PCE. This event greatly affected glide slope, course, and 

airspeed errors, and was reflected in the Theil coefficient values for each of those parameters. Figure 6

provides the results of the MWU analysis of the precision approach errors between the control and 

experimental groups. It should be noted, that both groups of pilots generally flew the approach procedure 

within ATP standards (Ref. 19) as required in the SOP. The Theil coefficient values, which were used in 

the MWU analysis, were basically a means of quantifying the pilot’s performance with respect to a 

perfectly flown approach.

As shown in Figure 6, the control group had better cumulative performance in minimizing localizer 

and airspeed errors. The experimental group on the other hand had better cumulative performance in 

minimizing glide slope error, but did not meet the significance criteria required to infer that there was a 

real difference between the two groups. It would appear, after considering the results from Figure 5 and 

Figure 6, that pilots who had the advantage of ICEPro displays tended to prioritize aircraft control over 

the preciseness at which they flew the localizer and the required airspeed. This is perhaps why the control 

group, who flew more precise airspeed, had significantly more tail stall upset events than the 

experimental group and was considered to be at greater risk of entering an out of control situation. 
NASA/TM—2011-216951 15

Pilot precision approach 

performance by run 

number and cumulative

z score:

z < -1.69 for 95% confidence 

in H0 rejection

Probability:

p < 0.05 required to 

satisfy α (alternate 

hypothesis)

Group Performance: Exp. 

versus Control

Positive real difference (PRD), 

Better (B), Worse (W), Same (S)

Localizer

Run 1 0.6328 0.7436 W

Run 2 0.2619 0.6116 W

Run 3 0.0436 0.5260 W

Cumulative 0.1528 0.5692 W

Glide slope

Run 1 –1.4402 0.0718 B

Run 2 0.5674 0.7221 W

Run 3 –1.4621 0.0689 B

Cumulative –0.4364 0.3233 B

Airspeed control

Run 1 1.8330 0.9681 W

Run 2 1.9640 0.9764 W

Run 3 1.4402 0.9281 W

Cumulative 1.9421 0.9752 W

Legend

Meets 95% confidence for rejecting H0

Experimental group better over control group but < 95% confidence for rejecting H0

No difference between experimental and control group

Experimental group performance worse than control group

Figure 6.—Pilot precision approach performance.

5.4 Exposure to Risk of an Out of Control Event

A number of tail stalls, which were encountered in the course of the approach procedure, resulted in 

PCEs before the pilot was able to stop the oscillations and recover to a stable and normal flight condition.

This event was described earlier in Section 4.5, and under “Tail-stall upsets and recoveries,” Section 5.2.

If a quick recovery from a tail stall event (TSE) could not be facilitated by the pilot, there was a risk 

that repeated oscillations result in a PCE which would deteriorate into an out of control situation.

Therefore, the numbers of repeated tails-stalls during PCEs were summed for each group during the final 

approach segment in four time duration windows of 5, 10, 15 and 20 sec. The methodology began by 

identifying an initial tail-stall event (TSE), which met the criterion (negative tail-stall AOA plus 5°). If 

another TSE occurred within 5 sec it would be counted with the first tail stall and considered a PCE. If in 

this event, TSEs continued to occur within 5 sec of the one previous, they would be added to the PCE 

until the duration of time between a repeat TSE was greater than 5 sec. This methodology was then 

repeated for time intervals of 10, 15, and 20 sec. The exposure to risk of reaching an out of control 

situation was therefore greater during a PCE based upon the number of repeat TSEs in any of the four 

time frames, the amount of time it took the pilot to recover, and the maximum negative pitch attitude 

reached during the event. The results of this analysis are provided in Table 3.
NASA/TM—2011-216951 16

TABLE 3.—EXPOSURE TO RISK BASED UPON PILOT COUPLING EVENTS

Group Time interval 

between repeat TSEs 

(sec)

Total 

number

of PCEs

Mean duration of PCEs

in the time interval 

(sec)

Mean max 

negative 

pitch

Control

5

16 15.16 –12.23

Experimental 4 8.66 –8.8

Control

10

69 18.54 –12.06

Experimental 29 17.98 –11.12

Control

15

84 23.08 –12.32

Experimental 36 23.22 –11.37

Control

20

90 27.64 –12.44

Experimental 41 25.85 –11.23

The most significant events occurred during the 5 sec interval. By using their ICEPro driven displays, 

the experimental group had 25 percent fewer PCEs, their events were approximately half as long in 

duration, and the maximum nose down attitude during the event was about 3.5° less than that of the 

control group. In effect, the experimental group could arrest a PCE quicker, with fewer oscillations than 

the control group. Most important however, was the maximum depth of the tail stall, which was measured 

by the maximum negative pitch attitude achieved during the recovery attempt. The more negative the 

attitude, the higher the negative column forces, and the more difficult it was for the pilot to arrest the 

negative pitching moment. Though not shown here in the data, some pilots had to use in excess of 90 to 

100 lbf to arrest the negative pitching moment, which was more likely to cause a positive pitch overshoot, 

and sometimes result in pusher activation. As previously mentioned, the test profile was intentionally 

designed to place pilots on the edge of a TSE during the approach, and as a result, most pilots in both 

groups experienced them, especially when momentarily distracted from their instrument cross-check. 

However, when a TSE occurred and especially when it deteriorated into a PCE, the experimental group 

was able to effectively utilize their AOA cues to damp the oscillations more quickly than the control 

group. They accomplished this recovery while minimizing their time at an unsafe negative AOA, and 

thereby were at less risk of reaching an out of control situation.

5.5 Workload

After completing each evaluation run, evaluation pilots were asked to complete a TLX workload 

assessment for the entire task. The flight profile was intentionally designed to place the each group of pilots 

very near the edges of the safe flight envelope especially during the final approach and missed approach 

segments. Further, the tasks were flown in IMC, and both the cognitive and physical aspects of workload 

were quite high for both groups, especially when recoveries from TSEs or PCEs were required. Because the 

experimental group had to integrate more information when performing the flight task than the control

group, there was a concern that their workload could be adversely affected because the ICEPro displays 

provided much more information than the basic displays, and had to be integrated into the pilot’s scan along 

with the instrument flying task. The control group did not have the additional information and thus had less 

cognitive workload. An analysis was conducted on each of the three runs for both groups, and the results 

shown in Figure 7 indicate that overall, the experimental group reported less total workload. Run 2 for 

example, taken by itself did meet the significance criteria and indicated that ICEPro displays made a real 

difference in reduced workload. However, on average for all runs, the results did not meet the significance 

criteria. Observation of both groups of pilots indicated that as they flew each successive profile, they 

became more comfortable with the task, and tended to report lower workload. In spite of the lower 

workload, it was also apparent that, by the second run, fatigue began setting in; this affected pilot 

concentration and control performance. The entire evaluation was conducted over an approximate 3 hr 

period with strategically inserted rest breaks, but it was apparent that the performance of the pilots from both 

groups seemed to peak by the second evaluation and decrease on the third run. The results of the post-test 

survey questions, which are discussed in the next section, provide further insight into these findings. 
NASA/TM—2011-216951 17

Pilot workload by 

run number and cumulative

z score:

z < -1.69 for 95% 

confidence in H0 rejection

Probability:

p < 0.05 to satisfy α 

(alternate hypothesis)

Group Performance:

Exp. versus Control

Positive real difference 

(PRD), Better (B), 

Worse (W), Same (S)

TLX Workload

Run 1 –0.5237 0.2926 B

Run 2 –1.8985 0.0274 PRD

Run 3 –1.4621 0.0689 B

Average all runs –1.0911 0.1328 B

Legend

Meets 95% confidence for rejecting H0

Experimental group better over control group but < 95% confidence for rejecting H0

No difference between experimental and control group

Control group performance better than experimental group

Figure 7.—Workload assessment.

5.6 Post-Test Survey Results

The questions in all four parts of the survey solicited pilot opinions to help gain insight of the two 

pilot groups with respect to their icing experience, perception of situation awareness, how well ICEPro 

was implemented, and workload issues associated with the flight task. Descriptive analysis was used for 

responses to Part I—Demographics and Part III—ICEPro Implementation. Parts II—Situation Awareness, 

and Part IV—Workload, were analyzed via the MWU methodology for real differences. The survey 

questions solicited either agreement, or frequency of occurrence. A Likert five-answer format was used 

with numbers 1 to 5 assigned to the answers from left to right. For the MWU analysis of Parts II and IV, 

the “direction” of the numbered response for each question, i.e., 1 to 5, or 5 to 1, was defined to indicate 

better performance. The survey questionnaire is provided in Appendix A for the reader’s reference, and 

histograms of the responses to questions in Parts I, II and IV are provided in Appendix B. Descriptiveonly assessments of experimental group pilot opinions are provided for Part III.

5.6.1 Part I—Demographics

The two pilot groups were intentionally selected from a relatively homogeneous sample. All pilots 

were essentially General Aviation (GA) instructor pilots who shared common experience with training, 

ratings, and types of aircraft flown. All pilots except one were staff instructors at ERAU and subscribed to 

the same operational and flight standards as defined by the University’s professional pilot training 

program. There was only one outlier in the control group, and that individual’s flight hours were not 

included in the standard deviation calculations as this individual’s flight time skewed the data 

considerably. With the elimination of that pilot’s flight hours, the standard deviation of the control group 

was 1054 hr, and the standard deviation of the experimental group was 698 hr. As shown in Figure B.1

nearly all pilots had minimal experience with aircraft icing or icing related training before participating in 

this test.

5.6.2 Part II—Situation Awareness

Figure 8 and Appendices A and B summarize the situational awareness responses to the questions.
NASA/TM—2011-216951 18

Pilot post-test survey 

questionnaire results

z score:

z < -1.69 for 95% 

confidence in H0

rejection

Probability:

p < 0.05 required to 

satisfy α (alternate 

hypothesis)

Group Performance: 

Exp. versus Control

Positive real difference 

(PRD), Better (B), 

Worse (W), Same (S)

Part II—Situational Awareness

1. Flight displays were adequate to determine if 

airframe icing was having an effect on 

aircraft characteristics.

–3.0987 0.00008 PRD

2. Knew minimum safe speed for a given wing 

flap setting within 5 kn.

–2.2242 0.0072 PRD

3. Knew how to adjust pitch attitude to avoid a 

wing stall or a tail stall upset.

–2.5750 0.0047 PRD

4. Knew wing flap settings for safe rate of 

climb in event of an engine failure.

–2. 5968 0.0044 PRD

5. Had to rely on aircraft control response to 

determine icing effects on pitch, yaw, and 

roll.

–4.4298 0.0000 PRD

6. Confident that the final approach airspeed 

would prevent stall. –0.3273 0.3634 B

7. Able to avoid aircraft handling problems or 

tail stall upsets when selecting wing flaps 

down.

1.1129 0.8717 W

8. Colored bands on airspeed tape useful to 

safely fly the aircraft during the entire flight.

–2.7714 0.0026 PRD

9. Relied on stick shaker to prevent 

inadvertently stalling. –0.1746 0.4221 B

10. Always knew when approaching a wing or 

tail stall condition. –1.8985 0.0274 PRD

Legend

Meets 95% confidence for rejecting H0

Experimental group better over control group but < 95% confidence to reject H0

No difference between experimental and control group

Experimental group performance worse than control group

Figure 8.—Situational awareness post-test questionnaire results.

aircraft control response. From the responses to question 8 the experimental group of pilots felt the 

colored bands on the airspeed tape were useful for safely flying the aircraft during the entire flight, while 

the control group did not. The histogram of the responses, Figure B.3, clearly reflects this difference, but 

the result was somewhat confusing since both groups had the same color bands on their airspeed tape 

(Figure 2), and a real difference was not expected. One explanation however, may have been that the 

ICEPro speed carets, when overlaid on the baseline PFD display speed tape, provided better low and high 

speed awareness in relation to the non-iced aircraft condition. Another possibility is that the experimental 

group misinterpreted the question and thought the “color bands” referred to the ICEPro speed carets. In 

any case, the wording of the question could have been better to ensure that no misinterpretation of the 

nomenclature was made. Question 10 indicated that the experimental group felt they had significantly 

better awareness when approaching a flight condition that could lead to a wing or tail stall than the control 

group. Many of the experimental pilots verbally commented during their test runs that the real time 

computed AOA brackets were most useful in that respect. The responses to questions 6 and 9 

respectively, indicated that the experimental group was more confident that their final approach airspeed 

would avoid wing stall, and that they did not have to rely as much on the shaker to avoid stalling. These 

results, although better for the experimental group, did not meet the significance criteria. This better but 

not significantly better result may have been due to the fact that the experimental group knew that they 
NASA/TM—2011-216951 19

could safely fly to just below the AOA upper bracket and avoid the shaker firing. If the brackets started 

flashing coincident with the aircraft reference symbol and the shaker did fire, they could immediately 

reduce AOA and avoid stall. The combination of AOA and shaker gave the experimental group good low 

speed and incipient stall awareness. The control group on the other hand, had to rely on basic (clean 

aircraft) shaker settings and the likewise basic (clean aircraft) airspeed bands on their PFD to avoid 

stalling.

Fortunately for the control group, the failure case icing configuration in the simulation model resulted 

in a very small stall speed increase and concurrent stall AOA decrease. As a result, if the shaker fired for 

the control group, or if they flew into the amber stall warning band on the airspeed indicator, they were 

generally able to react with a decreased pitch input in time to avoid stall. Had a more severe ice shape 

been modeled and the stall margin been less, or even negative, the control group responses may have been 

different as they could have stalled before the shaker fired. The responses to question 7 indicated that the 

experimental group felt less able to avoid handling problems or tail stall upsets while extending the wing 

flaps than the control group. Referring to Figure B.3, it is apparent that the greatest number of responses 

from the control group agreed that they could manage their control inputs to avoid handling problems, 

while there were an equivalent number of disagreement responses from the experimental group. This 

result was somewhat puzzling because it was apparent from the data in Figure 5 that quantitatively, the 

experimental group was significantly better than the control group in avoiding tail stalls. One possible 

explanation is that the experimental group was provided with displays that clearly showed the effects of 

wing flap extension on the safe AOA envelope, but were required by the test protocol to ignore those 

indications and extend the wing flaps regardless. This could well have led to their response choice 

because they expected handling problems as wing flaps were extended, and knew that they were 

unavoidable. The question may have been poorly worded in that respect.

5.6.3 Part III—ICEPro Integration

The experimental group was asked fifteen questions regarding the integration of ICEPro displays.

Since these questions only affected that group, a descriptive statistical approach was taken in the analysis 

of the data. The primary goal of these survey questions was to understand if the implementation of real 

time state assessment into flyable pilot cues did in fact provide utility of ICEPro for the pilot. The key 

findings that follow list the percentage of responses on either side of a “neutral” opinion. 

1. 40 percent agreed, 40 percent disagreed, that messages were not useful and cluttered the display

2. 53 percent agreed, 33 percent disagreed that combining visual, aural and tactile cues was important

3. 60 percent agreed, 40 percent disagreed that bars were useful to fly safely

4. 70 percent agreed, 20 percent disagreed that it was difficult to understand relation between AOA 

bars and Hi/Lo airspeed carets

5. 40 percent agreed, 40 percent disagreed that synoptic page was more useful than messages for state 

assessment 

6. 47 percent frequently or always, 40 percent never or infrequently favored using airspeed carets over 

AOA bars for safe flight

7. 40 percent agreed, 47 percent disagreed that decluttering did not affect ability to recover from stall 

8. 40 percent agreed, 47 percent disagreed that the “red” CLM and FLP LIM needed less flap setting 

for engine out performance.

9. 47 percent agreed, 40 percent disagreed that “control buzzing” made aircraft control difficult

10. 60 percent agreed, 30 percent disagreed that baseline airspeed tapes and carets were confusing

11. 60 percent frequently or always, 40 percent never or infrequently flew in middle of AOA bars

12. 47 percent frequently or always, 30 percent never or infrequently felt that AOA bars immediately 

captured attention when flashing.

13. 40 percent frequently or always, 47 percent never or infrequently felt airspeed cues were helpful for 

maneuvering/approach.
NASA/TM—2011-216951 20

14. 60 percent frequently or always, 40 percent never or infrequently felt low α cue helped avoid tail 

stall upset

15. 67 percent agreed, 20 percent disagreed that they immediately noticed when CLM and FLAP LIM 

messages were on at the same time

Summarizing these key findings, the experimental group felt that the AOA brackets were very useful 

in all phases of flight, and were effectively able to use the brackets to keep the aircraft in a safe flight 

envelope. But many had difficulty understanding the relationship between angle of attack and airspeed.

None of the participating pilots had experience with flying angle of attack prior to this test, and it is felt 

that this response was largely the result of this lack of experience. More pilots felt that presenting both the 

baseline airspeed along with the ICEPro generated airspeed carets was confusing. The original design 

intent of the ICEPro displays was to present the computed real time speeds as “advisory” information.

This way, the pilot would always have a reference to the baseline aircraft, which would prove useful if a 

system anomaly was encountered. There was a concern for the manner in which this would be done to 

avoid confusion and the survey seemed to bear this concern out. Along those same lines, pilots were 

essentially ambivalent as to the usefulness of the ICEPro speed carets when maneuvering and flying the 

approach task as shown in Figure 6. A majority of pilots felt that the integration of visual, aural, and 

tactile cues was important, but providing redundant messages on two displays was not necessary. Lastly, 

one major concern was the assessment of the multi-axis control excitation or “buzzing”, which was 

commanded by the ICEPro logic for aircraft state assessment. Some pilots were bothered by the buzzing, 

and some were not. It appeared that because it was expected, pilots accommodated to this as they would if 

having to fly through turbulence. In any case, there did not seem to be a strong opinion one way or the 

other regarding control buzzing.

5.6.4 Part IV—Workload

Workload assessments via the NASA TLX format were gathered after each run for each pilot. Posttest questionnaires were solicited in an attempt to understand some of the important factors that 

influenced the scores that pilots gave to workload assessment. The survey answers from the control and 

experimental was assessed via the MWU methodology. Figure 9 provides the results from the survey 

questionnaire.

Pilot workload was not measurably affected by ICEPro cueing. The data indicated the following:

1. Controlling vertical speed on glide slope was more difficult for experimental group (60 versus

43 percent). It appears that the control group prioritized safe flight over strict airspeed control.

2. Flight control inputs were more physically demanding for the control group (73 versus 60

percent), possibly due to the fact that the control group was not aware of the magnitude of angle 

of attack divergence. 

3. The control group reported that it took more concentration and effort for flying MAP safely (87 

versus 80 percent). The results however did not meet the significance criterion, but did show that 

the experimental group did better.

4. Staying on glide slope without pitch upset was more difficult for control group (79 versus

67 percent). This was likely because the control group made corrections without knowledge of 

AOA limits. On the other hand, the experimental group had a real time display of AOA limits and 

could remain within those limits while correcting to the glide path. By “respecting” these limits, 

the experimental group tended to give greater priority to remaining in a safe flight conditions and 

were less aggressive in maintaining a precise vertical flight path. It should be noted however, that 

both groups flew the approaches within ATP standards as previously discussed.

5. Maintaining glide slope detracted from localizer control. This affected the experimental group 

more than the control group (64 versus 47 percent). This is thought to be a result of AOA cueing, 

which captured the pilot’s attention.
NASA/TM—2011-216951 21

Pilot post-test survey 

questionnaire results

z score:

z < -1.69 for 95% 

confidence in H0

rejection

Probability:

p < 0.05 required to 

satisfy α (alternate 

hypothesis)

Exp. versus Control

Positive real difference 

(PRD), Better (B), 

Worse (W), Same (S)

Part IV—Workload

1. It was difficult to control vertical speed 

when on glide slope. 0.7856 0.7902 W

2. Operating the flight controls to avoid an 

upset condition when on final approach 

was a physically demanding effort.

–0.5455 0.2852 B

3. I felt it took a considerable amount of 

concentration and effort to safely execute 

the missed approach procedure.

–1.2002 0.1108 B

4. Keeping the aircraft on the glide slope 

without experiencing a pitch upset was a 

very demanding task.

–0.8074 0.2034 B

5. On final approach I spent so much time 

trying to fly the glide slope that I was 

unable to maintain good localizer course 

control.

0.5674 0.7221 W

Legend

Meets 95% confidence for rejecting H0

Experimental group better over control group but < 95% confidence for rejecting H0

No difference between experimental and control group

Control group performance better than experimental group

Figure 9.—Pilot workload survey results.

In summary, the workload was not measurably affected by the addition of ICEPro system. Pilots 

seemed to accommodate to those displays rather quickly during their training, and appeared to use them 

quite effectively to perform the flight task. TLX data showed clearly that the flight task was very 

demanding for both groups of pilots, and the physical difficulty of it far outweighed other workload 

dimensions.

6.0 Conclusions and Recommendations

A pilot-in-the-loop simulator evaluation was conducted to assess the utility of ICEPro for mitigating a 

hazardous icing encounter. Most of the results of this testing were statistically quantified for real 

differences between a control and experimental group by the Mann-Whitney “U” test, which is a nonparametric, independent measures analysis of data used to define pilot control performance, pilot opinion, 

and workload. Other data, which was not compared between two groups, was analyzed descriptively. The 

conclusions from this test indicate that the experimental group of pilots, who flew with ICEPro displays, 

demonstrated significantly better control performance in dealing with aircraft handling problems under 

icing conditions and had better awareness of a hazardous aircraft state. Pilot workload was not affected by 

the additional display information provided by ICEPro, but pilot opinions of the implementation of that 

system indicate that improvements could be made. Based on the findings of this test, future testing with 

ICEPro should include the effects of environmental uncertainties, such as in-flight turbulence and wind 

gusts, on the quality-of-state assessment. Additionally, the flight displays in this test were not optimized 

from a human factors standpoint, nor were they coupled in any way with flight guidance. Based upon the 

opinions extracted from the pilot survey, pilot control performance and state awareness might be 

improved by addressing those issues in future testing.

NASA/TM—2011-216951 23

Appendix A.—Post-Test Survey Questionnaire

Part I. Demographics

Name: _______________________ Male Female

Pilot Number: __________

Total Flight Hours: __________

Multi – Engine Hours: _________

FAA Ratings: Commercial Multi CFI CFII

Aircraft Flown and Hours in Type:

_____________________________________________________________________________________

_____________________________________

Icing related training and experience:

1. In my all my prior flying experience, I encountered in-flight icing 

Never Rarely Sometimes Very often Always

2. I felt that my prior icing related knowledge and experience would have adequately prepared me to 

perform this test scenario without the familiarization training I received on aircraft handling 

characteristics.

Strongly Disagree Disagree Undecided Agree Strongly Agree

3. The NASA videos and web based training materials provided me with information about icing that I 

had never known about before.

Strongly Disagree Disagree Undecided Agree Strongly Agree

4. Before this test, my icing related flight training was mostly focused on how to operate the ice 

protection system.

Strongly Disagree Disagree Undecided Agree Strongly Agree

Part II. Situation Awareness

1. My flight displays were adequate for me to determine when airframe icing was having an effect on 

aircraft characteristics.

Never Rarely Sometimes Very often Always

2. I felt as though I knew the minimum safe speed I could fly for a given wing flap setting within 5 knots. 

Never Rarely Sometimes Very often Always

3. I knew how to adjust my pitch attitude to avoid a wing stall or a tail stall upset. 

Never Rarely Sometimes Very Often Always

4. I wasn’t always sure which wing flap settings would allow a safe rate of climb in the event of an engine 

failure.

Strongly Disagree Disagree Undecided Agree Strongly Agree

5. I relied solely on aircraft control response to determine how icing affected pitch, yaw, or roll 

characteristics.

Strongly Disagree Disagree Undecided Agree Strongly Agree

6. I was confident that the final approach airspeed I chose to fly after I departed the final approach fix 

would prevent me from stalling.
NASA/TM—2011-216951 24

Strongly Disagree Disagree Undecided Agree Strongly Agree

7. While I was putting the flaps down during the approach and landing, I felt I could effectively manage 

my control inputs and airspeed to avoid aircraft handling problems or tail stall upsets.

Strongly Disagree Disagree Undecided Agree Strongly Agree

8. The various colored bands on the airspeed tape were useful for me to safely fly the aircraft during the 

entire flight. 

Never Infrequently Sometimes Frequently Always

9. I relied upon the stick shaker to prevent me from inadvertently stalling.

Never Rarely Sometimes Very Often Always

10. I could always tell when I was approaching a wing or tail stall condition. 

Strongly Disagree Disagree Undecided Agree Strongly Agree

Part III. Implementation of Envelope Protection Cueing (Experimental ICEPro Group Only)

1. I felt that it was important to combine the visual cues from ICEPro with aural alerts, and tactile 

feedback from the stick shaker. 

Strongly Disagree Disagree Undecided Agree Strongly Agree

2. The ROL DGRD, PTCH DGRD, or YAW DGRD messages on the PFD did not provide useful 

information, and only cluttered the display.

Strongly Disagree Disagree Undecided Agree Strongly Agree

3. When flying at slow speeds, I could easily use the AOA bars as a good pitch control reference for 

safely flying the airplane.

Strongly Disagree Disagree Undecided Agree Strongly Agree

4. I had difficulty understanding the relationship between the AOA bars and the high and low speed carets 

on the airspeed tape. 

Strongly Disagree Disagree Undecided Agree Strongly Agree

5. The color coded flight control surfaces on the flight control synoptic were more useful than the 

messages for assessing degraded control state. 

Strongly Disagree Disagree Undecided Agree Strongly Agree

6. I favored using the airspeed carets rather than the AOA bars to remain within a safe operating 

condition.

Never Infrequently Sometimes Frequently Always

7. When I unintentionally got into an upset condition, the disappearance of all ICEPro messages except 

the AOA bars on the PFD did not affect my ability to recover the airplane. (Answer only if a wing stall or 

tail stall upset occurred).

Strongly Disagree Disagree Undecided Agree Strongly Agree

8. When I extended the wing flaps and the red CLIMB LIM and FLAP LIM messages came on together, I 

immediately knew if an engine failed I would have to reduce my flap setting in order to climb. 

Strongly Disagree Disagree Undecided Agree Strongly Agree
NASA/TM—2011-216951 25

9. The occasional “buzzing” of the flight controls by ICEPro made flying the aircraft very difficult.

Strongly Disagree Disagree Undecided Agree Strongly Agree

10. I tended to confuse baseline airspeed limits with the airspeed limits (carets) that were posted by 

ICEPro.

Strongly Disagree Disagree Undecided Agree Strongly Agree

11. I tended to fly so as to keep my pitch attitude in the middle of the AOA bars when they were 

presented by ICEPro

Never Infrequently Sometimes Frequently Always

12. When the AOA bars started flashing, they immediately captured my attention.

Never Infrequently Sometimes Frequently Always

13. I felt that the airspeed carets from ICEPro were useful for helping me determine safe flight speeds 

when maneuvering during approach and landing.

Never Infrequently Sometimes Frequently Always

14. The low AOA cue on the PFD enabled me to avoid a tail stall upset.

Never Infrequently Sometimes Frequently Always

15. I immediately noticed when the CLIMB LIM and FLAP LIM messages were on at the same time.

Strongly Disagree Disagree Undecided Agree Strongly Agree

Part IV. Workload 

1. I found it difficult to control vertical speed when on glide slope.

Never Infrequently Sometimes Frequently Always

2. Operating the flight controls to avoid an upset condition when on final approach was a physically 

demanding effort.

Never Infrequently Sometimes Frequently Always

3. I felt it took a considerable amount of concentration and effort to safely execute the missed approach 

procedure.

Strongly Disagree Disagree Undecided Agree Strongly Agree

4. Keeping the aircraft on the glide slope without experiencing a pitch upset was a very demanding task.

Strongly Disagree Disagree Undecided Agree Strongly Agree

5. On final approach I spent so much time trying to fly the glide slope that I was unable to maintain good 

localizer course control.

Never Infrequently Sometimes Frequently Always

NASA/TM—2011-216951 27

Appendix B.—Results of Survey Questions from Parts I, II, and IV

Figure B.1.—Pilot demographics—Part I Survey questions 1 to 4.

Figure B.2.—Situational Awareness—Part II Survey questions 1 to 5

0

1

2

3

4

5

6

7

8

9

10

11

12

1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

Response Frequency

I-1 I-2 I-3 I-4

Control Group (n=14)

Experimental Group (n=15)

0

1

2

3

4

5

6

7

8

9

10

11

12

1 2 3 4 5* 1 2 3 4 5* 1 2 3 4 5* 1* 2 3 4 5 1* 2 3 4 5

Response Frequency

II-1 II-2 II-3 II-4 II-5 

Control Group (n=14)

Experimental Group (n=15)

*Direction of responses that indicated better situational awareness
NASA/TM—2011-216951 28

Figure B.3.—Situational Awareness—Part II Survey questions 6 to 10

Figure B.4.—Workload—Part IV Survey questions 1 to 5.

0

1

2

3

4

5

6

7

8

9

10

11

12

1 2 3 4 5* 1 2 3 4 5* 1 2 3 4 5* 1* 2 3 4 5 1 2 3 4 5*

Response Frequency

 II-6 II-7 II-8 II-9 II- 10 

Control Pilot Group (n=14)

Experimental Group (n=15)

* Direction of responses that indicated better situational awareness

0

1

2

3

4

5

6

7

8

9

10

11

12

1* 2 3 4 5 1* 2 3 4 5 1* 2 3 4 5 1* 2 3 4 5 1* 2 3 4 5

Response Frequency

IV-1 IV-2 IV-3 IV-4 IV-5 

Control Group (n=14)

Experimental Group (n=15)

* Direction of responses that indicated lower workload
NASA/TM—2011-216951 29

References

1. Gingras, David R., Barnhart, Billy, Ranaudo, Richard J., Ratvasky, Thomas P., Morelli, Eugene, 

“Envelope Protection for In-Flight Ice Contamination,” 47th AIAA Aerospace Sciences Meeting, 

Jan. 5–8, 2009, AIAA–2009–1458, NASA/TM—2010-216072.

2. Barnhart, Billy P., Dickes, Edward G., Gingras David R., Ratvasky, Thomas P., “Simulation Model 

Development for Icing Effects Flight Training,” NASA/TM—2003-212115, General Aviation 

Technology Conference and Exhibition 2002 sponsored by the Society of Automotive Engineers, 

Wichita, KS, Apr. 16–18, 2002, SAE–2002–01–1527.

3. Ratvasky, Thomas P., Blankenship, Kurt, Rieke, William J., and Brinker, David J., “Iced Aircraft 

Flight Data for Flight Simulator Validation,” NASA/TM—2003-212114, General Aviation 

Technology Conference and Exhibition 2002, Society of Automotive Engineers, Wichita, KS, 

Apr. 16–18, 2002, SAE–2002–01–1527.

4. Ratvasky, T.P, Ranaudo, R.J., Barnhart, B.P., Dickes, E.G, and Gingras, D.R, “Development and

Utility of a Piloted Flight Simulator for Icing Effects Training,” AIAA 41st Aerospace Sciences 

Meeting and Exhibit, Jan. 6–9, 2003, Reno, NV, AIAA–2003–0022, NASA/TM—2003-212116.

5. Ratvasky, T.P., Ranaudo, R.J., Blankenship, K.S., Lee, S., “Demonstration of an Ice Contaminated 

Effects Flight Training Device,” AIAA 44th Aerospace Sciences Meeting and Exhibit, Jan. 9–12, 

2006, Reno, NV, AIAA–2006–0677, NASA/TM—2006-214233.

6. Hart, S., and Lowell, S., “Development of NASA-TLX: Results of Empirical and Theoretical 

Research,” in Hancock, P., and Meshkati, N., (eds.), Human Mental Workload, North Holland Press, 

Amsterdam, 1988, pp. 239–250.

7. Green, Stephen D., “A Study of U.S. Inflight Icing Accidents and Incidents, 1978 to 2002,” 44th

AIAA Aerospace Sciences Meeting and Exhibit, Jan. 9–12, 2006, Reno, NV, AIAA–2006–82.

8. Nagel, David C., “Human Error in Aviation Operations”, Human Factors in Aviation, Chapter 9, 

edited by Weiner, E.L, Nagel, D.C., Academic Press, 1988.

9. National Transportation Safety Board, In-Flight Icing Encounter And Loss Of Control, Simmons 

Airlines, D.B.A. American Eagle Flight 4184 Avions De Transport Regional (ATR) Model 72–212, 

N401am, Roselawn, IN, Oct. 31,1994, NTSB Aircraft Accident Report, NTSB/AAR–96/01, 

Washington, DC., Jul. 1996.

10. Ratvasky, T.P., Van Zante, J.F., Sim, A., “NASA/FAA Tailplane Icing Program: Flight Test Report,”

NASA TP–209908, DOT/FAA/AR–99/85, March, 2000.

11. Ranaudo, R.J., Mikkelsson, K.L., McKnight, R.C., Ide, R.F., and Rehoorst, A.L., “The Measurement 

of Aircraft Performance and Stability and Control After Flight Through Natural Icing Conditions,” 

AIAA Paper No. 86–9758.

12. Morelli, E.A. “Real-Time Parameter Estimation in the Frequency Domain,” Journal of Guidance, 

Control, and Dynamics, Vol. 23, No. 5, Sep.–Oct. 2000, pp. 812–818.

13. Klein, V. and Morelli, E.A. “Aircraft System Identification - Theory and Practice,” AIAA Education 

Series, AIAA, Reston, VA, Aug. 2006.

14. Morelli, E.A. and Smith, M.S. “Real-Time Dynamic Modeling – Data Information Requirements and 

Flight Test Results,” Journal of Aircraft, Vol. 46, No. 6, Nov.–Dec. 2009, pp. 1894–1905

15. Web reference, http://en.wikipedia.org/wiki/Theil_index [cited Aug. 14, 2009).

16. Morelli, E.A. “Flight-Test Experiment Design for Characterizing Stability and Control of Hypersonic 

Vehicles,” Journal of Guidance, Control, and Dynamics, Vol. 32, No. 3, May–Jun. 2009, pp. 949–959.

17. Sheskin, D.J., The Handbook of Parametric and Nonparametric Statistical Procedures, CRC Press, 

Boca Raton, FL, 1997, pp., 181–190

18. Trochim, William M., “Likert Scaling,” Research Methods Knowledge Base, 2nd Edition. Oct. 20, 

2006.

19. Federal Aviation Administration, “Airline Transport and Aircraft Type Rating Practical Test 

Standards for Airplane,” FAA–S–8081–5F, July 2008, pp 2-20–2-27.
REPORT DOCUMENTATION PAGE Form Approved 

OMB No. 0704-0188 

The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the 

data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this 

burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. 

Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB 

control number. 

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 

1. REPORT DATE (DD-MM-YYYY)

01-10-2011 

2. REPORT TYPE

Technical Memorandum 

3. DATES COVERED (From - To)

4. TITLE AND SUBTITLE

Piloted Simulation to Evaluate the Utility of a Real Time Envelope Protection System for 

Mitigating In-Flight Icing Hazards 

5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S)

Ranaudo, Richard, J.; Martos, Borja; Norton, Bill, W.; Gingras, David, R.; Barnhart, Billy, P.; 

Ratvasky, Thomas, P.; Morelli, Eugene 

5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

WBS 645846.02.07.11.02 

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

National Aeronautics and Space Administration 

John H. Glenn Research Center at Lewis Field 

Cleveland, Ohio 44135-3191 

8. PERFORMING ORGANIZATION

 REPORT NUMBER

E-17556 

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

National Aeronautics and Space Administration 

Washington, DC 20546-0001 

10. SPONSORING/MONITOR'S

 ACRONYM(S)

NASA 

11. SPONSORING/MONITORING

 REPORT NUMBER

NASA/TM-2011-216951 

12. DISTRIBUTION/AVAILABILITY STATEMENT

Unclassified-Unlimited 

Subject Categories: 05 and 08 

Available electronically at http://www.sti.nasa.gov 

This publication is available from the NASA Center for AeroSpace Information, 443-757-5802

13. SUPPLEMENTARY NOTES

14. ABSTRACT

The utility of the Icing Contamination Envelope Protection (ICEPro) system for mitigating a potentially hazardous icing condition was 

evaluated by 29 pilots using the NASA Ice Contamination Effects Flight Training Device (ICEFTD). ICEPro provides real time envelope 

protection cues and alerting messages on pilot displays. The pilots participating in this test were divided into two groups; a control group 

using baseline displays without ICEPro, and an experimental group using ICEPro driven display cueing. Each group flew identical precision 

approach and missed approach procedures with a simulated failure case icing condition. Pilot performance, workload, and survey 

questionnaires were collected for both groups of pilots. Results showed that real time assessment cues were effective in reducing the number 

of potentially hazardous upset events and in lessening exposure to loss of control following an incipient upset condition. Pilot workload with 

the added ICEPro displays was not measurably affected, but pilot opinion surveys showed that real time cueing greatly improved their 

situation awareness of a hazardous aircraft state. 

15. SUBJECT TERMS

Aircraft icing; Flight safety; Flight simulation training; Ice protection system 

16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF

 ABSTRACT

UU 

18. NUMBER

 OF 

 PAGES

35 

19a. NAME OF RESPONSIBLE PERSON

STI Help Desk (email:help@sti.nasa.gov) 

a. REPORT

U 

b. ABSTRACT

U 

c. THIS 

PAGE

U 

19b. TELEPHONE NUMBER (include area code)

443-757-5802 

Standard Form 298 (Rev. 8-98)

Prescribed by ANSI Std. Z39-18