Borja Martos, Richard Ranaudo, and Billy Norton

University of Tennessee Space Institute, Tullahoma, Tennessee 

David Gingras and Billy Barnhart

Bihrle Applied Research Inc., Hampton, Virginia 

Development, Implementation, and Pilot Evaluation 

of a Model-Driven Envelope Protection System to 

Mitigate the Hazard of In-Flight Ice Contamination 

on a Twin-Engine Commuter Aircraft 

NASA/CR—2014-218320

October 2014
1$6$67,3URJUDPLQ3UR¿OH

6LQFHLWVIRXQGLQJ1$6$KDVEHHQGHGLFDWHGWRWKH

DGYDQFHPHQWRIDHURQDXWLFVDQGVSDFHVFLHQFH7KH

1$6$6FLHQWL¿FDQG7HFKQLFDO,QIRUPDWLRQ67,

SURJUDPSOD\VDNH\SDUWLQKHOSLQJ1$6$PDLQWDLQ

WKLVLPSRUWDQWUROH

7KH1$6$67,3URJUDPRSHUDWHVXQGHUWKHDXVSLFHV

RIWKH$JHQF\&KLHI,QIRUPDWLRQ2I¿FHU,WFROOHFWV

RUJDQL]HVSURYLGHVIRUDUFKLYLQJDQGGLVVHPLQDWHV

1$6$¶V67,7KH1$6$67,SURJUDPSURYLGHVDFFHVV

WRWKH1$6$$HURQDXWLFVDQG6SDFH'DWDEDVHDQG

LWVSXEOLFLQWHUIDFHWKH1$6$7HFKQLFDO5HSRUWV

6HUYHUWKXVSURYLGLQJRQHRIWKHODUJHVWFROOHFWLRQV

RIDHURQDXWLFDODQGVSDFHVFLHQFH67,LQWKHZRUOG

5HVXOWVDUHSXEOLVKHGLQERWKQRQ1$6$FKDQQHOV

DQGE\1$6$LQWKH1$6$67,5HSRUW6HULHVZKLFK

LQFOXGHVWKHIROORZLQJUHSRUWW\SHV

 7(&+1,&$/38%/,&$7,215HSRUWVRI

FRPSOHWHGUHVHDUFKRUDPDMRUVLJQL¿FDQWSKDVH

RIUHVHDUFKWKDWSUHVHQWWKHUHVXOWVRI1$6$

SURJUDPVDQGLQFOXGHH[WHQVLYHGDWDRUWKHRUHWLFDO

DQDO\VLV,QFOXGHVFRPSLODWLRQVRIVLJQL¿FDQW

VFLHQWL¿FDQGWHFKQLFDOGDWDDQGLQIRUPDWLRQ

GHHPHGWREHRIFRQWLQXLQJUHIHUHQFHYDOXH

1$6$FRXQWHUSDUWRISHHUUHYLHZHGIRUPDO

SURIHVVLRQDOSDSHUVEXWKDVOHVVVWULQJHQW

OLPLWDWLRQVRQPDQXVFULSWOHQJWKDQGH[WHQWRI

JUDSKLFSUHVHQWDWLRQV

 7(&+1,&$/0(025$1'806FLHQWL¿F

DQGWHFKQLFDO¿QGLQJVWKDWDUHSUHOLPLQDU\RU

RIVSHFLDOL]HGLQWHUHVWHJTXLFNUHOHDVH

UHSRUWVZRUNLQJSDSHUVDQGELEOLRJUDSKLHVWKDW

FRQWDLQPLQLPDODQQRWDWLRQ'RHVQRWFRQWDLQ

H[WHQVLYHDQDO\VLV

 &2175$&7255(32576FLHQWL¿FDQG

WHFKQLFDO¿QGLQJVE\1$6$VSRQVRUHG

FRQWUDFWRUVDQGJUDQWHHV

 &21)(5(1&(38%/,&$7,21&ROOHFWHG

SDSHUVIURPVFLHQWL¿FDQGWHFKQLFDO

FRQIHUHQFHVV\PSRVLDVHPLQDUVRURWKHU

PHHWLQJVVSRQVRUHGRUFRVSRQVRUHGE\1$6$

 63(&,$/38%/,&$7,216FLHQWL¿F

WHFKQLFDORUKLVWRULFDOLQIRUPDWLRQIURP

1$6$SURJUDPVSURMHFWVDQGPLVVLRQVRIWHQ

FRQFHUQHGZLWKVXEMHFWVKDYLQJVXEVWDQWLDO

SXEOLFLQWHUHVW

 7(&+1,&$/75$16/$7,21(QJOLVK

ODQJXDJHWUDQVODWLRQVRIIRUHLJQVFLHQWL¿FDQG

WHFKQLFDOPDWHULDOSHUWLQHQWWR1$6$¶VPLVVLRQ

6SHFLDOL]HGVHUYLFHVDOVRLQFOXGHFUHDWLQJFXVWRP

WKHVDXULEXLOGLQJFXVWRPL]HGGDWDEDVHVRUJDQL]LQJ

DQGSXEOLVKLQJUHVHDUFKUHVXOWV

)RUPRUHLQIRUPDWLRQDERXWWKH1$6$67,

SURJUDPVHHWKHIROORZLQJ

 $FFHVVWKH1$6$67,SURJUDPKRPHSDJHDW

http://www.sti.nasa.gov

 (PDLO\RXUTXHVWLRQWRhelp@sti.nasa.gov

 )D[\RXUTXHVWLRQWRWKH1$6$67,

,QIRUPDWLRQ'HVNDW±±

 3KRQHWKH1$6$67,,QIRUPDWLRQ'HVNDW

 ±±

 :ULWHWR

67,,QIRUPDWLRQ'HVN

 1$6$&HQWHUIRU$HUR6SDFH,QIRUPDWLRQ

6WDQGDUG'ULYH

+DQRYHU0'±
Borja Martos, Richard Ranaudo, and Billy Norton

University of Tennessee Space Institute, Tullahoma, Tennessee 

David Gingras and Billy Barnhart

Bihrle Applied Research Inc., Hampton, Virginia 

Development, Implementation, and Pilot Evaluation 

of a Model-Driven Envelope Protection System to 

Mitigate the Hazard of In-Flight Ice Contamination 

on a Twin-Engine Commuter Aircraft 

NASA/CR—2014-218320

October 2014

1DWLRQDO$HURQDXWLFVDQG

6SDFH$GPLQLVWUDWLRQ

*OHQQ5HVHDUFK&HQWHU

&OHYHODQG2KLR

3UHSDUHGXQGHU*UDQW11;$'$
Acknowledgments

7KHGHYHORSPHQWDQGWHVWLQJRI ,&(3URZDV IXQGHGXQGHUD IHGHUDOJUDQWZKLFKZDVDVSRQVRUHG\HDUFRRSHUDWLYH UHVHDUFKHIIRUWZLWK

1$6$XQGHUWKH1$6$5HVHDUFK$QQRXQFHPHQW11+=($1$SSHQGL[%RIWKH$YLDWLRQ6DIHW\3URJUDP7KHREMHFWLYHVRIWKH15$

ZHUHGH¿QHGXQGHUWKH ,QWHJUDWHG9HKLFOH+HDOWK0DQDJHPHQW3URMHFWWRSLF ,9+0(QYLURQPHQWDO+D]DUGVZKLFKDUHFDXVHGE\WKH

3(IIHFWVRI,FLQJRQ$LUFUDIW6WDWH ́7KHVXFFHVVIXOGHYHORSPHQWWHVWLQJDQGHYDOXDWLRQRI,&(3URZRXOGQRWKDYHEHHQSRVVLEOHZLWKRXWWKH

FRQWULEXWLRQVRINH\LQGLYLGXDOVDQGRUJDQL]DWLRQVZKLFKPDGHXSWKHUHVHDUFKWHDP7KHDXWKRUZLVKHVWRUHFRJQL]H0U5LFKDUG5DQDXGR

8QLYHUVLW\RI7HQQHVVHH6SDFH,QVWLWXWHZKRZDVSULQFLSDOLQYHVWLJDWRURIWKHRULJLQDO15$DQGZURWHDOORIWKHUHVXOWVWKHUHRILQWKLVUHSRUW

7KHDXWKRUDOVRZLVKHVWRUHFRJQL]H0U7KRPDV5DWYDVN\1$6$*OHQQ5HVHDUFK&HQWHUDQG'U(XJHQH0RUHOOL1$6$/DQJOH\5HVHDUFK

&HQWHUIRUWKHLUJXLGDQFHDQGVXSSRUWWKURXJKRXWWKLVSURMHFW0U5DWYDVN\¶VH[SHULHQFHDQGWHFKQLFDOH[SHUWLVHZLWKLFLQJUHODWHGSUREOHPV

ZHUHHVVHQWLDOGXULQJWKHGHYHORSPHQWDQGWHVWLQJSKDVHVRI,&(3UR'U(XJHQH0RUHOOLSURYLGHGFULWLFDOO\LPSRUWDQWWHFKQLFDODVVLVWDQFHGXULQJ

WKHLQWHJUDWLRQDQGDSSOLFDWLRQRIWKH5HDO7LPH3DUDPHWHU,GHQWL¿FDWLRQDOJRULWKPZKLFKKHDQGRWKHUVIURPKLVRUJDQL]DWLRQKDGGHYHORSHG

RYHUDSHULRGRIVHYHUDO\HDUVVHH5HIVWR,QDGGLWLRQ0U'DYLG*LQJUDVDQG0U%LOO\%DUQKDUW%LKUOH$SSOLHG5HVHDUFKZKRDUH

H[SHUWVLQWKH¿HOGRIPRGHOLQJDQGVLPXODWLRQVXFFHVVIXOO\GHVLJQHGGHYHORSHGLQWHJUDWHGDQGWHVWHGWKH,&(3URV\VWHPLQ1$6$¶V,&()7'

0U%DUQKDUWZDVGLUHFWO\LQYROYHGGXULQJWKHWHVWDQGHYDOXDWLRQSKDVHRI,&(3URDQGDORQJZLWK0U*LQJUDVSURYLGHGLPSRUWDQWVLPXODWLRQ

IXQFWLRQDOLW\DQGVROYHGQXPHURXVWHFKQLFDODQGLQWHJUDWLRQSUREOHPVQHFHVVDU\IRUDVXFFHVVIXOHYDOXDWLRQ0U%LOO\1RUWRQDQHQJLQHHULQJ

PDQDJHPHQWGRFWRUDOFDQGLGDWHDW876,GHYHORSHGDQGFRQGXFWHGWKHVWDWLVWLFDOGDWDKDQGOLQJPHWKRGVZKLFKZHUHHVVHQWLDOWRWKHDQDO\VLVRI

SLORWSHUIRUPDQFHPHWULFV/DVWO\WKHDXWKRUVZLVKWRH[SUHVVWKHLUVLQFHUHDSSUHFLDWLRQWRWKHIDFXOW\DQGVWDIIRIWKH(PEU\5LGGOH$HURQDXWLFDO

8QLYHUVLW\IRUWKHLUVXSSRUWRIWKLVWHVWLQJDWWKHLU(DJOH:RUNVIDFLOLW\LQ'D\WRQD%HDFK)ORULGD7KHDGGLWLRQDO\HDUH[WHQVLRQWRVWXG\WKH

HIIHFWVRIDWPRVSKHULFWXUEXOHQFHRQ,&(3URZRXOGQRWKDYHEHHQSRVVLEOHZLWKRXWWKHKHOSRI1$6$¶V9667SURMHFW7KHDXWKRUVZRXOGOLNH

WRWKDQN'U&KULVWLQH%HOFDVWURDQGKHUWHDPIRUWKLVDGGLWLRQDOVSRQVRUVKLSDQGWHFKQLFDOVXSSRUW

$YDLODEOHIURP

1$6$&HQWHUIRU$HURVSDFH,QIRUPDWLRQ

6WDQGDUG'ULYH

+DQRYHU0'±

1DWLRQDO7HFKQLFDO,QIRUPDWLRQ6HUYLFH

6KDZQHH5RDG

$OH[DQGULD9$

$YDLODEOHHOHFWURQLFDOO\DWKWWSZZZVWLQDVDJRY

7UDGHQDPHVDQGWUDGHPDUNVDUHXVHGLQWKLVUHSRUWIRULGHQWL¿FDWLRQ

RQO\7KHLUXVDJHGRHVQRWFRQVWLWXWHDQRI¿FLDOHQGRUVHPHQW

HLWKHUH[SUHVVHGRULPSOLHGE\WKH1DWLRQDO$HURQDXWLFVDQG

6SDFH$GPLQLVWUDWLRQ

Level of Review7KLVPDWHULDOKDVEHHQWHFKQLFDOO\UHYLHZHGE\1$6$WHFKQLFDOPDQDJHPHQW25H[SHUWUHYLHZHUV
NASA/CR—2014-218320 iii 

Contents 

Abstract ......................................................................................................................................................... 1

1.0 Introduction ........................................................................................................................................... 1

2.0 ICEPro Concept .................................................................................................................................... 4

3.0 Flight Characteristic Predictions ........................................................................................................... 7

4.0 Dynamics Inversion Control Evaluation System .................................................................................. 8

5.0 Real-Time Model Estimates ................................................................................................................. 9

6.0 System Implementation ...................................................................................................................... 11

7.0 Engineering Station ............................................................................................................................ 12

8.0 System Calibration .............................................................................................................................. 14

8.1 Icing Severity Parameter Determination..................................................................................... 14

8.2 Thresholds and Latching............................................................................................................. 15

8.3 RT-PID Reset ............................................................................................................................. 15

8.4 Conditions for Warning/Caution Messages ................................................................................ 15

8.5 Stall Warning and Decluttering .................................................................................................. 16

8.6 Accommodations for Pilot Evaluations ...................................................................................... 17

9.0 Pilot in the Loop Test Methodology ................................................................................................... 17

9.1 Experiment Design ..................................................................................................................... 17

9.2 ICEFTD Simulator Description and Capability.......................................................................... 17

9.3 Pre-Test Pilot Proficiency Training ............................................................................................ 20

9.4 Simulator Flight Evaluations ...................................................................................................... 21

9.5 Data Acquisition and Handling .................................................................................................. 22

10.0 Pilot in the Loop Results and Discussion ........................................................................................... 23

10.1 Pilot Demographics .................................................................................................................... 23

10.2 Pilot Control Performance .......................................................................................................... 23

10.2.1 Shaker Events—Stall Warning ....................................................................................... 24 

10.2.2 Pusher Events—Wing Stall Upsets ................................................................................. 25 

10.2.3 Stall Messages ................................................................................................................. 26 

10.2.4 Tail-Stall Upsets and Recoveries .................................................................................... 26

10.3 Pilot Precision Approach Performance ....................................................................................... 26

10.4 Exposure to Risk of an Out of Control Event ............................................................................. 28

10.5 Workload .................................................................................................................................... 29

10.6 Post-Test Survey Results ............................................................................................................ 29

10.6.1 Part I—Demographics ..................................................................................................... 30 

10.6.2 Part II—Situation Awareness .......................................................................................... 30 

10.6.3 Part III—ICEPro integration ........................................................................................... 32 

10.6.4 Part IV—Workload ......................................................................................................... 33 

11.0 Atmospheric Turbulence Test Methodology ...................................................................................... 35

11.1 Experiment Design ..................................................................................................................... 35

11.2 Task Descriptions ....................................................................................................................... 35

11.2.1 Task 1: Simulating Real World RT-PID Performance in Turbulence ............................ 35 

11.2.2 Task 2: Effect of Sensor Noise and Turbulence .............................................................. 35 

11.2.3 Task 3: Effect of Turbulence for Different Flight Phases at Selected Noise Level ........ 35 

11.2.4 Task 4: Evaluate the Effectiveness of Manual Pilot Inputs Alone for Making High 

Confidence Vehicle State Estimates When Performing Typical Maneuvering Flight 

Tasks Under Varying Levels of Atmospheric Turbulence ............................................. 35 

11.3 Measures of Success ................................................................................................................... 36

11.4 Changes to ICEPro Logic ........................................................................................................... 36

11.5 Atmospheric Turbulence Probe .................................................................................................. 37

12.0 Atmospheric Turbulence Results and Discussion ............................................................................... 37
NASA/CR—2014-218320 iv 

12.1 Task 1 Simulating Real World RT-PID Performance in Turbulence ......................................... 37

12.2 Task 2: Effect of Sensor Noise and Turbulence ......................................................................... 38

12.3 Task 3 Effect of Turbulence for Different Flight Phases at Selected Noise Level ..................... 39

13.0 Conclusions and Recommendations ................................................................................................... 40

13.1 ICEPro system design and implementation ................................................................................ 40

13.2 Pilot in the loop test and evaluation of ICEPro .......................................................................... 40

13.3 Atmospheric Turbulence and Sensor Noise................................................................................ 40

14.0 Recommendations ............................................................................................................................... 41

References ................................................................................................................................................... 41

Appendix A.—Symbols List ....................................................................................................................... 43

Appendix B.—Post-Test Survey Questionnaire ......................................................................................... 45

Appendix C.—Results of Survey Questions from Parts I, II, and IV ......................................................... 51

Appendix D.—One Year Atmospheric Turbulence Extension Final Report—Effect of 

Turbulence and Sensor Noise On ICEPro .......................................................................................... 53

D.1 Abstract ....................................................................................................................................... 56

D.2 Introduction ................................................................................................................................ 56

D.2.1 Research Background ..................................................................................................... 56 

D.2.2 Research Objectives ........................................................................................................ 58 

D.3 Discussion of Study and Results................................................................................................. 59

D.3.1 Task 1 .............................................................................................................................. 61 

D.3.2 Task 2 .............................................................................................................................. 76 

D.3.3 Task 3 .............................................................................................................................. 90 

D.4 Conclusions and Recommendations ......................................................................................... 103

D.4.1 Summary of Results ...................................................................................................... 103 

D.4.2 Additional Questions Raised by Research .................................................................... 105 

D.4.3 Recommended Future Studies ...................................................................................... 106 

D.5 References ................................................................................................................................ 106

Appendix E.—ICEPro Atmospheric Turbulence Test Plan ...................................................................... 107

E.1 Abstract ..................................................................................................................................... 108

E.2 Introduction .............................................................................................................................. 108

E.2.1 Background ................................................................................................................... 108 

E.2.2 Participating Organizations ........................................................................................... 108 

E.2.3 Overview of Test Concepts ........................................................................................... 108

E.2.4 Outline of Document ..................................................................................................... 109 

E.3 Simulator Flight Tests .............................................................................................................. 109

E.3.1 Test Objectives .............................................................................................................. 109 

E.3.2 Test Guidelines ............................................................................................................. 109 

E.3.3 Performance Parameters to be Measured ...................................................................... 110 

E.3.4 Measures of Success (MOS) ......................................................................................... 110 

E.3.5 Organizational Responsibilities .................................................................................... 111 

E.4 Simulator Flight Test Tasks ...................................................................................................... 111

E.4.1 Task 1 (UTSI) ............................................................................................................... 111 

E.4.2 Primary Objective—Performance Testing in Turbulence ............................................. 111 

E.4.3 Primary Objective—Development of Algorithms for Atmospheric Turbulence .......... 111 

E.4.4 Secondary Objective—Preliminary Feasibility Study without OBES .......................... 112 

List of Tables 

Table 1.—Aerodynamic Parameters Identified by RT-PID ........................................................................ 10

Table 2.—Summary of Pilot Control Performance..................................................................................... 24

Table 3.—Normalized Shaker Events ......................................................................................................... 25
NASA/CR—2014-218320 v 

Table 4.—Normalized Pusher Events ......................................................................................................... 25

Table 5.—Pilot Precision Approach Performance ...................................................................................... 27

Table 6.—Exposure to Risk Based Upon Pilot Coupling Events ............................................................... 28

Table 7.—Workload Assessment ................................................................................................................ 29

Table 8.—Situational Awareness Post-Test Questionnaire Results ........................................................... 30

Table 9.—Pilot Workload Survey Results .................................................................................................. 34

Table D.1.—AIMMS-20 probe performance.............................................................................................. 60

List of Figures

Figure 1.—ICEPro Envelope Protection Cues on the Primary Flight Display (left) and Flight 

Control Status Page (right) ................................................................................................................... 4

Figure 2.—ICEPro System Architecture ...................................................................................................... 5

Figure 3.—Simplified ICEPro System Flow Chart....................................................................................... 6

Figure 4.—Process used for Aerodynamic Model Validation ...................................................................... 7

Figure 5.—Example of Comparisons of the Simulation Model and Flight Aerodynamic 

Coefficients ........................................................................................................................................... 8

Figure 6.—Tracking algorithm for acceleration signal input to the dynamic inversion scheme .................. 9

Figure 7.—NASA's Ice Contamination Effects Flight Training Device (ICEFTD) ................................... 11

Figure 8.—System Status Window ............................................................................................................. 12

Figure 9.—D-ICES Status Window ............................................................................................................ 13

Figure 10.—RT-PID Status Window .......................................................................................................... 13

Figure 11.—Comparison of RT-PID Results (red) with D-ICES Predictions (blue), with and 

without Contamination ....................................................................................................................... 14

Figure 12.—ICEFTD Configuration for Test and Evaluation of ICEPro ................................................... 18

Figure 13.—Evaluation Pilot Station .......................................................................................................... 18

Figure 14.—Engineering Station ................................................................................................................ 19

Figure 15.—Air Traffic Control Station ..................................................................................................... 19

Figure 16.—Evaluation Profile Approach Task Flown by Each Pilot Group During ICEPro 

Testing and Evaluation ....................................................................................................................... 21

Figure 17.—Localizer performance—THEIL inequality calculation ......................................................... 27

Figure 18.—Agreed versus Disagreed survey responses ............................................................................ 32

Figure 19.—Additional survey responses ................................................................................................... 33

Figure C.1.—Pilot demographics—Part I Survey questions 1 to 4 ............................................................ 51

Figure C.2.—Situational Awareness—Part II Survey questions 1 to 5 ...................................................... 51

Figure D.1.—ICEPro System Flow Chart .................................................................................................. 57

Figure D.2.—Turbulence Definition (derived from MIL-F-8785C)........................................................... 60

Figure D.3.—Effect of Turbulence on Modeling Results (Dr. Eugene A. Morelli, personal 

communication, April 24, 2011) ......................................................................................................... 61

Figure D.4.—Comparison between vertical accelerations of the boom and airplane center of 

gravity ................................................................................................................................................. 62

Figure D.5.—Fourier coefficients for measured AOA ............................................................................... 63

Figure D.6.—Angle of attack upwash versus frequency ............................................................................ 64

Figure D.7.—Time delay versus frequency ................................................................................................ 64

Figure D.8.—Test Condition 1—Effect on CZD .......................................................................................... 68

Figure D.9.—Test Condition 1—Effect on CmGe ......................................................................................... 69

Figure D.10.—Test Condition 2—Effect on CZD ........................................................................................ 70

Figure D.11.—Test Condition 2—Effect on CmGe ....................................................................................... 70

Figure D.12.—Number of exceedances for CZD .......................................................................................... 71

Figure D.13.—Number of exceedances for CmGe ........................................................................................ 72
NASA/CR—2014-218320 vi 

Figure D.14.—Test condition 3—Effect on CZD ......................................................................................... 73

Figure D.15.—Test condition 3—Effect on CZGe ........................................................................................ 73

Figure D.16.—Test condition 3—Effect on CmD ........................................................................................ 74

Figure D.17.—Test condition 3—Effect on Cmq ......................................................................................... 75

Figure D.18.—Test condition 3—Effect on CmGe

 ........................................................................................ 75

Figure D.19.—Unfiltered D-ICES Performance with Clean Aircraft ......................................................... 78

Figure D.20.—Unfiltered D-ICES Performance with Iced Aircraft ........................................................... 78

Figure D.21.—Severe Turbulence Flight Conditions ................................................................................. 79

Figure D.22.—Filtered Clean Aircraft with Fixed Random Seed .............................................................. 80

Figure D.23.—Filtered Iced Aircraft with Fixed Random Seed ................................................................. 80

Figure D.24.—Filtered Clean Aircraft with Random Seed ......................................................................... 81

Figure D.25.—Filtered Iced Aircraft with Random Seed ........................................................................... 82

Figure D.26.—Effect of flow angle error on CmD ....................................................................................... 83

Figure D.27.—Effect of flow angle error on CmGe

 ....................................................................................... 84

Figure D.28.—ISP for all turbulence levels and flow angle errors (D and E) ............................................. 85

Figure D.29.—Roll degrade indication analyses ........................................................................................ 87

Figure D.30.—Angle of attack bracket indication analyses ....................................................................... 88

Figure D.31.—Filtered AOA bracket indication analyses .......................................................................... 89

Figure D.32.—D-ICES Level Flight Scenario ............................................................................................ 91

Figure D.33.—D-ICES Flap Transition Scenario ....................................................................................... 92

Figure D.34.—D-ICES Heading Change Scenario ..................................................................................... 92

Figure D.35.—Uniced Elevator Deflection Theil inequality Coefficient Analysis .................................... 93

Figure D.36.—Iced Elevator Deflection Theil inequality Coefficient Analysis ......................................... 93

Figure D.37.—D-ICES Descent with Flap Change Scenario ..................................................................... 94

Figure D.38.—D-ICES Task 2 with Random Seed .................................................................................... 94

Figure D.39.—Phase 1 Roll degrade indication analyses ........................................................................... 95

Figure D.40.—Phase 2 Roll degrade indication analyses ........................................................................... 96

Figure D.41.—Phase 2 AOA bracket analyses ........................................................................................... 97

Figure D.42.—Phase 3 Roll degrade indication analyses ........................................................................... 98

Figure D.43.—Phase 3 Yaw degrade indication analyses .......................................................................... 98

Figure D.44.—Phase 4 Roll degrade indication analyses ........................................................................... 99

Figure D.45.—Phase 4 Climb Limit indication analyses .......................................................................... 100

Figure D.46.—Phase 4 Flap Limit indication analyses ............................................................................. 101

Figure D.47.—Phase 4 AOA bracket analyses ......................................................................................... 102

Figure E.1.—Test and evaluation profile. Total duration of test is approximately 15 min. Quotes 

are clearances read to the EP, italics are required EP actions per the test protocol. ......................... 116
NASA/CR—2014-218320 1 

Development, Implementation, and Pilot Evaluation of a 

Model-Driven Envelope Protection System to Mitigate 

the Hazard of In-Flight Ice Contamination on a 

Twin-Engine Commuter Aircraft 

Borja Martos, Richard Ranaudo, and Billy Norton 

University of Tennessee Space Institute 

Tullahoma, Tennessee 37388 

David Gingras and Billy Barnhart 

Bihrle Applied Research Inc. 

Hampton, Virginia 23666 

Abstract 

Fatal loss-of-control accidents have been directly related to in-flight airframe icing. The prototype 

system presented in this report directly addresses the need for real-time onboard envelope protection in 

icing conditions. The combination of prior information and real-time aerodynamic parameter estimations 

are shown to provide sufficient information for determining safe limits of the flight envelope during inflight icing encounters. The Icing Contamination Envelope Protection (ICEPro) system was designed and 

implemented to identify degradations in airplane performance and flying qualities resulting from ice 

contamination and provide safe flight-envelope cues to the pilot. The utility of the ICEPro system for 

mitigating a potentially hazardous icing condition was evaluated by 29 pilots using the NASA Ice 

Contamination Effects Flight Training Device. Results showed that real time assessment cues were 

effective in reducing the number of potentially hazardous upset events and in lessening exposure to loss 

of control following an incipient upset condition. Pilot workload with the added ICEPro displays was not 

measurably affected, but pilot opinion surveys showed that real time cueing greatly improved their 

awareness of a hazardous aircraft state. The performance of ICEPro system was further evaluated by 

various levels of sensor noise and atmospheric turbulence. 

1.0 Introduction 

Ice formations on aircraft can negatively affect performance, stability and control characteristics, and 

handling qualities. Accidents that have occurred due to these issues have been documented by the 

National Transportation Safety Board (NTSB) and the Federal Aviation Administration (FAA) over many 

years. In a recent study (Ref. 1) of aircraft icing accidents and incidents reported between 1978 and 2002, 

loss of control events were usually preceded by stall and more likely to occur during approach and 

landing. This is the busiest phase of flight where historically over 50 percent of all accidents occur, and 

where human error is responsible for approximately 75 percent of those accidents (Ref. 2). Ice 

contamination, on a critical airframe surface such as the wings and tail, which might be due to an Ice 

Protection System (IPS) failure, encountering an icing certification exceedance condition, or operator 

error can have a detrimental effect on aircraft performance, stability and control, and handling qualities. 

This situation is further exacerbated by the required changes in airspeed and configuration as the pilot 

slows and configures during the descent, approach, and landing phases. 

One notable accident referenced above involved an ATR 72 that flew through SLD icing conditions 

near Roselawn, Indiana, on October 31, 1994. Even with the IPS operating normally, the aircraft 

experienced a roll control anomaly that led to the loss of control and impact into the ground. This accident 

and others since then have led to numerous safety recommendations from the National Transportation 
NASA/CR—2014-218320 2 

Safety Board (NTSB) (Refs. 3 and 4) and the Commercial Aviation Safety Team (CAST) (Ref. 5) and 

motivated research activities within NASA and the FAA. 

In its Tailplane Icing Program (TIP), NASA employed various artificial ice shapes, which were 

attached to the horizontal tail plane (Ref. 6) of a research aircraft. Numerous test maneuvers were 

conducted where wing flap settings, engine power, and flight speed were varied. The results showed a 

considerable decrease in longitudinal stability and elevator control effectiveness as a function of wing flap 

angle, power setting, and airspeed. In extreme cases, a tail stall upset would occur, which resulted in an 

uncontrollable nose down pitching motion. Airframe ice contamination also increases drag, reducing 

climb performance. In another research program, NASA sponsored an activity called the Smart Icing 

Systems led by the University of Illinois, Urbana-Champaign (UIUC). The Smart Icing Systems took a 

systems level approach to detecting icing conditions, operating ice protection systems and alerting the 

flight crew to the status of the aircraft (Ref. 7). 

Most recently NASA issued Research Announcement (NRA) NNH06ZEA001N, Appendix B, of the 

Aviation Safety Program. The research announcement was defined under the Integrated Vehicle Health 

Management (IVHM) Project, topic IVHM 3.1, Environmental Hazards, which are caused by the “Effects 

of Icing on Aircraft State”. The function of IVHM is to manage the health of the vehicle by detecting, 

diagnosing, prognosing, and mitigating in-flight environmental hazards. The main objective was to 

develop tools that measure the state of the aircraft in atmospheric icing conditions and assess the effects 

of icing on the aircraft for various environmental icing conditions. The original scope for this work 

included simulation and experimental testing but was later constrained to pilot in the loop simulations. 

The goals for this work were: 

1. Develop a system that performs real time assessment of icing effects on aircraft stability, control, and 

performance. 

2. Provide the flight deck with flight guidance cues, configuration and envelope limits, and related state 

assessment messaging to avoid upset conditions and loss of control. 

3. Enable the pilot to make better tactical decisions regarding flight speed and configuration selections 

to maintain the aircraft in a safe envelope of flight. 

The University of Tennessee Space Institute (UTSI) in partnership with Bihrle Applied Research 

(BAR) completed a cooperative research effort addressing the IVHM function, objective, scope, and 

goals. This effort culminated in the development of a real time vehicle state assessment system, which 

was described by Gingras et al. (Ref. 8) and is referred to as the Icing Contamination Envelope Protection 

system (ICEPro). The ICEPro system was developed with the philosophy that a knowledge-based system, 

which compares expected aircraft stability and control characteristics with those existing in real time, 

could effectively provide envelope protection cues and alerting messages to help the pilot keep the 

aircraft flying safely. The knowledge base consists of a model for the baseline or clean aircraft, and an ice 

contamination model with known safe flight envelope limits as demonstrated by either wind tunnel or 

flight test. Angle of attack (AOA) cues, which are driven by ICEPro, show the pilot safe limits that will 

protect against either a wing stall or tail stall upset event. Messages alert the pilot to losses in stability and 

control as a function of flight configuration and condition, and to configuration limits as a function of 

icing and wing flap setting. 

The algorithms in ICEPro were initially developed and tested in a desktop simulation environment, 

and then integrated and further developed in the NASA Ice Contamination Effects Flight Training Device 

(ICEFTD). The flight models used in simulation to develop ICEPro for testing and evaluation were based 

upon the characteristics of the NASA Twin Otter icing research aircraft in a no-ice baseline condition, 

and a failure of the ice protection system (IPS). The failure case was defined by the ice accretions that 

would form during a 22.5 min FAR 25 Appendix C icing encounter as calculated by LEWICE version 

2.0. The data for the simulator came from a previous NASA project where a 6.5 percent scale Twin Otter 
NASA/CR—2014-218320 3 

model was tested in the Wichita State University 7u10 Low Speed wind tunnel and BAR’s Large 

Amplitude Multi-Purpose (LAMP) rotary balance wind tunnel using scaled ice shapes (Ref. 9). Flight 

data from aircraft testing with baseline and failure case ice shapes was also gathered to refine the 

simulation model data to improve the match full scale flight test results (Ref. 10). 

After development and testing in a desktop environment, ICEPro was integrated with the NASA 

ICEFTD. Researchers from BAR and UTSI completed final development and integration testing to ensure 

readiness for the planned pilot evaluations. A final check of test readiness was then performed by having 

two experienced NASA test pilots complete the evaluation profile, and all required pre-test preparation 

and training. This included a review of: the test plan, NASA web based icing training, standard operating 

procedures (SOP) for the Twin Otter, the NASA Task Load Index or TLX workload method (Ref. 11), a 

tutorial on flying with ICEPro displays (only for the pilot using the experimental displays), and post-test 

survey questionnaire. One NASA pilot was selected to represent the control group and flew with baseline 

displays. The other was selected to represent the experimental group and flew with the ICEPro modified 

displays. Before the test began, each pilot was briefed on the entire test process. The test protocol at the 

time called for minimal training in aircraft handling with hazardous ice formations, somewhat similar to 

what a typical pilot would receive in an aircraft transition course. However, once testing began, it became 

apparent that the flight task was so difficult that without practice, there was a high probability of a 

random “crash” regardless of the displays used. It was apparent that a different approach would have to be 

taken. After assessing the pilot’s comments from the debriefings, it was therefore decided to modify the 

test protocol and train the pilots in both groups to proficiency for executing the challenging approach task 

using their respective flight displays. Utility benefits could then be assessed by comparing pilot 

performance, workload, and opinions between the control and experimental groups who were flying the 

same task with the same training and preparation, but with different flight displays. This adjustment in the 

test plan was an essential change, and would not have been discovered without this pre-test. The NASA 

pilots also completed the NASA TLX forms and answered all the survey questions during their respective 

evaluations. A few small changes in the survey questions were also required to eliminate ambiguity 

problems. After these adjustments were made to the test plan, pilot-in-the- loop (PIL) evaluations of 

ICEPro were scheduled at the Embry Riddle Aeronautical University (ERAU), Daytona Beach, Florida. 

However, the simulation could not duplicate all of the real-world conditions that could have firstorder effects on system performance, such as those due to real atmospheric turbulence. Therefore, in order 

to assess system performance under these conditions and minimize the technical risk of an eventual flighttest validation program, the effects of atmospheric turbulence on ICEPro were studied. This led to a one 

year extension awarded under NASA’s Vehicle Systems Safety Technologies (VSST) project. 

This contractor report summarizes the results of this research, which focused on the concept, design, 

development, implementation, and evaluation of the real-time vehicle state assessment system. This 

evaluation included pilot cueing for flight envelope protection, and evaluation of the effectiveness of the 

ICEPro system in piloted simulations using the ICEFTD. Sections 2.0 to 9.0 focus on the concept, design, 

development, and implementation of ICEPro. Section 10.0 explains the methodology used for pilot-inthe-loop testing and overall experiment design. Section 11.0 presents the results and discussions of pilotin-the-loop testing. Section 12.0 discusses the atmospheric turbulence test methodology and finally 

Section 13.0 cover conclusions and recommendations. There are five appendices attached to this report. 

Appendix A is the Symbols List. Appendix B includes the post-test survey questionnaire used in the 

pilot-in-the-loop simulations. Appendix C includes the results of the post-test survey questionnaire. 

Appendix D includes the progress report of the effects of atmospheric turbulence on ICEPro through the 

funded one year extension. Appendix E includes the atmospheric turbulence test plan. 
NASA/CR—2014-218320 4 

2.0 ICEPro Concept 

The ICEPro concept is best summarized by the hypothesis posed by the development team during 

preliminary stages of the design process. 

By the real-time processing of measured state and control information, estimates of stability 

and control can be used in conjunction with a knowledge base, comprised of a-priori data, to 

provide pilots with envelope-limiting cueing in order to avoid loss-of-control or adverse 

conditions resulting from in-flight icing. 

The ICEPro system was designed and implemented to identify degradations in airplane performance 

and flying qualities resulting from ice contamination and to provide safe flight-envelope cues to the pilot. 

The ICEPro concept relies on the actions of a proficient pilot responding to appropriate cueing and 

information on the primary and secondary displays in the cockpit. It is believed that, with these cues, the 

pilot can make informed decisions as to how the airplane should be flown. The ICEPro concept provides 

envelope protection by issuing advisories, cautions, and warnings in the form of airspeed limits, angle of 

attack limits, flap position limits, and control effectiveness status. The indications are intended to be 

consistent with typical advisories and warnings that appear on flight displays, so that they are intuitive 

and effective. Figure 1 provides a graphic illustrating the cueing that is used in the ICEPro system. 

When the system does not detect significant change from the nominal prior airplane characteristics, 

no cuing or messages appear and the instruments display the nominal flight information. When the system 

senses icing contamination that exceeds the system thresholds and triggers the latching logic, cues and 

possibly messages appear in conjunction with the normal display information. 

Figure 1.—ICEPro Envelope Protection Cues on the Primary Flight Display (left) and Flight Control Status 

Page (right) 
NASA/CR—2014-218320 5 

There are three types of information that are provided when icing contamination is determined. See 

Figure 1 for illustration. 

x Airspeed limit carets and angle of attack limit brackets. 

The airspeed limit carats for icing are to the right of the speed tape and annotated with a * to 

indicate icing. The angle of attack limit brackets are superimposed on the attitude indicator. The 

upper bracket indicates the wing stall AOA. The lower bracket indicates the tail stall AOA. The 

color of the brackets change from white to amber when the AOA is less than 2° lower than stall 

AOA and then change to red when the AOA reaches the stall AOA. The brackets will also flash 

for several seconds with each color transition, white to amber, and amber to red. 

x Status messages 

These will appear, both on the primary flight display and on the Flight Control Display (FCD) 

page in the message box. They indicate that there is a control and/or stability degradation in the 

indicated axis. There is also a message that appears whenever the single-engine climb capability 

is outside of safe limits. A flap limit message will also appear when the flaps should be retracted 

to a safer deflection. 

x Flight control status indicators 

The FCD page includes a rough diagram of the airplane showing its control surfaces. The 

surfaces are colored and the color will change whenever degradation is determined in their axis. 

With no degradation, the surfaces are green and turn amber when there is a caution displayed for 

that axis and red when there is a warning. The color-coded displays are driven by current 

estimates of control effectiveness and thresholds defined inside ICEPro. 

The basic design architecture of the system is shown in Figure 2. The ICEPro System has three modes 

of operation: a MONITOR mode that provides initial detection of in-flight icing, IDENTIFICATION or 

ID mode that provides a diagnosis of the condition, and a REPORTING mode, providing the prognosis of 

the condition and mitigation through cueing. Details of the system design can be found in Reference 8. 

Figure 3 presents a simplified flow chart. Details, such as latching/delatching logic and specific logic 

for driving each of the caution/warning messages have been omitted to reduce the complexity of the 

diagram. As indicated in the diagram, the system is heavily dependent on input from two components: 

1) The Dynamic Inversion Control Evaluation System (D-ICES), an algorithm that computes what the 

control surface deflections should be for a clean aircraft compared to what the control deflections actually 

are, and 2) the Real-Time Parameter IDentification (RT-PID) routine. Results from these components are 

used to determine an Icing Severity Parameter (ISP) which is used in logic for issuing pilot cues and 

warnings during the reporting mode. 

Figure 2.—ICEPro System Architecture 
NASA/CR—2014-218320 6 

Figure 3.—Simplified ICEPro System Flow Chart 

Yes 

Yes 

Yes 

No

a priori aero 

coefficients PID aerodynamics

Yes 

No 

Monitor Mode 

IMOD=0 

Flight Data 

IMODE = 1 

Go to ID Mode 

IMODE = 0 

Stay in Monitor Mode 

ID Mode 

IMOD=1

Start RTPID 

Calculate ISP 

įe deviation) > 

threshold 

IMODE = 0 

Go to Monitor Mode 

IMODE = 1 

Stay in ID Mode 

IMODE = 2 

Go to Report Mode

Report Mode 

IMOD=2 

Update ISP 

IMODE = 1 

Go to ID Mode 

No

Yes 

ISP > 

Report 

Threshold 

No

No

ISP >Report

Threshold 

Deviation > 

Threshold 

Check individual Stab & 

Control values + ROC 

Update Display

Cautions /Warnings 

įe deviation > 

threshold 

D-ICES Data 

(controls)
NASA/CR—2014-218320 7 

3.0 Flight Characteristic Predictions 

Flight predictions are made inside ICEPro using a flight model of a DeHavilland DHC-6 Twin Otter. 

This flight model was derived from a comprehensive nonlinear 6-degree-of-freedom simulation that was 

developed by Bihrle Applied Research and engineers at the NASA Glenn Research Center (GRC) 

(Ref. 9). The model was based on data from extensive wind-tunnel tests using subscale models of the 

Twin-Otter and supplemented with flight data provided by NASA. This model covers the entire flight 

envelope including stall and post-stall flight regimes with and without icing effects resulting from ice 

accretion on the main wing, horizontal tail, and vertical tail. During previous efforts, the model had 

undergone extensive validation against flight data to achieve high confidence predictions of aerodynamic 

characteristics in clean configuration, as well as the aerodynamic characteristics from an iced 

configuration representing an IPS failure situation. Figure 4 contains a flow chart of the process used 

during this effort to validate the aerodynamic simulation modeling. 

Figure 5 contains an example of the resulting comparisons of model output for normal force 

coefficient, axial force coefficient, and pitching moment coefficient, with flight data from an elevator 

doublet maneuver on the clean airplane. This process was used to update the aerodynamic simulation 

model based on flight data. This effort resulted in the knowledge base system or database referred to in 

this work. 

Figure 4.—Process used for Aerodynamic Model Validation 

Sim-predicted 

CNf , CA, CY, & C1, Cm, 

Cn at c.g. 

Flight-extracted 

CNf , CA, CY, & C1, Cm, 

Cn at c.g. 
NASA/CR—2014-218320 8 

Figure 5.—Example of Comparisons of the Simulation Model and 

Flight Aerodynamic Coefficients 

4.0 Dynamics Inversion Control Evaluation System 

This technique, the Dynamics Inversion Control Evaluation System (D-ICES) makes use of the 

system knowledge base to predict the control deflections necessary to achieve the vehicle motion. DICES tracks the current pitch, roll, and yaw rate in each axis using a proportional, integral, and derivative 

(PID) control with lead compensation, Figure 6. With this tracking, the amount of angular acceleration 

required to match the aircraft motion is determined. Using information from the knowledge base, 

predictions of the current “clean” stability of the vehicle are entered into the aircraft equations of motion 

along with weight and balance information and the tracking accelerations to compute a “residual” 

acceleration vector. 

By assuming this residual acceleration is a direct result of control movement, a prediction of control 

deflections can be obtained (Eq. (1)). Changes in the vehicle aerodynamic state from the “clean” can be 

detected by tracking these predictions of the control deflection in flight and comparing them to actual 

control deflections. 

Measured Predicted Predicted

1

Predicted Control Measured Stability Predicted ˆ

x Ax Bu

u B x Ax 

 

  a o ¬ 1⁄4



 (1) 

CN

CA Cm, (ref)
NASA/CR—2014-218320 9 

Figure 6.—Tracking algorithm for acceleration signal input to the dynamic inversion scheme 

The actual comparisons in the logic are based on the calculated Theil inequality coefficients (Ref. 12) 

for velocity and elevator deflections. The Theil inequality coefficients are used in a moving time window 

to evaluate how well the extracted data is matching the expected levels during that time period. These 

statistics form the basis for the Icing Severity Parameter to be discussed in Section 11.0. The form of the 

Theil inequality coefficients is shown in Equation (2). 

 

1

2

1 1

2 2

1 ˆ

1 1 ˆ

N

N N

y y N LSE U

RMS

y y N N



 



¦

¦ ¦ ¦

 (2) 

Preliminary indications of icing can be assumed when the Theil inequality coefficients for these 

parameters exceed a threshold level that has been calibrated through simulated exercise of the system. 

5.0 Real-Time Model Estimates 

Because the system compares model predictions described above with the current state, real-time 

estimates are used. These estimates are computed by invoking real-time parameter identification (RTPID). The RT-PID algorithm being employed was developed by Morelli (Ref. 13), and is based on a 

frequency domain approach implemented in version 2.0 of System Identification Programs for Aircraft 

(SIDPAC).(Refs. 14 and 15) The approach uses a linearized model of the vehicle dynamics, Equation (3), 

where A, B, C, and D matrices containing stability and control derivatives. 

     

  

0 o x t Ax t Bu t x x

y t Cx t Du t

  

 



 (3) 

Time-varying estimates of the stability and control derivatives are determined using measured airplane 

control surface deflections and measured states from air data, angular rates, and attitude. Outputs are 

rotational and translational accelerations. To determine the estimates, a cost function is formulated in the 

frequency domain from the Fourier transform of the model in Equation (3), 
NASA/CR—2014-218320 10 

  

  

j x Ax Bu

y Cx Du

Z Z Z Z

Z Z Z



 (4) 

The cost function is 

   2

1

1

2

m

k nk k k

n

J j x n axn bun

 

 Z   ¦   for m frequencies (5) 

where ak and bk are the kth row of matrices A and B, respectively, and xn x un u   k kn k kn   {Z {Z  ,     . 

Each line in Equation (4) can be analyzed separately in this way, which implements an equation-error 

formulation as described in Reference 14. The least squares cost function in each case can be formulated as 

   1 †

2

J YX YX T T (6) 

where † indicates complex conjugate transpose, 

 

 

1 1 k

m k

j x

Y

j xm

a o Z

« » « »

« » Z ¬ 1⁄4



#



 

 

1 1 T T

T T

x u

X

xm um

a o

« »

 « »

« »

¬ 1⁄4

 

# #

 

 (7) 

and the unknown parameters from A and B are contained in the parameter vector T. The least squares 

parameter vector estimate is obtained as the value of T that minimizes the cost function in Equation (6) 

(Ref. 13). 

  

      

  

1 † †

1 2 †

† 2

ˆ Re Re

ˆ ˆˆ cov Re ˆ

1 ˆ ˆ ˆ

T

X X XY

E XX

YX YX

m





a o T ¬ 1⁄4

a o a o T { TT TT V « » ¬ 1⁄4 ¬ 1⁄4

V  T  T

 (8) 

Performing the identification in the frequency domain is advantageous for this application because of 

computational efficiency, robustness to noise and data dropouts, and reliable confidence bounds from the 

covariance matrix. To ensure reliable estimates, the ICEPro features a self-checking algorithm that 

examines the confidence bound of the parameter estimates. Table 1 lists the parameters that are being 

identified in real-time. 

TABLE 1.—AERODYNAMIC PARAMETERS IDENTIFIED BY RT-PID 

CND CYGa Clp

CNGe CYGr Cl

CNq CYp CnE

CmD CYr CnGa

CmGe ClE CnGr

Cmq ClGa Cnp

CYE ClGr Cnr
NASA/CR—2014-218320 11 

If confidence of the estimated parameters falls below acceptable levels due to lack of data information 

content (low pilot activity), an automated excitation of the aircraft control surfaces is triggered to improve 

data information content. This interface with the subject vehicle is performed as though an autopilot is 

driving control deflections using multi-frequency orthogonal inputs (Ref. 15). Amplitudes of the control 

deflection inputs are adjusted to provide sufficient excitation in each axis while minimizing vehicle 

motion. 

6.0 System Implementation 

The ICEPro system was developed and deployed in Bihrle Applied Research’s D-Six PC-Based 

simulation environment. All of the ICEPro functions, including the software for driving the flight displays 

are provided in a single computer that runs the software in real time. The system can be interfaced with a 

desktop simulation, NASA’s Ice Contamination Effects Flight Training Device (Ref. 16), ICEFTD 

(Figure 7), or mounted in the NASA Twin Otter aircraft for flight demonstrations. The system includes all 

of the Data Acquisition and Executive functions which control the various modules that make up the 

system. 

The data acquisition portion receives data from the data bus on the airplane or from an emulation of 

the airplane data bus if it is working with the simulator. It then makes the appropriate data available to the 

various other modules for processing. The Executive portion provides all of the logic and calculation 

modules required to determine the icing state and what cues should be provided to the pilot. 

The ICEPro system was first developed using a desktop simulation running in D-Six. The module 

development and all of the initial check out and system calibration were executed in this manner. Once 

the system was sufficiently mature to permit piloted evaluations, it was migrated to the ICEFTD. On the 

desktop implementation, the ICEPro modules and the Twin Otter simulation were run on the same 

computer. For the ICEFTD implementation, a separate computer was used for the ICEPro modules. 

Communication with the ICEFTD was accomplished over an Ethernet connection. This implementation 

was used to facilitate an eventual implementation in the actual airplane. If this were to occur, the same 

computer would be similarly connected to the airplane’s data acquisition system. 

Figure 7.—NASA's Ice Contamination Effects Flight Training Device (ICEFTD) 
NASA/CR—2014-218320 12 

The ICEFTD piloted simulations provided synthetic environment to assess and tune the final 

thresholds and latching logic for the cues and messages. They also afforded the opportunity to assess the 

effect of control surface excitation on the pilot. The piloted simulation was also instrumental in evaluating 

the appearance and validity of the caution/warning messages, and the development and evaluation of 

aural and tactile cues. These messages and cues were incorporated for the guest pilot evaluations that 

were subsequently performed following the development stage. 

7.0 Engineering Station 

An engineering station was set up that enabled the operator to set key parameters of the ICEPro 

system, monitor the performance of the system, and interact with the implemented logic during the 

development and evaluation phases of the program. The engineering station consisted of a set of 

Graphical User Interfaces (GUIs) that were displayed on two monitors attached to the computer running 

ICEPro. The first window (Figure 8) shows the status of the system. It shows in which mode the system is 

currently operating (Monitor, ID, or Report), whether the controls are being excited, the duration and 

magnitude of the excitations in each axis, as well as the time since reset of both the D-ICES and Real 

Time Parameter Identification (RT-PID) modules. There is also a window that displays system status 

messages and buttons to reset the system modules and settings. The duration and magnitude of the surface 

excitations can be adjusted in this GUI and the excitation can be manually triggered when the system is in 

the ID or Report modes. It is also possible to manually reset either the D-ICES or RT-PID routines from 

this window. 

There are a series of default thresholds, time delays, etc. that are used by the ICEPro logic modules 

that are preloaded with the system. All of the default values can be overridden by inputting new values in 

an included text file. There are two buttons at the bottom of the GUI to either update these system settings 

from the text file, or to reset them to the default values. 

Figure 8.—System Status Window 
NASA/CR—2014-218320 13 

The D-ICES status window, Figure 9, shows the a priori model values of the longitudinal and lateraldirectional derivatives for both the clean and iced (Failure IPS icing shapes) airplanes at the current 

aircraft states. These linearized derivatives are extracted at the current state from the stored nonlinear 

database contained in the D-ICES routines. There is also a similar window for the RT-PID routines, 

Figure 10 that shows comparable real time PID derivatives. The values represented in Figure 9 and Figure 

10 are only meant to illustrate the GUI and are not representative. 

The RT-PID module only runs in the ID and REPORT modes, therefore, the values are not updated 

when the system is operating in MONITOR mode. The operator can compare the values from the D-ICES 

and RT-PID status windows to evaluate the differences between the a priori clean or iced values, and the 

actual values as determined by RT-PID. A comparison was plotted off-line and is shown in Figure 11. 

Figure 9.—D-ICES Status Window 

Figure 10.—RT-PID Status Window 
NASA/CR—2014-218320 14 

Figure 11.—Comparison of RT-PID Results (red) with D-ICES Predictions (blue), with and without Contamination 

Because the ICEPro system modules are running within a D-Six project, all of the normal D-Six 

interface tools are also available at the engineering station. As a result, any of the aircraft variables that 

are passed from the ICEFTD are available to monitor in real time listings, strip charts, or as post-run 

engineering plots. Pertinent ICEPro variables are similarly available for real time monitoring and post-run 

analysis. At the end of a run, the time histories of these variables can be recorded for subsequent analysis 

or for play back. The system also provides strip charts of any of the RT-PID derivative values along with 

their expected values and the calculated error bands. 

8.0 System Calibration 

A considerable amount of effort was expended fine-tuning the system to provide adequate warning 

and guidance in an icing encounter without any false indications or nuisance messaging. These efforts 

included three main areas: determining which terms to include in the ISP calculation, tuning thresholds 

and latching and delatching logic, and developing reset logic for the RT-PID routines. 

8.1 Icing Severity Parameter Determination 

As shown in Equation (9), the ISP was formulated as the sum of the difference between the expected 

(D-ICES clean aircraft) and actual (RT-PID derived values) derivative values divided by the difference 

between the a priori clean and iced (Failure IPS icing shapes) values for the selected derivatives. The ISP 

is a single number with a range between 0 and 1. Experimental testing of the system determined that the 

threshold for ice contamination detection was an ISP of 0.5. During developmental testing, it was found 

to be impractical to use all of the RT-PID derivative terms. For example, the yawing moment terms for 

this aircraft did not change appreciably with icing condition, and did not prove useful to include in the 

ISP calculation. The damping terms, such as Cmq and Clp, which did show differences with icing, were the 

most difficult for RT-PID to obtain with great accuracy. These damping terms proved to adversely affect 

the robustness of the ISP calculation and therefore were not used. 

 

 

expected actual

1 expected iced

Value Value

Value Value

n i

i i ISP

n

 

a o  « »

 ¬ 1⁄4 

¦

 (9) 

The control terms, i.e., the Ge and Ga moment derivatives, were generally determined with great 

accuracy, and showed appreciable changes with icing condition. Consequently, these terms were used in 

Time [s]

Pitch Rate [deg/s]

Clean Airplane

0 5 10 15 20 -2

-1.5

-1

-0.5

0

0.5

1

Expected CmAlpha

Predicted CmAlpha

Confidence Interval

Time [s]

Pitch Rate [deg/s]

Ice Contaminated Airplane

0 5 10 15 20 -2

-1.5

-1

-0.5

0

0.5

1

Expected CmAlpha

Predicted CmAlpha

Confidence Interval

CmD [1/rad] 

CmD [1/rad]

CmD

CmD

CmD

CmD
NASA/CR—2014-218320 15 

the ISP calculation. The terms for longitudinal and lateral stability, CmD and ClE were also used. The final 

terms included in the ISP calculation were: 

CmD ClE

CmGe ClGD

CND

Additional logic was also added to consider the relative predicted error in each term, in the form of 

_ value

_ error

RT PID

RT PID



 (10) 

Where, the term (RT-PID_error) is the 2 sigma confidence bound of the estimate, and the term (RTPID_value) is the value of the estimate. If the relative predicted error for any term became larger than an 

experimentally determined threshold, the term was dropped from the ISP calculation. As a result, any 

term whose error represented a sufficiently large portion of its total value would not adversely influence 

the ISP calculation. As will be discussed further, the relative error in the pitching moment due to elevator 

deflection derivative was such a critical value to the ISP calculation (one of the earliest indicators of 

degraded RT-PID results), that its relative error was used as one of the reset criteria to the RT-PID 

routines. 

8.2 Thresholds and Latching 

For each change in mode of operation and messaging, there were thresholds that needed to be met 

before the change could occur or the message appeared. For example, an experimentally derived ISP 

threshold of 0.5 is required for the system to transition from monitor mode to ID mode. The value of 0.5 

was selected so that the ISP threshold could be satisfied with only one term in the ISP formulation. There 

was also latching and delatching logic that specified the condition had to be met for a specified period of 

time before it would appear and, conversely, the condition had to disappear for a period of time before it 

could be removed. This logic was incorporated so that the messages would not “flash” off and on if the 

conditions were right on the edge of display thresholds. 

8.3 RT-PID Reset 

As was mentioned earlier, it was found necessary to force a reset of the RT-PID routines periodically 

for best operation. The RT-PID calculations make use of past values which works well if there is no 

change in the aircraft configuration or flight condition. However, when there are changes, the time 

histories of past values can incorrectly influence the calculations at the new configuration and/or flight 

condition. Consequently, a reset, which flushes the buffers and essentially restarts the calculations, was 

required whenever those conditions occurred. In addition, evaluations of the system showed that better 

results would occur with periodic resets even when no significant changes occurred. For the evaluation 

phase, the system was set to do an RT-PID reset after 50 sec if one had not occurred for either of the other 

two specified reasons. While control excitations were not directly commanded upon reset, they generally 

did occur shortly after the reset unless there was sufficient control activity by the pilot. 

8.4 Conditions for Warning/Caution Messages 

Messages that warned of deficiencies in a particular axis or of aircraft performance limitations were 

displayed on the PFD. The conditions under which these messages would appear, and whether they 

appeared as cautions (amber messages) or warnings (red messages) were based on the amount of 
NASA/CR—2014-218320 16 

degradation from the baseline clean airplane values. Whenever a caution or warning message is displayed, 

the corresponding primary control surface on the airplane representation on the FCD page is also colored 

appropriate to the message (amber for cautions and red for warnings). The caution or warning message is 

also repeated in the message section of the FCD page. 

The initial trials of the ICEPro system considered stability and damping terms in each axis, as well as 

control effectiveness terms to determine when a caution or warning message should be displayed. In 

practice, it was found that meaningful stability and damping thresholds were rarely attained. Therefore, 

these terms were dropped and only the control effectiveness terms were used. Consequently, the 

conditions that will generate a pitch, roll, or yaw degraded message are as shown below (because the 

control derivatives are all negative terms, the inequality signs may seem reversed): 

0.50CmGe _ noice d CmGe  0.25CmGe _ noice , then amber PTCH DGRD and amber Ge on FCD 

CmGe t 0.25CmGe _ noice , then red PTCH DGRD and red Ge on FCD 

0.50CnGr _ noice d CnGr  0.25CnGr _ noice , then amber YAW DGRD and amber Gr on FCD 

CnGr t 0.25CnGr _ noice , then red YAW DGRD and red Gr on FCD 

0.50ClGa _ noice d ClGa  0.25ClGa _ noice , then amber ROLL DGRD and amber Ga on FCD 

ClGa t 0.25ClGa _ noice , then red ROLL DGRD and red Ga on FCD 

Calculations of single engine rate of climb capability were made using the simulation data and Twin 

Otter Airplane Flight Manual values. Tables of these values as functions of flap deflection, ice condition, 

and altitude were stored in the ICEPro routines. Whenever the system was in the Reporting mode and the 

flight and icing condition resulted in predicted single engine rate of climb levels below specified 

thresholds, a climb limit caution or warning message would appear on the PFD and in the FCD message 

box. In addition, a corresponding flap limit message would appear and the flaps representation on the 

FCD page would be colored either amber or red, corresponding to caution or warning messages. This was 

meant to alert the pilot that the flap deflection should be reduced for safe operation in the event of an 

engine failure. The thresholds used during the evaluations were: 

ISP6LQJ(QJ52&ISPWKHQDPEHU&/0%/,0DQG)/$3/,0

6LQJ(QJ52&ISPWKHQUHG&/0%/,0DQG)/$3/,0

8.5 Stall Warning and Decluttering 

A stall warning was added that consisted of a large red STALL message appearing near the middle of 

the PFD and an aural two-note tone. The aural warning would come on whenever the calculated stall 

angle of attack was attained, and the message would appear on the PFD when the angle of attack was 2° 

above the calculated stall angle. During the system development, it was found that in the low-speed 

highly dynamic environment present during stall, RT-PID was unable to keep up with the rapid change in 

aerodynamic characteristics. Extraneous warning/caution messages were observed but would go away 

after the airplane was stabilized. Rather than spend significant time refining the thresholds and logic, it 

was decided, based on pilot evaluations during the development, that it was best to eliminate the 

messaging entirely during a stall and recovery. This allowed the pilot to concentrate on recovering the 

airplane without the distractions. Consequently, logic was incorporated to “declutter” the PFD by 

eliminating all ICEPro related messaging during this period. The angle of attack brackets and speed carets 

did remain on the PFD, since these would aid the pilot during the recovery effort. While these cues do 

depend on the RT-PID results, the ISP calculation is filtered to yield a more or less steady-state level. 

Therefore, the ISP calculation is marginally influenced by the rapid changes during a stall and recovery. 
NASA/CR—2014-218320 17 

8.6 Accommodations for Pilot Evaluations 

In preparation for the piloted evaluations that took place during August 2009, several additions were 

added to the simulation. To represent to the pilot an icing protection system failure, a message stating 

“IPS failure” was displayed in the FCD message box along with an aural cue. A stick shaker and stick 

pusher function were added to the longitudinal yoke loader. These were configured to operate in a manner 

consistent with real-world shakers and pushers and would act as a forceful cue to the pilot that wing stall 

was about to, or had occurred. The shaker was sufficiently rapid and forceful that it could not be missed 

during the most intense flight condition. The shaker was configured to come on at the same time as the 

angle of attack brackets turned from white to amber, which was 2° below the stall angle of attack. The 

pusher was forceful enough that it would force the airplane back to an unstalled angle of attack. 

Monitoring when the shaker or pusher was activated proved to be an important tool in interpreting and 

assessing the performance during post-flight analysis. The stick pusher came on at approximately the 

same time that the stall message appeared on the PFD. 

9.0 Pilot in the Loop Test Methodology 

9.1 Experiment Design 

The purpose of the pilot in the loop evaluations was to determine if ICEPro had utility for mitigating 

a potentially hazardous icing encounter (Ref. 17). The independent parameters were the two displays 

(baseline, and ICEPro with envelope protection), and the dependent parameters were technical 

performance, workload, and pilot opinion. An ICEPro display is shown in Figure 1. A baseline display is 

Figure 1 without airspeed limit carets, angle of attack limit brackets, stall messages, and flight control 

status indicators. A nonparametric, independent measures experiment was therefore selected where 

differences in the dependent variables were analyzed using the Mann-Whitney “U” (MWU) test (Ref. 18). 

A critical “U” was determined, based on the statistical one-tailed test where a 95 percent confidence 

criteria or better was required to correctly reject the null hypothesis (H0). H0 assumed that there was no 

difference between the two groups for the tested variable, therefore the level of significance or alpha level 

UHTXLUHGWRUHMHFWWKHQXOOK\SRWKHVLVZDVFKRVHQWREHĮ  The experiment was planned for an 

evaluation by 30 pilots with relatively similar flight experience. All pilots were Embry-Riddle 

Aeronautical University (ERAU) instructor pilots with a median flight experience of approximately 

1300 flight hours. Each possessed commercial, multi-engine, and instrument ratings, but none had any 

specific experience in actual icing conditions or any prior training in aircraft handling and upset recovery 

due to a hazardous icing condition. The 30 pilots were randomly divided into a control group (baseline 

flight displays and no ICEPro system) and an experimental group (displays modified with ICEPro cues 

and messages). A scheduling problem resulted in a total of 29 pilot evaluations and unequal sample sizes 

between the control and experimental groups (nc = 14 and ne = 15). This disparity was accounted for by 

the MWU methodology as provided for in Reference 18. 

9.2 ICEFTD Simulator Description and Capability 

The aircraft model and subsystem components for the ICEFTD simulator were assembled in BAR’s 

commercial off the shelf (COTS) proprietary simulator development environment, D-Six. The simulator 

provided highly representative flight characteristics of the Twin Otter icing research aircraft. Although 

originally designed as a pilot training device, the ICEFTD’s ability to accurately represent aircraft icing 

characteristics made it a very useful research tool with several advantageous features. One is its capability 

for accurately representing flight characteristics beyond stall AOA, which permits pilots to experience 

post-stall gyrations and tail stall hard-over’s that can occur during upset conditions. Another is the pitch 

control system, which can provide control force gradients and deflections that closely match those 

recorded from flight tests of the NASA Twin Otter during upset events. Lastly, an “aircraft emulator” was 

designed and integrated in order to represent the same sensor feedback and output response as that of the 
NASA/CR—2014-218320 18 

NASA Twin Otter research sensors. Figure 12 is an overview of the ICEFTD as modified for this test. 

The items in red font are those which support ICEPro operation. 

The D-Six simulator also provided excellent three-screen graphics of a visual scene, which was an 

essential element of the final approach and missed approach task. A video recording camera system was 

also employed for each pilot evaluation run in the event a replay of the pilot’s actions needed to be 

observed. During testing, the evaluation pilot wore a headset and was completely enclosed in the 

simulator by a system of curtains as shown in Figure 13 and Figure 14. This isolated the pilot from any 

outside disturbances or distractions, and enhanced the fidelity of the simulation environment. 

Figure 12.—ICEFTD Configuration for Test and Evaluation of ICEPro 

Figure 13.—Evaluation Pilot Station
NASA/CR—2014-218320 19 

A member of the test team operated the simulator and performed data acquisition and processing 

tasks from a remote engineering station. Consequently, this person could not see the ICEFTD instrument 

panel. Therefore, a copy of the PFD displayed to the pilot was provided as shown in Figure 14. 

A second member communicated with the pilot while acting as test conductor and air traffic control 

(ATC) as shown in Figure 15. 

Figure 14.—Engineering Station 

Figure 15.—Air Traffic Control Station 
NASA/CR—2014-218320 20 

9.3 Pre-Test Pilot Proficiency Training 

Before performing the evaluation tasks for data collection, each pilot was subjected to approximately 

1.5 hr of flight training in the ICEFTD. Pilots were trained for the evaluation task using the respective 

flight displays depending on whether they were in the control or experimental group. Training began by 

having each pilot execute the evaluation approach task profile as shown in Figure 16 in a no-ice or 

baseline configuration under instrument meteorological conditions (IMC). The IMC condition for training 

and testing was set at 400 ft and 1 nautical mile (n mi) visibility. The white boxes in Figure 16 describe 

the flight condition and events during the task and the green shaded boxes show the commands given by 

the test conductor, who acted as ATC at various stages during its execution. The missed approach 

procedure (MAP) with an engine failure was initiated by the test conductor at the approximate position as 

shown in red font. Pilots practiced the approach task according to the standard operation procedures 

(SOP), which specified speeds, configurations, and performance standards throughout the profile. This 

phase of training was important in order for the test team to determine that each evaluation pilot 

possessed the requisite instrument flight skills to satisfactorily perform the task. 

The next phase of training was conducted up and away under visual meteorological conditions (VMC) 

to familiarize pilots and develop their skill in dealing with aircraft characteristics due to the ice protection 

failure case condition. This training emphasized recognition of impending upset conditions and the effects 

of wing flap configuration and flight speed on aircraft handling and control characteristics. Pilots in the 

control group were taught how to most effectively use their baseline flight displays to control the aircraft 

and recognize and recover from upset conditions. Pilots in the experimental group received the same 

training emphasis, but were taught how to integrate their scan and control strategy with envelope protection 

cues from ICEPro to facilitate safe flight control and upset recoveries. During this training, evaluation pilots 

were also instructed to fly with both hands on the yoke. This was a learning experience because nearly all of 

them were used to one handed operation in aircraft with relatively light control forces. 

Once the up and away training was completed, each of the pilots practiced the approach procedure 

with the hazardous failure case icing condition as would be performed in the actual evaluation. A certain 

amount of time was also spent having the pilots practice the missed approach procedure under iced 

conditions. This was the most difficult task to perform because of high pitch control forces and the 

requirement to null the asymmetric thrust with high rudder forces when an engine was failed. Pilots had to 

practice this maneuver several times before they could consistently accomplish it in a proficient manner 

and not lose control. Evaluation pilots were continually reminded to fly according to the SOP, which was 

intentionally designed to place the aircraft near the edges of the flight envelope at all times. This protocol 

forced considerable attention to precise aircraft control throughout the approach task, which was essential 

for making a good utility assessment of ICEPro. The evaluation profile approach task for both groups 

required considerable mental concentration, physical strength and very aggressive control inputs to 

prevent or recover from upset conditions. All pilots, regardless of size or gender, were able to effectively 

perform the approach task where pitch control forces reached 90 to 100 lbf at times. In general, most 

pilots only required one practice approach before they were ready to begin the evaluation phase. 

In addition to the flight training, pilots were briefed on and practiced completing the NASA TLX 

workload form. Following this training, pilots were critiqued by the test team, and then given a short 

break before they began their evaluation runs. 
NASA/CR—2014-218320 21 

Figure 16.—Evaluation Profile Approach Task Flown by Each Pilot Group During ICEPro Testing and Evaluation 

9.4 Simulator Flight Evaluations 

The evaluation phase required that a total of three satisfactory approach tasks be performed by each 

pilot. The test conductor sat behind the ICEFTD and acted as ATC and was able to view the evaluation 

pilot through a small opening in the rear curtain. During each evaluation run, the pilots were completely 

enclosed in the simulator cab, and no coaching or extraneous discussions were permitted. As shown in 

Figure 16, the approach task consisted of “radar vectors” to intercept a precision approach procedure. The 

pilot would execute the approach according the SOP, and “break out” of the clouds at 400 ft above 

ground (AGL), and continue the descent to the runway on a visual glide path. At approximately 100 ft 

AGL, “ATC” would instruct a missed approach procedure, which the pilot would acknowledge and then 

initiate. As soon as both power levers were fully advanced, the test conductor would fail an engine, and 

the pilot would continue the procedure while raising the wing flaps from 30° to 10°. The test was 

terminated when the evaluation pilot was stabilized on the missed approach heading. 

Once complete, the evaluation pilot would fill out an electronic TLX form, which automatically 

computed the numerical workload score, and received a short debriefing from the test team. The pilot 

would then re-enter the simulator for the next evaluation run. 

The entire evaluation process for each pilot including the pre-test training exercises and three 

evaluation runs took approximately 3 hr, which seemed to be the limit before fatigue became a factor in 

pilot performance. After testing was completed for each pilot, they were asked to fill out a post-test 

survey questionnaire, included in Appendix B. This questionnaire consisted of four sections using a 
NASA/CR—2014-218320 22 

Likert (Ref. 19) style format: Part I provided pilot demographic and experience information; Part II asked 

about situational awareness; Part III asked for the experimental group’s assessment of ICEPro integration; 

and Part IV asked general questions about workload factors. 

9.5 Data Acquisition and Handling 

Pilot flight performance data parameters, which were available through a flight data file that was 

resident in D-Six, were sampled at 10 Hz. MATLAB routines were used to collect and reduce these data 

for further analysis. For the final approach segment of the task, the Theil inequality coefficient was used 

to evaluate how closely the pilot’s flight path and airspeed agreed with that of a perfectly flown approach. 

A Theil inequality coefficient was calculated for measured lateral and vertical deviations from the 

centerline of the approach path, and the airspeed error from the required 75 kt indicated airspeed (KIAS) 

approach speed. MATLAB routines were also used to record the number of times the stick shaker and 

pusher fired, and number and depth of tail stall AOA exceedence events over the duration of the entire 

flight task. In addition to these performance data, other parameters were recorded during each run to 

provide a more complete picture of the pilot’s control strategy which was useful in training and debriefing 

the evaluation pilots. These included time histories of pressure altitude (HPSLWFKDWWLWXGHșSLWFKUDWH

(q), normal acceleration (nZHOHYDWRUSRVLWLRQįe), elevator control force (FeDQGZLQJIODSVHWWLQJįF). 

Pilots completed the NASA TLX immediately after each data run, and then completed the survey 

questionnaire at the termination of the evaluation session. 

The pilot performance data quantified how well the pilot flew the approach task, how many stick 

shaker and pusher events occurred, and the number of times and extent to which the tail stall AOA was 

exceeded by 5° or more. It should be noted that reaching a tail stall AOA, which was one of the measured 

parameters, did not always result in an upset as long as the pilot immediately recognized the condition 

and promptly arrested the negative pitch rate. The added 5° margin took into account pilot reaction time 

and aircraft dynamic response to the controls. Therefore, tail stall upsets were defined by the number of 

times the pilot reached or exceeded the 5° margin, and the maximum negative AOA achieved in the 

event. This latter metric was also used to define “upset risk”, since the deeper the stall, the lower 

likelihood of recovery. In addition to pilot performance data, the numerical scores of the TLX workload 

assessments and the numerical results from the Likert-type post-test questionnaires were analyzed by the 

MWU process. The TLX analysis was of interest to understand if the added information of the ICEPro 

displays adversely affected workload. The post-test survey analysis was used to assess if there was a real 

and positive difference in hazardous aircraft state awareness, and perception of workload between the two 

groups. Additionally, a section of the survey provided noncomparative information on how well the 

experimental group felt the ICEPro was integrated with flight displays. 

The utility evaluation of ICEPro was based upon assessing real differences between the control and 

experimental groups. The MWU analysis process began by rank ordering the scores of each group, and 

computing a “U” value based on the rank sums. In this one directional test, the “U” for the experimental 

group had to be greater than 144 to ensure a 95 percent probability for correctly rejecting H0. A 

cumulative probability p was calculated from the z VFRUHDQGZDVFRPSDUHGWRĮ WKHUHTXLUHPHQW

for accepting the alternate hypothesis. If this probability was less than 0.05, the null hypothesis was 

rejected. A negative z score (number of standard deviations) of more than magnitude 1.69 (i.e., less than 

–1.69) was required to produce the cumulative probability which met this requirement. The parameters 

which were assessed included all quantified measures of pilot performance, TLX workload, and 

subjective opinions from the survey questionnaires. 

An assessment of recovery from tail stall upsets was also performed. In many cases, when a tail stall 

event was encountered, a series of pitch oscillations would ensue from which the pilot would have to 

recover. An analysis of repeated events was performed to determine if the experimental group could 

recover from these oscillations using ICEPro cueing sooner than the control group, and thereby be less 

prone to entering an out-of-control condition. A pilot coupling event (PCE) represents a hazardous 

situation, which if not arrested quickly, could progress to a loss of control. In this context, the risk of an 
NASA/CR—2014-218320 23 

unrecoverable upset was associated with the number of repeated tail stalls in a PCE scenario, and the 

depth of each stall beyond the 5° margin previously discussed. A time window analysis was therefore 

used to analyze the number of tail stalls occurring within a continuous 5, 10, 15, and 20 sec span, while 

the pilot was attempting a recovery. Search scripts in MATLAB were used to analyze the beginning and 

ending of each PCE, and total the number and severity of each tail stall within the PCE. The results were 

then reported in a tabular format for both groups, and analyzed for significance by the MWU method. An 

important outcome of this analysis was to assess the utility of ICEPro AOA cues for minimizing the 

recovery time from a PCE, and thereby reduce risk of a loss of control situation. 

10.0 Pilot in the Loop Results and Discussion 

10.1 Pilot Demographics 

All evaluation pilots were instructor pilots in the ERAU professional pilot training program. Thirty 

pilot volunteers were broken down into two groups—a control group who flew baseline displays, and an 

experimental group who flew with ICEPro modified displays. All evaluation pilots held FAA commercial 

licenses with multi-engine and instrument ratings. The demographic data was collected in Part I of the 

post-test survey questionnaire, and is discussed below. Due to a scheduling problem, the control group 

consisted of one less pilot that the experimental group. The demographics of the groups were as follows: 

Subjects and gender: 

14 control group (12 male, 2 female) 

15 experimental group (12 male, 3 female) 

Median flight hours: 

Control group Experimental group

Total 1350 Total 1250 

Multi-engine 122 Multi-engine 100 

Icing related experience and training as related from Part I of the post-test survey questionnaire: 

x 75 percent had no prior in-flight icing experience 

x 89 percent did not feel that their prior icing related knowledge/experience would have prepared 

them to perform the test scenario (training was required in order for pilots to execute the flight 

task) 

x 72 percent agreed that the NASA videos/web based training gave more information about icing 

than they had known before (viewing NASA web-based training videos was a pre-test 

requirement) 

x Mixed responses were given for icing related flight training being focused on IPS operation (this 

depended on the types and classes of aircraft in which pilots had been previously trained) 

10.2 Pilot Control Performance 

Pilot control performance assessment from the control and experimental groups are summarized in 

Table 2. Results highlighted in green indicate that the significance or real difference criterion for the 

assessed parameter was met, and the performance of the experimental group was better than that of the 

control group. Results highlighted in blue indicate that the assessed parameter showed better performance 

by the experimental group, but the result did not meet the real difference criterion. Results highlighted in 

orange, indicate that there was no difference between the performance of the two groups, and results 

highlighted in magenta indicate that the experimental group did not perform as well as the control group. 

In general the cumulative pilot performance for the two most critical events, pusher activation and tail 

stall, met the real difference criterion. These events were considered most important from a safety of 
NASA/CR—2014-218320 24 

flight standpoint because a wing or tail stall upset had occurred, and potential for loss of control was very 

high unless the pilot promptly applied the correct sequence of recovery controls. The shaker activation 

event indicated that the control group had slightly better performance than the experimental group, but 

this finding can be misleading without considering important aspects of shaker implementation and pilot 

use of this functionality in a human factors context. This is explained in the following discussion of the 

results, which were tabulated in Table 2. The first column of Table 2 identifies the event that was 

evaluated during the approach and missed approach task, the second column indicates the number of 

standard deviations (z) required in meeting the real difference criterion, the third column indicated the 

cumulative probability (p), and the last column provides a performance assessment between the 

experimental and control group based upon the analysis. 

10.2.1 Shaker Events—Stall Warning 

The results of the pilot performance for avoiding stick shaker events as shown in Table 2 indicate 

that, cumulatively, the control group had a slightly better performance than the experimental group. 

Shaker firings typically occurred in banked turns if speed got too low, during positive pitch overshoots 

incited by a tail stall induced PCE, and when pilots made an overly aggressive recovery after stick pusher 

firing. The following discussion places these results into perspective and discusses factors that may have 

affected their outcome. The control group flew with basic aircraft displays and a simple stick shaker 

functionality, which always provided a 7 percent stall speed margin based upon a clean wing; i.e., no-ice. 

Therefore, when the shaker fired during the approach task in the icing condition, the actual margin to stall 

was less than 7 percent. 

On the other hand, the experimental group flew with ICEPro generated displays, which provided real 

time AOA cues and shaker functionality that maintained a 7 percent stall speed margin for the icing 

TABLE 2.—SUMMARY OF PILOT CONTROL PERFORMANCE 

Pilot control 

performance by run 

number and 

cumulative 

z score: 

z < –1.69 for 95% 

confidence in H0 rejection 

Probability: 

p < UHTXLUHGWRVDWLVI\Į

(alternate hypothesis) 

Group performance: 

Exp. versus control 

Positive real difference (PRD), 

Better (B), Worse (W), Same (S) 

Shaker events 

Run 1 1.0474 0.8575 W 

Run 2 –0.2837 0.3799 B 

Run 3 –0.5892 0.2705 B 

Cumulative runs 0.0873 0.5434 S 

Pusher events 

Run 1 –0.0655 0.4652 S 

Run 2 –1.2875 0.0952 B 

Run 3 –2.1822 0.0137 PRD 

Cumulative runs –1.7457 0.0385 PRD 

Stall Messages 

No stall messages appeared for the experimental group 

Tail stall events 

Run 1 –1.2657 0.0989 B 

Run 2 –1.8330 0.0318 PRD 

Run 3 –1.7894 0.0350 PRD 

Cumulative runs –2.1822 0.0137 PRD 

Legend 

 Meets 95% confidence for rejecting H0 

 Experimental group better over control group but < 95% confidence for rejecting H0 

 No difference between experimental and control group 

 Experimental group performance worse than control group 
NASA/CR—2014-218320 25 

condition. Experimental group pilots could avoid shaker events if they remained below the upper AOA 

limit bracket as shown in Figure 1. The control group had no AOA information on their basic flight 

displays. In order to avoid shaker activation, they had to closely maintain the SOP required airspeeds 

throughout the entire task. Notably, the control group was technically at greater risk when the shaker fired 

because their stall speed margin was less than 7 percent. A study of the frequency of shaker events is 

helpful in order to better understand the results of Table 2. Table 3 provides the number of recorded 

shaker events per run for each group of pilots. 

During run one, 14 control group pilots experienced 87 shaker events, whereas 15 experimental group 

pilots experienced 113 shaker events. On average, the control group experienced 6.21 events per pilot, 

and the experimental group experienced 7.53 events per pilot. The corresponding MWU analysis for run 

one in Table 2 clearly shows better (but not significant) performance from the control group. During the 

subsequent runs, however, the experimental group’s performance got better, and this is reflected in the 

analysis for runs two and three in Table 2. If all three runs are considered cumulatively, the normalized 

ratio between the two groups is slightly better for the experimental group, but the cumulative results of 

the statistical analysis shown in Table 2 slightly favors the performance of the control group. Regardless, 

the differences between control and experimental groups as shown in Table 2 and Table 3 are very small. 

What is important to observe here is that the experimental group did a much better job of avoiding shaker 

events during subsequent runs, while the control group’s performance remained nearly constant. This was 

likely due to increased proficiency in using ICEPro AOA cues, which was not fully realized during pretest training. Unfortunately, practical limits on pre-test training were necessary in order to conduct all 

required evaluations within time and program cost constraints. 

10.2.2 Pusher Events—Wing Stall Upsets 

Table 4 summarizes the number of pusher events per run for each group. Pusher events were 

generally precipitated by the same factors that incited shaker events and occurred when the pilot could not 

arrest an AOA increase, which resulted in a wing stall upset. This was particularly evident when the 

missed approach procedure was executed. Here, pilots immediately advanced the throttles to full power, 

raised the wing flaps to 10° and the nose to the takeoff attitude in one coordinated motion. As soon the 

power levers reached the full power stop and the evaluation pilot reached for the wing flap lever, the left 

engine was failed by the test conductor, and the pilot then had to null the yaw and adjust pitch so as to 

maintain a 76 KIAS climb speed. This was a very dynamic maneuver, and in many cases it precipitated a 

PCE. As in the case of the shaker events, the experimental group of pilots improved with each evaluation 

run, whereas the control group actually got worse. The raw data in Table 4 supports the statistical 

assessment that the ICEPro displays resulted in better pilot control performance and lower risk of an upset 

due to either wing or tail stall. 

TABLE 3.—NORMALIZED SHAKER EVENTS 

Control group (14 pilots) Experimental group (15 pilots) 

Run Shaker events Normalized Run Shaker events Normalized 

1 87 6.21 1 113 7.53 

2 92 6.57 2 86 5.73 

3 87 6.21 3 72 4.80 

Cum 266 6.33 Cum 271 6.02 

TABLE 4.—NORMALIZED PUSHER EVENTS 

Control group (14 pilots) Experimental group (15 pilots) 

Run Pusher events Normalized Run Pusher events Normalized 

1 12 0.86 1 13 0.87 

2 16 1.14 2 10 0.67 

3 19 1.16 3 6 0.40 

Cum 47 1.05 Cum 29 0.64 
NASA/CR—2014-218320 26 

10.2.3 Stall Messages 

A “STALL” message was incorporated into the PFD in the event the pilot attempted to override the 

pusher and thereby remain in a stall upset condition. This was a functionality associated with ICEPro 

displays, and appeared to be unnecessary as all the experimental pilots promptly applied recovery controls 

to reduce the AOA below the upper bar when the pusher fired. No stall messages appeared during any of 

the runs flown by the experimental group. 

10.2.4 Tail-Stall Upsets and Recoveries 

Tail stall upsets resulted when pilots flew too fast, exceeding the speeds specified at various segments 

of the flight task, or when they allowed the AOA to get too low. This condition was greatly aggravated 

with increasing wing flap deflections, and was most sensitive during the final approach and missed 

approach procedure when the flaps were 30q down. As described previously, a tail stall was defined as 

reaching a negative AOA that was 5° below that where the tail had technically reached a stall condition. 

This margin was determined experimentally, as it appeared that if a pilot immediately applied recovery 

controls before reaching this negative AOA, recovery would be prompt and the chance of a PCE 

minimized. When a tail stall occurs in an aircraft equipped with mechanical controls, such as the one 

simulated in this test, the pilot feels an immediate increase in pull force and notes a rapid build-up in 

negative pitch rate. In order to arrest the condition, the pilot must quickly exert very high pull forces (on 

the order of 90 to100 lbf) to reverse the downward motion of the nose. If successful, the nose starts back 

up; but, as it does, pitch forces immediately lighten and a positive pitch overshoot can easily occur. This 

can precipitate a PCE, which can develop into a loss of control situation should a negative AOA be 

reached that the pilot is not able to physically overcome. Pre-test pilot training acquainted both control 

and experimental group pilots with these characteristics, and the basic methods for recovery. 

During training, pilots practiced tail stall upset recoveries by keeping the pitch divergence to a 

minimum. Additionally, the experimental group was taught to use the ICEPro displays to facilitate 

recovery by adjusting pitch during a PCE to remain within the safe envelope defined by the AOA 

brackets. When pitch rate was negative, the pilot would attempt to pull the nose up, but remain below the 

high AOA bracket. If the pitch rate was positive, the pilot would push the nose down, but remain above 

the low AOA bracket. If done properly, this would result is a very expeditious recovery—normally within 

one or two cycles. Of note, this strategy contributed to fewer pusher events as the experimental group 

tended to be better at avoiding stall AOA than the control group. 

The learning bias evident in the shaker and pusher data seem to indicate that had more time been 

available for the experimental group to practice with the ICEPro displays, their performance during these 

events would have been better. Runs 2, 3, and cumulative of the tail stall event clearly show positive real 

difference in the performance of the experimental group and provide better than 95 percent confidence for 

correctly rejecting H0.

10.3 Pilot Precision Approach Performance 

The final approach segment required the pilots to intercept and accurately fly a precision approach to the 

minimum descent altitude of 200 ft. As shown in Figure 1, the flight displays for both groups did not 

provide computed steering or a flight director format. Therefore, the pilots had to fly what is termed “raw 

data” and null localizer and glide slope errors by controlling heading and descent rate. As described earlier, 

this task was also performed in IMC until reaching the ceiling of 400 ft above ground level. When the pilots 

“broke out” of the cloud at this altitude, they continued their descent to 100 ft and were directed to execute a 

missed approach. The difficulty in flying the approach task was largely due to control aspects discussed 

earlier. Some pilots were very smooth and precise throughout the task, and were able to execute it without 

too much difficulty. This was because they did not allow the attitude of the aircraft to vary much beyond 

that required to maintain a given flight condition. Generally, these were pilots who had very efficient 

instrument scans, and who made very small but immediate corrections when required. Pilots whose 

instrument scans were less efficient and allowed large pitch or roll variations to occur were more prone to 

get into a pitch axis PCE. A PCE event greatly affected glide slope, course, and airspeed errors, and was 

reflected in the Theil inequality coefficient values for each of those parameters. Table 5 provides the results 
NASA/CR—2014-218320 27 

of the MWU analysis of the precision approach errors between the control and experimental groups. It 

should be noted, that both groups of pilots generally flew the approach procedure within Airline Transport 

Pilot (ATP) standards (Ref. 20) as required in the SOP. The Theil inequality coefficient values, which were 

used in the MWU analysis, were basically a means of quantifying the pilot’s performance with respect to a 

perfectly flown approach. A representative localizer Theil inequality value for one such run is shown in 

Figure 17. Similar plots were obtained for glide slope Theil inequality values. As shown in Table 5, the 

control group had better cumulative performance in minimizing localizer and airspeed errors. The 

experimental group had better cumulative performance in minimizing glide slope error, but did not meet the 

significance criteria required to infer that there was a real difference between the two groups. It would 

appear, after considering the results from Table 2 and Table 5, that pilots who had the advantage of ICEPro 

displays tended to prioritize aircraft control over the preciseness at which they flew the localizer and the 

required airspeed. This is perhaps why the control group, who flew more precise airspeed, had significantly 

more tail stall upset events than the experimental group and was considered to be at greater risk of entering 

an out of control situation. 

Figure 17.—Localizer performance—THEIL inequality calculation 

TABLE 5.—PILOT PRECISION APPROACH PERFORMANCE 

Pilot precision approach 

performance by run 

number and cumulative 

z score: 

z < –1.69 for 95% 

confidence in H0 rejection 

Probability: 

p < 0.05 required to 

VDWLVI\ĮDOWHUQDWH

hypothesis)

Group Performance: 

Exp. versus Control 

Positive real difference (PRD), Better 

(B), Worse (W), Same (S) 

Localizer 

Run 1 0.6328 0.7436 W

Run 2 0.2619 0.6116 W 

Run 3 0.0436 0.5260 W 

Cumulative 0.1528 0.5692 W 

Glide slope 

Run 1 –1.4402 0.0718 B 

Run 2 0.5674 0.7221 W 

Run 3 –1.4621 0.0689 B 

Cumulative –0.4364 0.3233 B 

Airspeed control 

Run 1 1.8330 0.9681 W 

Run 2 1.9640 0.9764 W 

Run 3 1.4402 0.9281 W 

Cumulative 1.9421 0.9752 W

Legend 

 Meets 95% confidence for rejecting H0

 Experimental group better over control group but < 95% confidence for rejecting H0 

 No difference between experimental and control group 

 Experimental group performance worse than control group Localizer [deg] 

Time [s] 
NASA/CR—2014-218320 28 

10.4 Exposure to Risk of an Out of Control Event 

A number of tail stalls, which were encountered in the course of the approach procedure, resulted in 

PCEs before the pilot was able to stop the oscillations and recover to a stable and normal flight condition. 

This event was described earlier in Section 8.5, and under “Tail-stall upsets and recoveries”, Section 

10.2.4. If a quick recovery from a tail stall event (TSE) could not be facilitated by the pilot, there was a 

risk that repeated oscillations result in a PCE which would deteriorate into a LOC situation. Therefore, the 

numbers of repeated tail-stalls during PCEs were summed for each group during the final approach 

segment in four time duration windows of 5, 10, 15 and 20 sec. The methodology began by identifying an 

initial tail-stall event (TSE), which met the criterion (negative tail-stall AOA plus 5°). If another TSE 

occurred within 5 sec it would be counted with the first tail stall and considered a PCE. If in this event, 

TSEs continued to occur within 5 sec of the one previous, they would be added to the PCE until the 

duration of time between a repeat TSE was greater than 5 sec. This methodology was then repeated for 

time intervals of 10, 15, and 20 sec. The exposure to risk of reaching a LOC situation was therefore 

greater during a PCE based upon the number of repeat TSEs in any of the four time frames, the amount of 

time it took the pilot to recover, and the maximum negative pitch attitude reached during the event. The 

results of this analysis are provided in Table 6. 

The most significant events occurred during the 5 sec interval. By using their ICEPro driven displays, 

the experimental group had 75 percent fewer PCEs, their events were approximately half as long in 

duration, and the maximum nose down attitude during the event was about 3.5° less than that of the 

control group. In effect, the experimental group could arrest a PCE quicker, with fewer oscillations than 

the control group. Most important however, was the maximum depth of the tail stall, which was measured 

by the maximum negative pitch attitude achieved during the recovery attempt. The more negative the 

attitude, the higher the negative column forces, and the more difficult it was for the pilot to arrest the 

negative pitching moment. Though not shown here in the data, some pilots had to use in excess of 90 to 

100 lbf to arrest the negative pitching moment, which was more likely to cause a positive pitch overshoot, 

and sometimes result in pusher activation. 

As previously mentioned, the test profile was intentionally designed to place pilots on the edge of a 

TSE during the approach, and as a result, most pilots in both groups experienced them, especially when 

momentarily distracted from their instrument cross-check. However, when a TSE occurred and especially 

when it deteriorated into a PCE, the experimental group was able to effectively utilize their AOA cues to 

damp the oscillations more quickly than the control group. They accomplished this recovery while 

minimizing their time at an unsafe negative AOA, and thereby were at less risk of reaching a LOC 

situation. 

TABLE 6.—EXPOSURE TO RISK BASED UPON PILOT COUPLING EVENTS 

Group Time interval between 

repeat TSEs 

(sec) 

Total number 

of PCEs 

Mean duration of PCEs 

in the time interval 

(sec) 

Mean max 

negative pitch 

Control 5 16 15.16 –12.23 

Experimental 4 8.66 –8.8 

Control 10 69 18.54 –12.06 

Experimental 29 17.98 –11.12 

Control 15 84 23.08 –12.32 

Experimental 36 23.22 –11.37 

Control 20 90 27.64 –12.44 

Experimental 41 25.85 –11.23 
NASA/CR—2014-218320 29 

10.5 Workload 

After completing each evaluation run, the pilots were asked to complete a TLX workload assessment 

for the entire task. The flight profile was intentionally designed to place each group of pilots very near the 

edges of the safe flight envelope especially during the final approach and missed approach segments. 

Further, the tasks were flown in IMC, and both the cognitive and physical aspects of workload were quite 

high for both groups, especially when recoveries from TSEs or PCEs were required. Because the 

experimental group had to integrate more information than the control group, there was a concern that 

their workload could be adversely affected. 

The ICEPro displays provided much more information than the basic displays, and had to be 

integrated into the pilot’s scan along with the instrument flying task. The control group did not have the 

additional information and thus had less cognitive workload. An analysis was conducted for each of the 

three runs in both groups. The results shown in Table 7 indicate that overall, the experimental group 

reported less total workload. Run 2 for example, taken by itself did meet the significance criteria and 

indicated that ICEPro displays made a real difference in reduced workload. However, on average for all 

runs, the results did not meet the significance criteria. 

Observation of both groups of pilots indicated that as they flew each successive profile, they became 

more comfortable with the task, and tended to report lower workload. In spite of the lower workload, it 

was also apparent that, by the second run, fatigue began setting in; this affected pilot concentration and 

control performance. The entire evaluation was conducted over an approximate 3 hr period with 

strategically inserted rest breaks, but it was apparent that the performance of the pilots from both groups 

seemed to peak by the second evaluation and decrease on the third run. The results of the post-test survey 

questions, which are discussed in the next section, provide further insight into these findings. 

10.6 Post-Test Survey Results 

The questions in all four parts of the survey solicited pilot opinions to help gain insight of the two 

pilot groups with respect to their icing experience, perception of situation awareness, how well ICEPro 

was implemented, and workload issues associated with the flight task. Descriptive analysis was used for 

responses to Part I—Demographics and Part III—ICEPro Implementation. Parts II—Situation Awareness, 

and Part IV—Workload, were analyzed via the MWU methodology for real differences. The survey 

questions solicited either agreement, or frequency of occurrence. A Likert five-answer format was used 

with numbers 1 to 5 assigned to the answers from left to right. For the MWU analysis of Parts II and IV, 

the “direction” of the numbered response for each question, i.e., 1 to 5, or 5 to 1, was defined to indicate 

better performance. The survey questionnaire is provided in Appendix B for the reader’s reference, and 

histograms of the responses to questions in Parts I, II and IV are provided in Appendix C. Descriptiveonly assessments of experimental group pilot opinions are provided for Part III. 

TABLE 7.—WORKLOAD ASSESSMENT 

Pilot workload by run 

number and cumulative 

z score: 

z < –1.69 for 95% 

confidence in H0 rejection 

Probability: 

p < WRVDWLVI\Į

(alternate hypothesis) 

Group Performance: 

Exp. versus Control 

Positive real difference (PRD), 

Better (B), Worse (W), Same (S) 

TLX Workload 

Run 1 –0.5237 0.2926 B 

Run 2 –1.8985 0.0274 PRD 

Run 3 –1.4621 0.0689 B 

Average all runs –1.0911 0.1328 B 

Legend 

 Meets 95% confidence for rejecting H0

 Experimental group better over control group but < 95% confidence for rejecting H0 

 No difference between experimental and control group 

 Control group performance better than experimental group 
NASA/CR—2014-218320 30 

10.6.1 Part I—Demographics 

The two pilot groups were intentionally selected from a relatively homogeneous sample. All pilots 

were essentially General Aviation (GA) instructor pilots who shared common experience with training, 

ratings, and types of aircraft flown. All pilots except one were staff instructors at ERAU and subscribed to 

the same operational and flight standards as defined by the University’s professional pilot training 

program. There was only one outlier in the control group, and that individual’s flight hours were not 

included in the standard deviation calculations as this individual’s flight time skewed the data 

considerably. With the elimination of that pilot’s flight hours, the standard deviation of the control group 

was 1054 hr, and the standard deviation of the experimental group was 698 hr. As shown in Figure C.1 

(Appendix C) nearly all pilots had minimal experience with aircraft icing or icing related training before 

participating in this test. 

10.6.2 Part II—Situation Awareness 

Table 8 and Appendix B and Appendix C summarize the situational awareness responses to the 

questions. 

TABLE 8.—SITUATIONAL AWARENESS POST-TEST QUESTIONNAIRE RESULTS 

Pilot post-test survey questionnaire results z score: 

z < –1.69 for 95% 

confidence in H0

rejection 

Probability: 

p < 0.05 required 

WRVDWLVI\Į

(alternate 

hypothesis) 

Group Performance: Exp. vs. 

Control 

Positive real difference 

(PRD), Better (B), Worse 

(W), Same (S) 

Part II—Situational Awareness 

1. Flight displays were adequate to determine if 

airframe icing was having an effect on aircraft 

characteristics. 

–3.0987 0.00008 PRD 

2. Knew minimum safe speed for a given wing 

flap setting within 5 kt. 

–2.2242 0.0072 PRD 

3. Knew how to adjust pitch attitude to avoid a 

wing stall or a tail stall upset. 

–2.5750 0.0047 PRD 

4. Knew the wing flap settings for safe rate of 

climb in event of an engine failure. 

–2. 5968 0.0044 PRD 

5. Had to rely on aircraft control response to 

determine icing effects on pitch, yaw, and roll. 

–4.4298 0.0000 PRD 

6. Confident that the final approach airspeed 

would prevent stall. 

–0.3273 0.3634 B 

7. Able to avoid aircraft handling problems or 

tail stall upsets when selecting wing flaps down. 

1.1129 0.8717 W 

8. Colored bands on airspeed tape useful to 

safely fly the aircraft during the entire flight. 

–2.7714 0.0026 PRD 

9. Relied on stick shaker to prevent inadvertently 

stalling. 

–0.1746 0.4221 B 

10. Always knew when approaching a wing or 

tail stall condition. 

–1.8985 0.0274 PRD 

Legend 

 Meets 95% confidence for rejecting H0

 Experimental group better over control group but < 95% confidence to reject H0 

 No difference between experimental and control group 

 Experimental group performance worse than control group 
NASA/CR—2014-218320 31 

A positive real difference (PRD) indicates that the experimental group had significantly better 

situational awareness than the control group for the issue in question. As shown in Table 8, MWU 

analysis of the responses indicated that a PRD existed between the control and experimental groups with 

respect to questions 1 to 5, 8, and 10. To summarize each of these questions, the experimental group had 

significantly better ability to: 

1. Determine that icing was having an effect on flight characteristics, 

2. Know the minimum safe airspeeds to fly for a given wing flap configuration, 

3. Adjust pitch to avoid wing or tail stalls, 

4. Understand when engine out climb performance was being affected by the wing flap setting, and 

5. To know that icing was affecting pitch, roll and yaw characteristics without relying on aircraft control 

response. 

From the responses to question 8 the experimental group of pilots felt the colored bands on the 

airspeed tape were useful for safely flying the aircraft during the entire flight, while the control group 

did not. The histogram of the responses, Figure C.3 (Appendix C), clearly reflects this difference, but 

the result was somewhat confusing since both groups had the same color bands on their airspeed tape 

(Figure 1), and a real difference was not expected. One explanation however, may have been that the 

ICEPro speed carets, when overlaid on the baseline PFD display speed tape, provided better low and high 

speed awareness in relation to the non-iced aircraft condition. Another possibility is that the experimental 

group misinterpreted the question and thought the “color bands” referred to the ICEPro speed carets. In 

any case, the wording of the question could have been better to ensure that no misinterpretation of the 

nomenclature was made. 

Question 10 indicated that the experimental group felt they had significantly better awareness when 

approaching a flight condition that could lead to a wing or tail stall than the control group. Many of the 

experimental pilots verbally commented during their test runs that the real time computed AOA brackets 

were most useful in that respect. 

The responses to questions 6 and 9 respectively, indicated that the experimental group was more 

confident that their final approach airspeed would avoid wing stall, and that they did not have to rely as 

much on the shaker to avoid stalling. These results, although better for the experimental group, did not 

meet the significance criteria. This better, but not significantly better result may have been due to the fact 

that the experimental group knew that they could safely fly to just below the AOA upper bracket and 

avoid the shaker firing. If the brackets started flashing coincident with the aircraft attitude indication and 

the shaker did fire, they could immediately reduce AOA and avoid stall. The combination of AOA 

brackets and shaker gave the experimental group good low speed and incipient stall awareness. The 

control group on the other hand, had to rely on basic (clean aircraft) shaker settings and the likewise basic 

(clean aircraft) airspeed bands on their PFD to avoid stalling. 

Fortunately for the control group, the failure case icing configuration in the simulation model resulted 

in a very small stall speed increase and concurrent stall AOA decrease. As a result, if the shaker fired for 

the control group, or if they flew into the amber stall warning band on the airspeed indicator, they were 

generally able to react with a decreased pitch input in time to avoid stall. Had a more severe ice shape 

been modeled and the stall margin been less, or even negative, the control group responses may have been 

different as they could have stalled before the shaker fired. The responses to question 7 indicated that the 

experimental group felt less able to avoid handling problems or tail stall upsets while extending the wing 

flaps than the control group. Referring to Figure C.3 (Appendix C), it is apparent that the greatest number 

of responses from the control group agreed that they could manage their control inputs to avoid handling 

problems, while there were an equivalent number of disagreement responses from the experimental 

group. This result was somewhat puzzling because it was apparent from the data in Table 6 that 

quantitatively, the experimental group was significantly better than the control group in avoiding tail 

stalls. One possible explanation is that the experimental group was provided with displays that clearly 

showed the effects of wing flap extension on the safe AOA envelope, but were required by the test 
NASA/CR—2014-218320 32 

protocol to ignore those indications and extend the wing flaps regardless. This could have led to their 

response choice because they expected handling problems as wing flaps were extended, and knew that 

they were unavoidable. The question may have been poorly worded in that respect. 

10.6.3 Part III—ICEPro integration 

The experimental group was asked 15 questions regarding the integration of ICEPro displays. Since 

these questions only affected that group, a descriptive statistical approach was taken in the analysis of the 

data. The primary goal of these survey questions was to understand if the implementation of real time 

state assessment into flyable pilot cues did provide utility to the pilot. The key findings that follow list the 

percentage of responses on either side of a “neutral” opinion. 

Figure 18 and Figure 19 represent the agreed / disagreed and always / frequently and infrequently / 

never responses recorded for the experimental group, respectively. 

Figure 18.—Agreed versus Disagreed survey responses 

40 

53 

60 

70 

40 

40 

40 

47 

60 

67 

40 

33 

40 

20 

40 

47 

47 

40 

40 

20 

0 10 20 30 40 50 60 70 80 90 100

1. Messages were not useful and cluttered the

display . .

2. Combining visual, aural and tactile cues was

important

3. AOA brackets were useful to fly safely . . . . . . . . . . .

. . .

4. It was difficult to understand relation between

AOA brackets and Hi/Lo airspeed carets

5. Synoptic page was more useful than messages for

state assessment

7. De-cluttering did not affect ability to recover from

stall

8. the “red” CLM and FLP LIM needed less flap setting 

for engine out performance. 

9. “Control buzzing” made aircraft control difficult . . . 

. . . 

10. Baseline airspeed tapes and carets were

confusing . . .

15. Immediately noticed when CLM and FLAP LIM

messages were on at the same time

Agreed Disagreed
NASA/CR—2014-218320 33 

Figure 19.—Additional survey responses 

Summarizing these key findings, the experimental group felt that the AOA brackets were very useful 

in all phases of flight, and were effectively able to use the brackets to keep the aircraft in a safe flight 

envelope. However, many had difficulty understanding the relationship between AOA and airspeed. None 

of the participating pilots had experience with flying AOA prior to this test, and it is felt that this response 

was largely the result of this lack of experience. 

More pilots felt that presenting both the baseline airspeed along with the ICEPro generated airspeed 

carets was confusing. The original design intent of the ICEPro displays was to present the computed real 

time speeds as “advisory” information. This way, the pilot would always have a reference to the baseline 

aircraft, which would prove useful if a system anomaly was encountered. There was a concern for the 

manner in which this would be done to avoid confusion and the survey seemed to bear this concern out. 

Along those same lines, pilots were essentially ambivalent as to the usefulness of the ICEPro speed carets 

when maneuvering and flying the approach task as shown in Figure 16. A majority of pilots felt that the 

integration of visual, aural, and tactile cues was important, but providing redundant messages on two 

displays was not necessary. 

Lastly, one major concern was the assessment of the multi-axis control excitation or “buzzing”, which 

was commanded by the ICEPro logic for aircraft state assessment. Some pilots were bothered by the 

buzzing, and some were not. It appeared that because it was expected, pilots accommodated to this as they 

would if having to fly through turbulence. In any case, there did not seem to be a strong opinion one way 

or the other regarding control buzzing. 

10.6.4 Part IV—Workload 

Workload assessments via the NASA TLX format were gathered after each run for each pilot. Posttest questionnaires were solicited in an attempt to understand some of the important factors that 

influenced the scores that pilots gave to workload assessment. The survey answers from the control and 

experimental was assessed via the MWU methodology. Table 9 provides the results from the survey 

questionnaire. Pilot workload was not measurably affected by ICEPro cueing. The data indicated the 

following: 

1. Controlling vertical speed on glide slope was more difficult for experimental group (60 versus 

43 percent). It appears that the control group prioritized safe flight over strict airspeed control. 

2. Flight control inputs were more physically demanding for the control group (73 versus 60 percent), 

possibly due to the fact that the control group was not aware of the magnitude of AOA divergence. 

47 

60 

47 

40 

60 

40 

40 

30 

47 

40 

0 10 20 30 40 50 60 70 80 90 100

6. Favored using airspeed carets over AOA brackets

for safe flight

11. Flew in middle of AOA brackets . . . . . . . . . . . . . . .

. . . . . .

12. Felt that AOA brackets immediately captured

attention when flashing

13. Felt airspeed cues were helpful for maneuvering/

. approach

14. Felt lower AOA bracket helped avoid tail stall

upset . . . .

Always/Frequently Infrequently/Never
NASA/CR—2014-218320 34 

3. The control group reported that it took more concentration and effort for flying MAP safely (87 

versus 80 percent). The results however did not meet the significance criterion, but did show that the 

experimental group did better. 

4. Staying on glide slope without pitch upset was more difficult for control group (79 versus 67 percent). 

This was likely because the control group made corrections without knowledge of AOA limits. On 

the other hand, the experimental group had a real time display of AOA limits and could remain within 

those limits while correcting to the glide path. By “respecting” these limits, the experimental group 

tended to give greater priority to remaining in a safe flight conditions and were less aggressive in 

maintaining a precise vertical flight path. It should be noted however, that both groups flew the 

approaches within ATP standards as previously discussed. 

5. Maintaining glide slope detracted from localizer control. This affected the experimental group more 

than the control group (64 versus 47 percent). This is thought to be a result of AOA cueing, which 

captured the pilot’s attention. 

In summary, the workload was not measurably affected by the addition of ICEPro system. Pilots 

seemed to accommodate to those displays rather quickly during their training, and appeared to use them 

quite effectively to perform the flight task. TLX data showed clearly that the flight task was very 

demanding for both groups of pilots, and the physical difficulty of it far outweighed other workload 

dimensions. 

TABLE 9.—PILOT WORKLOAD SURVEY RESULTS 

Pilot post-test survey 

questionnaire results 

z score: 

z < –1.69 for 95% 

confidence in H0 rejection 

Probability: 

p < 0.05 required to 

VDWLVI\ĮDOWHUQDWH

hypothesis) 

Exp. versus Control 

Positive real difference 

(PRD), Better (B), Worse 

(W), Same (S) 

Part IV—Workload 

1. It was difficult to control vertical speed 

when on glide slope. 

0.7856 0.7902 W 

2. Operating the flight controls to avoid 

an upset condition when on final 

approach was a physically demanding 

effort. 

–0.5455 0.2852 B 

3. I felt it took a considerable amount of 

concentration and effort to safely execute 

the missed approach procedure. 

–1.2002 0.1108 B 

4. Keeping the aircraft on the glide slope 

without experiencing a pitch upset was a 

very demanding task. 

–0.8074 0.2034 B 

5. On final approach I spent so much time 

trying to fly the glide slope that I was 

unable to maintain good localizer course 

control. 

0.5674 0.7221 W 

Legend 

 Meets 95% confidence for rejecting H0

 Experimental group better over control group but < 95% confidence for rejecting H0 

 No difference between experimental and control group 

 Control group performance better than experimental group 
NASA/CR—2014-218320 35 

11.0 Atmospheric Turbulence Test Methodology 

The pilot-in-the-loop simulations could not duplicate all of the real-world conditions that could have 

first-order effects on system performance, such as those due to real atmospheric turbulence. Therefore, in 

order to assess system performance under these conditions and minimize the technical risk of an eventual 

flight-test validation program, the effects of atmospheric turbulence on ICEPro were studied. This led to a 

one year extension awarded under NASA’s Vehicle Systems Safety Technologies (VSST) project. This 

section summarizes the study which is fully reported in the Year 5 progress report that can be found in 

Appendix D. 

11.1 Experiment Design 

Four tasks and their measures of success (MOS) were defined in the ICEPro atmospheric turbulence 

test plan (Appendix E). These tasks involved studying the source of the scatter in RT-PID estimates in 

turbulence, understanding the effect of sensor noise and turbulence on RT-PID and D-ICES, determining 

the performance characteristics of a turbulence probe, and studying the effect of turbulence in different 

phases of flight. In order to test each task as described below, a test matrix was developed. The test matrix 

included representative flight phases in terms of airspeeds, attitude, and configuration. Utilizing D-Six 

scripting capabilities, a series of scripts were developed to explore the test matrix through batch testing. 

11.2 Task Descriptions 

This summary will describe the questions asked in each of the four tasks. For more detail, please see 

Appendix D. 

11.2.1 Task 1: Simulating Real World RT-PID Performance in Turbulence 

Determine whether the scatter in the turbulence estimates arise from, 1) colored noise in the 

regressor(s), and if so what is the source of the colored noise and in which regressor(s), 2) insufficient 

model structure, and if so how should the model structure be adjusted or, 3) some other means. In the 

system identification process, regressor(s) are defined as the independent variables. 

11.2.2 Task 2: Effect of Sensor Noise and Turbulence 

Modify, test, and validate the use of RT-PID and D-ICES algorithms with atmospheric turbulence as 

an input for various levels of atmospheric turbulence, and to identify an acceptable error in the wind 

measurement during turbulence for RT-PID and D-ICES algorithms. Additionally, verify MOS during 

straight and level flight for varying atmospheric turbulence levels and icing conditions. 

11.2.3 Task 3: Effect of Turbulence for Different Flight Phases at Selected Noise Level 

Modify, test, and validate the use of RT-PID and D-ICES algorithms with atmospheric turbulence as 

an input. Select an acceptable atmospheric turbulence measurement error based on findings from Task 2 

and commercially available atmospheric turbulence sensors. Verify MOS during each phase of flight for 

varying atmospheric turbulence levels and icing conditions. 

11.2.4 Task 4: Evaluate the Effectiveness of Manual Pilot Inputs Alone for Making High 

Confidence Vehicle State Estimates When Performing Typical Maneuvering Flight Tasks 

Under Varying Levels of Atmospheric Turbulence 

Determine the portion or portions of the maneuvers, which provide frequency content at the natural 

frequency of the short period, dutch-roll, and roll modes and thereby support the achievement of high 

confidence parameter estimates from RT-PID. This effort was not performed due to time and funding 

constraints.
NASA/CR—2014-218320 36 

11.3 Measures of Success 

To ensure that test objectives are attained, measures of success (MOS) have been formulated. A MOS 

is defined as a comparative requirement which must be satisfied by the results of testing. The overall 

measure of success for the primary and secondary objectives will be that the error bounds and accuracy in 

RT-PID and Theil inequality coefficients in D-ICES algorithms meets or exceeds its non-atmospheric 

turbulence performance in atmospheric turbulence. This success criterion focuses on the performance of 

these two algorithms and not on ICEPro’s utility to mitigate a potentially hazardous icing encounter or on 

the pilot’s performance during such encounter. It assumes that if the RT-PID and D-ICES algorithms 

perform equally in and out of atmospheric turbulence, the ICEPro system will perform equally. The 

following MOS are defined for the primary objective. 

RT-PID Algorithm 

1. Message accuracy 

Message error will be determined by the error bounds and accuracy of its associated stability and 

control coefficient. For example, the red (warning level) pitch degrade logic is a function of elevator 

control power 

CmGe t 0.25CmGe _ noice

The elevator control power must meet or exceed its non-atmospheric turbulence error bound and 

accuracy in atmospheric turbulence. This is repeated for all ICEPro messages. 

2. ISP accuracy 

ISP error will be determined by comparing the true ISP value with the estimated ISP value. ISP is a 

function of five stability and control coefficients: Longitudinal static stability, elevator control power, 

dihedral effect, aileron control power, and normal lift coefficient. The true ISP value is known since 

the five true stability and control parameters are known to the simulation. The ISP error must meet or 

exceed its non-atmospheric turbulence error bound and accuracy in atmospheric turbulence. 

D-ICES Algorithm 

1. Control deflection accuracy 

Control deflection error is determined by comparing the estimated deflection Theil inequality 

coefficient with the baseline no turbulence case. The control deflection error must meet or exceed its 

non-atmospheric turbulence error in atmospheric turbulence. 

11.4 Changes to ICEPro Logic 

Several specific features were added to the ICEPro module for the current effort. The new features 

included adding several input parameters within the D-Six variable list. These parameters permitted 

overriding ICEPro’s logic in the Monitor mode forcing the system into Identification mode. In addition, 

user specified surface buzzing times were created to override normal ICEPro buzzing logic. Buzzing 

refers to exercising the controls through a set of orthogonal multi-sine inputs to provide additional control 

content for RT-PID operation. Finally, the use of these new features was optional allowing ICEPro to 

operate in an unmodified logic set. 

Utilizing D-Six scripting capabilities, a series of scripts were developed to facilitate batch testing. 

These scripts interfaced an Excel file containing the test matrix, set up the simulation initial conditions, 

stepped through the simulation using control inputs provided by virtual pilot logic contained in the 
NASA/CR—2014-218320 37 

scripts, recorded selected time history variables, stopped the simulation, and exported the recorded data to 

MATLAB. The script virtual pilot logic used the differences between the desired flight path angle, 

heading, and airspeed and the current actual values as inputs to the elevator, aileron, and throttles as 

control inputs. Sideslip feedback to the rudder was also used to coordinate turns. Equations (11) show the 

basic control laws used by the virtual pilot. The gains were set to low values that worked satisfactorily for 

the no turbulence cases, but were less than optimal with turbulence. These values were used for all cases 

to minimize excessive control movements during turbulent conditions. For some cases, the throttle 

commands were set to zero to eliminate excessive throttle movements encountered with turbulent 

conditions. 

e K  K dt 3 G command J J desired  J  J J  J desired  J  (11a) 

Gacommand K\   \desired \  KIM (11b) 

Grcommand KEE (11c) 

K   V V K VV V      GThrottle desired (11d) 

11.5 Atmospheric Turbulence Probe 

In support of Task 3, a search was conducted to identify candidate COTS turbulence probes. Several 

probes were identified using state of the art LIDAR technology. Unfortunately, the accuracy of such 

technology requires an order of magnitude improvement for the present work. The Atmospheric 

Integrated Meteorological Measurement System AIMMS-20 probe by Aventech Research was chosen 

because 1) It provides the most accurate wind determination out of the candidate probes and 2) it is 

already part of GRC’s inventory. Table D.1 in Appendix D summarizes the AIMMS-20 performance 

based on an error analysis. 

12.0 Atmospheric Turbulence Results and Discussion 

A detailed discussion of Tasks 1 to 3 results can be found in the Year 5 progress report (Appendix D). 

The following results and discussion provide a short summary. 

12.1 Task 1 Simulating Real World RT-PID Performance in Turbulence 

Several practical issues related to real-time parameter estimation for a linear longitudinal dynamics 

model in atmospheric turbulence using indirect turbulence measurements were examined and discussed. 

The frequency response of the airflow vanes, wing response to atmospheric turbulence, and the structural 

response of the air data boom were identified as sources of colored noise. The frequency response of the 

airflow vanes for a wide variety of combinations of natural frequency and damping ratio did not adversely 

impact parameter estimation results in turbulence, except for CZq and CZGe. This was attributed to noisy 

explanatory variables in the atmospheric turbulence implementation and relatively low aerodynamic 

contributions as compared to CZD. Airflow vane dynamics do not substantially contribute to the scatter in 

the estimates, because all of the added dynamics are outside the range of frequencies analyzed. The 

structural modes of the boom lead to similar conclusions, because the natural frequencies of the boom 

structural modes are outside the range of frequencies used for dynamic modeling. 

A major source of colored noise was identified as the frequency-dependent upwash and time delay 

induced by the wing-bound vortex system and the longitudinal separation between the AOA measurement 

and the wing. It was also surmised that the frequency-dependent sidewash was a major source of colored 
NASA/CR—2014-218320 38 

noise. The frequency-dependent upwash and time delay appear to be significant contributors to biasing 

and increased scatter and uncertainty for parameter estimates in atmospheric turbulence. 

Practical issues were examined using data from a Twin Otter DHC-6 longitudinal linear simulation, 

with realistic noise sequences added to the computed aircraft responses. This allowed a clear view of the 

effect of each source of colored noise to the modeling problem, because the true values of the model 

parameters were known. This approach was used initially to show that real-time parameter estimation can 

be done accurately in all atmospheric turbulence conditions if the AOA and angle of sideslip 

measurements are accurate. Flight test data from the GRC DHC-6 Twin Otter aircraft was used to validate 

the effect of the identified colored noise sources. 

Based on these findings, several practical recommendations for flight testing in atmospheric 

turbulence are suggested, as well as areas for future study. First, mount the air data boom at the nose of 

the aircraft to minimize the effect of the wing-bound and wing-tip vortices. Second, if the air data boom is 

mounted on the nose of the aircraft, an upwash and time delay calibration as a function of frequency is 

required. Third, airflow angle measurements must be corrected for frequency dependent upwash, 

sidewash, and time delay corrections. 

The following Task 2 and Task 3 results were carried out assuming that these precautions had been 

taken care of. This assumed that the frequency response was adequate (Good assumption based on the 

results of this section for a well calibrated, stiff, nose mounted sensor). This left the sensor’s static 

performance to be evaluated in Task 2 and Task 3. In other words, if these precautions are taken, the RTPID estimates will behave similarly inside and outside of atmospheric turbulence. 

12.2 Task 2: Effect of Sensor Noise and Turbulence 

RT-PID 

x In general, the original MOS for message accuracy were not achieved since the errors in the 

estimates and in the error bounds increase as a function of turbulence intensity. A new MOS 

criterion was formulated where the error in the estimates and in the error bounds with 

atmospheric turbulence is tolerated as long as the ICEPro messages and AOA brackets are 

equivalently displayed inside and outside of turbulence. 

x In practice, for flight in zero, light, and moderate atmospheric turbulence the average airflow 

angles are similar. In severe turbulence, the average airflow angles are significantly different due 

to wing stall. The differences do not affect message accuracy, but do affect the AOA bracket 

accuracy. The data from stalled runs was not considered useful because stalled flight does not 

produce reliable data. As a result, all severe turbulence test conditions have been omitted from 

Task 2 analysis. 

x With the exception of the roll degrade message in calm atmosphere, there were no false positives 

or false negatives. The roll degrade message was caused by low pilot activity resulting in poor 

RT-PID estimates. In each of the turbulence cases, the additional aileron inputs provided by the 

virtual pilot improved the RT-PID estimates so that no false positives (roll degrade message) 

were annunciated. 

x In general the new MOS message criterion was considered to be successful. 

D-ICES 

x It was found that the application of turbulence and sensor noise on D-ICES had minimal impact 

on performance without filtering. 

ż Only one condition was found that yielded a false positive that would have caused the system 

to enter ID Mode when no icing was present. 

ż There was one condition where the elevator deflection Theil inequality Coefficient for an iced 

airplane failed to exceed the threshold and two where the value originally exceeded the 

threshold, but with time decreased until it was below the threshold. 
NASA/CR—2014-218320 39 

x However, when the state vectors were low pass filtered to remove higher frequency disturbances 

due to turbulence and noise, D-ICES accurately detected all icing conditions in the presence of 

turbulence and sensor noise. Filtering did not have any adverse ramifications on logic, thresholds, 

or other system parameters or functions. 

x Three potential false positive cases were found that showed the elevator Theil Coefficient built up 

over time to exceed the threshold when no ice was present. An examination of the data in this 

region led to supposition that the D-ICES inversion process had trouble determining the correct 

elevator deflection with pitch due to elevator characteristics at those flight conditions. This 

condition is specific to the Twin Otter flight model. 

Using the results of Task 2, it was determined that the objective to identify an acceptable error in the 

wind measurement was not required and that the performance of an air data probe such as the AIMMS-20 

probe was satisfactory. 

12.3 Task 3 Effect of Turbulence for Different Flight Phases at Selected Noise Level 

For the purpose of this task the four phases of flight are defined as: 

Phase 1. Flaps up straight and level flight at 110 KIAS and 2500 ft – 30 sec duration 

Phase 2. Flaps up straight and level flight at 110 KIAS and 2500 ft followed by a constant altitude flap 

transition (0° to 20°) and deceleration to 80 KIAS – 1 min duration 

Phase 3. Heading capture at 80 KIAS, 2500 ft, and flaps 20° from 090° to 360° – 30 sec duration 

Phase 4. Descent from 3000 ft at 75 KIAS and flaps 20° followed by a flap transition (30°) and 

deceleration to 70 KIAS with localizer and glide slope tracking – 2 min duration 

RT-PID 

x Phase 1 runs had no pitch degrade, climb limit, or flap limit messages for zero, light and moderate 

turbulence levels. The MOS criterion for AOA bracket was applied to Phase 1 and considered to 

be successful. 

x Phase 2 runs had no pitch degrade, yaw degrade, climb limit, or flap limit messages for zero, 

light, and moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 

2 and considered to be successful. 

x Phase 3 runs had no pitch degrade, climb limit, or flap limit messages for zero, light and moderate 

turbulence levels. The MOS criterion for AOA bracket was applied to Phase 3 and considered to 

be successful. 

x Phase 4 runs had no pitch degrade or yaw degrade messages for zero, light or moderate 

turbulence levels. 

x The MOS criterion for AOA bracket was applied to Phase 4: Applying the MOS criterion would 

be unsuccessful in this phase since a pilot is able to discern the unconservative variation in the 

AOA brackets. It is recommended that additional piloted runs be conducted to verify this 

conclusion by the virtual pilot. 

D-ICES 

x Phase 1 runs were a repeat of the Task 2 conditions and the results were similar. 

x Phase 2 runs experienced no false positives during this scenario. 

x In Phase 3 runs, a false negative was seen for the iced airplane with no turbulence, but correctly 

predicted positives for the low and moderate turbulence levels. The false positive was believed to 

be due to a singular condition where there was little difference between the iced and clean 
NASA/CR—2014-218320 40 

airplane’s control effectiveness at the chosen flight condition. This condition is specific to the 

Twin Otter flight model. 

x In Phase 4 runs, there were no false indications, even though the results remained close to a 

singular condition seen in the level turning flight case. However, the Theil inequality coefficient 

values are still sufficient to exceed the threshold for transitioning to ID mode for all of the iced 

conditions. 

x In most cases, D-ICES performed as designed in the presence of turbulence and noise. However, 

the exceptions noted above, especially the false negatives, do indicate that some further 

development of the system is warranted. One possible area to examine would be utilizing other 

terms related to the Theil inequality coefficient, namely the bias, variance and covariance terms in 

some way to develop a more robust detection scheme. 

x In general, it was found that the performance of the AIMMS-20 probe and the new MOS criterion 

were satisfactory for Task 3. 

13.0 Conclusions and Recommendations 

13.1 ICEPro System Design and Implementation 

In keeping with the NASA goal to improve the safety of commercial and general aviation aircraft, an 

envelope protection system was developed using no direct ice detection devices. Through the use of 

measured data, a priori information, and real-time stability and control estimates, this system computes an 

icing severity parameter. This parameter is used to trigger appropriate pilot cueing to warn the pilot of 

hazardous conditions and provide guidance for action to mitigate the situation. The main advantage of the 

system is that it uses data that are readily available in modern avionics systems and once adjusted to 

specific aircraft makes and models, and mated to modern display systems, should provide sufficient pilot 

cueing for envelope protection against icing related incidents. 

13.2 Pilot in the Loop Test and Evaluation of ICEPro 

The utility of the Icing Contamination Envelope Protection (ICEPro) system for mitigating a 

potentially hazardous icing condition was evaluated by 29 pilots using the ICEFTD. The pilots 

participating in this test were divided into two groups; a control group using baseline displays without 

ICEPro, and an experimental group using ICEPro driven display cueing. Each group flew identical 

precision approach and missed approach procedures with a simulated ice protection failure icing 

configuration. Pilot performance, workload, and survey questionnaires were collected for both groups of 

pilots. Results showed that real time assessment cues were effective in reducing the number of potentially 

hazardous upset events and in lessening exposure to loss of control following an incipient upset condition. 

Pilot workload with the added ICEPro displays was not measurably affected, but pilot opinion surveys 

showed that real time cueing greatly improved their situation awareness of a hazardous aircraft state. The 

conclusions from this test indicate that the experimental group of pilots, who flew with ICEPro displays, 

demonstrated significantly better control performance in dealing with aircraft handling problems under 

icing conditions and had better awareness of a hazardous aircraft state. Pilot workload was not affected by 

the additional display information provided by ICEPro, but pilot opinions of the implementation of that 

system indicate that improvements could be made. 

13.3 Atmospheric Turbulence and Sensor Noise 

The conclusions from this task confirm the hypothesis that the scatter in the estimates observed 

during flight in turbulence is due to airflow angle colored noise. Several important sources of colored 

noise were identified and tested. A major source of colored noise was identified as the frequencydependent upwash and time delay induced by the wing-bound vortex system and the longitudinal 
NASA/CR—2014-218320 41 

separation between the AOA measurement and the wing. It was also surmised that the frequencydependent sidewash was a major source of colored noise. The frequency-dependent upwash and time 

delay appear to be significant contributors to biasing and increased scatter and uncertainty for parameter 

estimates in atmospheric turbulence. 

These findings also disproved the hypothesis that additional model structure was needed. 

Additionally, several precautions should be taken for improved estimates in atmospheric turbulence. 

1) Mount the flow angle sensors as far forward of the wing as practical, preferably on the nose of the 

aircraft. This will minimize airframe flex and the amplitude of the upwash correction. 2) Calibrate the 

change in upwash and sidewash correction versus frequency. 3) Eliminate the use of booms for flow 

angle sensors. This will minimize boom flex. In general, it was found that the performance of the 

AIMMS-20 probe was satisfactory for Task 2 and Task 3. Acceptable noise error was therefore defined 

by current specification of AIMMS-20 probe. As long as booms are not used and the calibration for wing 

response to turbulence has been taken care of, ICEPro performed similarly inside and outside of 

atmospheric turbulence. 

Subsequent analysis of the RT-PID and D-ICES algorithms in Task 2 and Task 3 demonstrated good 

performance with few exceptions. Additional recommendations have been made to look at the exceptions. 

14.0 Recommendations 

1. Flight displays in this test were not optimized from a human factors standpoint, nor were the envelope 

protection cues coupled with a flight director. Based upon the opinions extracted from the pilot 

survey, pilot control performance and state awareness may be improved by addressing those issues in 

future pilot in the loop testing. 

2. Execute Task 4 in the one year VSST extension. This task evaluates the effectiveness of manual pilot 

inputs alone for making high confidence vehicle state estimates when performing typical 

maneuvering flight tasks under varying levels of atmospheric turbulence. 

3. Flight test to validate RT-PID findings based on colored AOA regressor findings. 

4. Flight test to validate ICEPro. 

References 

1. Green, Stephen D., “A Study of U.S. Inflight Icing Accidents and Incidents, 1978 to 2002,” 44th

AIAA Aerospace Sciences Meeting and Exhibit, 9-12 January 2006, Reno, NV, AIAA 2006-82. 

2. Nagel, David C., “Human Error in Aviation Operations,” Human Factors in Aviation, Chapter 9, eds. 

Weiner, E.L, Nagel, D.C., Academic Press, 1988. 

3. National Transportation Safety Board, Safety Recommendation to Mitigate the Existing Risk to the 

Cessna 208 Fleet during the Current Icing Season. A-06-01 through -03, January 17, 2006. 

4. National Transportation Safety Board, Safety Recommendation to Mitigate the Existing Risk to the 

Saab 340 Fleet When Operating in Icing Conditions. A-06-48/51, July 10, 2006. 

5. Russell, P., Pardee, J., “Joint Safety Analysis Team- CAST Approved Final Report Loss of Control 

JSAT Results and Analysis,” December 2000. 

6. Ratvasky, T.P., Van Zante, J.F., Sim, A., “NASA/FAA Tailplane Icing Program: Flight Test Report,” 

NASA/TP—2000-209908, DOT/FAA/AR-99/85, March 2000. 

7. Bragg, M.B., Basar, T., Perkins, W.R., Selig, M.S., Voulgaris, P.G., Melody, J.W., Sarter, N.B., 

“Smart Icing Systems for Aircraft Icing Safety,” AIAA 2002-0813, January 2002. 

8. Gingras, D.R., Barnhart, B., Ranaudo, R., Ratvasky, T., and Morelli, E., “Envelope Protection for InFlight Ice Contamination,” AIAA-2009-1458, NASA/TM—2010-216072, January 2009. 

9. Barnhart, B.P., Dickes, E., Gingras, D.R., and Ratvasky, T.P., “Simulation model Development for 

Icing Effects Flight Training,” SAE 2002-01-1527, NASA/TM—2003-212115. 

10. Ratvasky, T., Blankenship, K., Rieke, W., Brinker, D., “Iced Aircraft Flight Data for Flight Simulator 

Validation,” SAE-2002-01-1528, April 2002. 
NASA/CR—2014-218320 42 

11. Hart, S., and Lowell, S., “Development of NASA-TLX: Results of Empirical and Theoretical 

Research,” Human Mental Workload, eds. Hancock, P., and Meshkati, N., North Holland Press, 

Amsterdam, 1988, pp. 239-250. 

12. Theil, H. (1967). Economics and Information Theory. Amsterdam: North Holland. 

13. Morelli, E.A., “Real-Time Parameter Estimation in the Frequency Domain,” Journal of Guidance, 

Control, and Dynamics, Vol. 23, No. 5, September-October 2000, pp. 812-818. 

14. Morelli, E.A., “System IDentification Programs for AirCraft (SIDPAC),” AIAA-2002-4704. 

15. Klein, V and Morelli, E.A., Aircraft System Identification Theory and Practice, American Institute for 

Aeronautics and Astronautics Inc., Reston, VA, 2006. 

16. Barnhart, B.P., Dickes, E., Gingras, D.R., and Ratvasky, T.P., “Simulation model Development for 

Icing Effects Flight Training,” SAE 2002-01-1527. 

17. Ranaudo, R., Martos, B., Norton, B., Gingras, D., Barnhart, B., Ratvasky, T., and Morelli, E., 

“Piloted Simulation to Evaluate the Utility of a Real Time Envelope System for Mitigating In-Flight 

Icing Hazards,” presented at AIAA Atmospheric and Space Environments Conference, August 2–5, 

2010, Toronto, Ontario, Canada, AIAA 2010-7987. 

18. Sheskin, D. J., The Handbook of Parametric and Nonparametric Statistical Procedures, CRC Press, 

Boca Raton, FL, 1997, pp., 181-190. 

19. Trochim, William M., “Likert Scaling”. Research Methods Knowledge Base, 2nd Edition. 

October 20, 2006 

20. Federal Aviation Administration, Airline Transport and Aircraft Type Rating Practical Test Standards 

for Airplane, FAA-S-8081-5F, July 2008, pp 2-20 – 2-27. 

 
NASA/CR—2014-218320 43 

Appendix A.—Abbreviations and Symbols List 

Abbreviations 

AOA angle of attack 

D-ICES Dynamic Inversion Control Evaluation System 

ERAU Embry-Riddle Aeronautical University 

FCD Flight Control Display 

H0 null hypothesis 

ICEFTD Ice Contamination Effects Flight Training Device 

ICEPro Icing Contamination Envelope Protection system 

IPS Ice Protection System 

ISP Icing Severity Parameter 

J cost function 

LSE least-squared error 

PCE Pilot Coupling Event 

RT-PID Real Time Parameter Identification 

PFD Primary Flight Display 

RMS root-mean square 

SLD Super-cooled Large Droplet 

Symbols 

A stability matrix 

B control matrix 

C stability matrix 

ClE rolling moment due to sideslip angle derivative, wCl wE

ClGa rolling moment due to aileron deflection derivative, wCl wGa

ClGr rolling moment due to rudder deflection derivative, wCl wGr

Clp rolling moment due to nondimensional roll rate derivative, wCl w  pb / 2V

Clr rolling moment due to nondimensional yaw rate derivative, wCl w  rb / 2V

CmD pitching moment due to AOA derivative, wCm wD

CmGe pitching moment due to elevator deflection derivative, wCm wGe

Cmq pitching moment due to nondimensional pitch rate derivative, wCm w  qc / 2V

CND normal force due to AOA derivative, wCN wD

CNGe normal force due to elevator deflection derivative, wCN wGe

CNq normal force due to nondimensional pitch rate derivative, wCN w  qc / 2V

CnE yawing moment due to sideslip angle derivative, wCn wE

CnGa yawing moment due to aileron deflection derivative, wCn wGa

CnGr yawing moment due to rudder deflection derivative, wCn wGr
NASA/CR—2014-218320 44 

Cnp yawing moment due to nondimensional roll rate derivative, wCn w  pb / 2V

Cnr yawing moment due to nondimensional yaw rate derivative, wCn w  rb / 2V

CYE side force due to sideslip angle derivative, wCY wE

CYGa side force due to aileron deflection derivative, wCY wGa

CYGr side force due to rudder deflection derivative, wCY wGr

CYp side force due to nondimensional roll rate derivative, wCY w  pb / 2V

CYr side force due to nondimensional yaw rate derivative, wCY w  rb / 2V

U Theil inequality coefficient 

ak k

th row of matrix A 

bk k

th row of matrix B 

b reference span 

c mean aerodynamic chord 

t time 

u control vector 

u



 control vector predicted from inversion routine 

uk

~ control 

V true velocity, ft/s 

x state vector 

x0 state vector initial condition 

xk

~ state 

y, yˆ measured and predicted output 

Į alternate hypothesis 

Ȧ frequency, rad/s 

ș parameter vector 

T

ˆ estimated Parameter 

Vˆ confidence bound 
NASA/CR—2014-218320 45 

Appendix B.—Post-Test Survey Questionnaire 

Part I. Demographics 

Name: _______________________ Male Female 

Pilot Number: __________ 

Total Flight Hours: __________ 

Multi – Engine Hours: _________ 

FAA Ratings: Commercial Multi CFI CFII 

Aircraft Flown and Hours in Type: 

_____________________________________________________________________________________ 

_____________________________________ 

Icing related training and experience: 

1. In my all my prior flying experience, I encountered in-flight icing 

Never Rarely Sometimes Very often Always 

2. I felt that my prior icing related knowledge and experience would have adequately prepared me to 

perform this test scenario without the familiarization training I received on aircraft handling 

characteristics. 

Strongly Disagree Strongly Agree Disagree Undecided Agree 

3. The NASA videos and web based training materials provided me with information about icing that I 

had never known about before. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

4. Before this test, my icing related flight training was mostly focused on how to operate the ice 

protection system. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 
NASA/CR—2014-218320 46 

Part II. Situation Awareness 

1. My flight displays were adequate for me to determine when airframe icing was having an effect on 

aircraft characteristics. 

Never Rarely Sometimes Very often Always 

2. I felt as though I knew the minimum safe speed I could fly for a given wing flap setting within 5 kt. 

Never Rarely Sometimes Very often Always 

3. I knew how to adjust my pitch attitude to avoid a wing stall or a tail stall upset. 

Never Rarely Sometimes Very Often Always 

4. I wasn’t always sure which wing flap settings would allow a safe rate of climb in the event of an engine 

failure. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

5. I relied solely on aircraft control response to determine how icing affected pitch, yaw, or roll 

characteristics. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

6. I was confident that the final approach airspeed I chose to fly after I departed the final approach fix 

would prevent me from stalling. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

7. While I was putting the flaps down during the approach and landing, I felt I could effectively manage 

my control inputs and airspeed to avoid aircraft handling problems or tail stall upsets. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

8. The various colored bands on the airspeed tape were useful for me to safely fly the aircraft during the 

entire flight. 

Never Infrequently Sometimes Frequently Always 
NASA/CR—2014-218320 47 

9. I relied upon the stick shaker to prevent me from inadvertently stalling. 

Never Rarely Sometimes Very Often Always 

 

10. I could always tell when I was approaching a wing or tail stall condition. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

Part III. Implementation of Envelope Protection Cueing (Experimental ICEPro Group Only) 

1. I felt that it was important to combine the visual cues from ICEPro with aural alerts, and tactile 

feedback from the stick shaker. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

2. The ROL DGRD, PTCH DGRD, or YAW DGRD messages on the PFD did not provide useful 

information, and only cluttered the display. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

3. When flying at slow speeds, I could easily use the AOA bars as a good pitch control reference for 

safely flying the airplane. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

4. I had difficulty understanding the relationship between the AOA bars and the high and low speed carets 

on the airspeed tape. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

5. The color coded flight control surfaces on the flight control synoptic were more useful than the 

messages for assessing degraded control state. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

6. I favored using the airspeed carets rather than the AOA bars to remain within a safe operating 

condition. 

Never Infrequently Sometimes Frequently Always 
NASA/CR—2014-218320 48 

7. When I unintentionally got into an upset condition, the disappearance of all ICEPro messages except 

the AOA bars on the PFD did not affect my ability to recover the airplane. (Answer only if a wing stall or 

tail stall upset occurred). 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

8. When I extended the wing flaps and the red CLIMB LIM and FLAP LIM messages came on together, I 

immediately knew if an engine failed I would have to reduce my flap setting in order to climb. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

9. The occasional “buzzing” of the flight controls by ICEPro made flying the aircraft very difficult. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

10. I tended to confuse baseline airspeed limits with the airspeed limits (carets) that were posted by 

ICEPro. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

11. I tended to fly so as to keep my pitch attitude in the middle of the AOA bars when they were 

presented by ICEPro 

Never Infrequently Sometimes Frequently Always 

12. When the AOA bars started flashing, they immediately captured my attention. 

Never Infrequently Sometimes Frequently Always 

13. I felt that the airspeed carets from ICEPro were useful for helping me determine safe flight speeds 

when maneuvering during approach and landing. 

Never Infrequently Sometimes Frequently Always 

14. The low AOA cue on the PFD enabled me to avoid a tail stall upset. 

Never Infrequently Sometimes Frequently Always 
NASA/CR—2014-218320 49 

15. I immediately noticed when the CLIMB LIM and FLAP LIM messages were on at the same time. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

Part IV. Workload 

1. I found it difficult to control vertical speed when on glide slope. 

Never Infrequently Sometimes Frequently Always 

2. Operating the flight controls to avoid an upset condition when on final approach was a physically 

demanding effort. 

Never Infrequently Sometimes Frequently Always 

3. I felt it took a considerable amount of concentration and effort to safely execute the missed approach 

procedure. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

4. Keeping the aircraft on the glide slope without experiencing a pitch upset was a very demanding task. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

5. On final approach I spent so much time trying to fly the glide slope that I was unable to maintain good 

localizer course control. 

Never Infrequently Sometimes Frequently Always 
NASA/CR—2014-218320 50 
NASA/CR—2014-218320 51 

Appendix C.—Results of Survey Questions from Parts I, II, and IV 

Figure C.1.—Pilot demographics—Part I Survey questions 1 to 4 

Figure C.2.—Situational Awareness—Part II Survey questions 1 to 5

0

1

2

3

4

5

6

7

8

9

10

11

12

12345 12345 12345 12345

Response Frequency 

I-1 I-2 I-3 I-4 

Control Group (n=14)

Experimental Group (n=15)

0

1

2

3

4

5

6

7

8

9

10

11

12

1 2 3 4 5* 1 2 3 4 5* 1 2 3 4 5* 1* 2 3 4 5 1* 2 3 4 5

Response Frequency 

 II-1 II-2 II-3 II-4 II-5 

Control Group (n=14)

Experimental Group (n=15)

*Direction of responses that indicated better situational awareness 
NASA/CR—2014-218320 52 

Figure C.3.—Situational Awareness—Part II Survey questions 6 to 10 

Figure C.4.—Workload—Part IV Survey questions 1 to 5 

0

1

2

3

4

5

6

7

8

9

10

11

12

1 2 3 4 5* 1 2 3 4 5* 1 2 3 4 5* 1* 2 3 4 5 1 2 3 4 5*

Response Frequency 

 II-6 II-7 II-8 II-9 II- 10 

Control Pilot Group (n=14)

Experimental Group (n=15)

* Direction of responses that indicated better situational awareness 

0

1

2

3

4

5

6

7

8

9

10

11

12

1* 2 3 4 5 1* 2 3 4 5 1* 2 3 4 5 1* 2 3 4 5 1* 2 3 4 5

Response Frequency 

 IV-1 IV-2 IV-3 IV-4 IV-5 

Control Group (n=14)

Experimental Group (n=15)

* Direction of responses that indicated lower workload 
NASA/CR—2014-218320 53 

Appendix D.—One Year Atmospheric Turbulence Extension Final 

Report—Effect of Turbulence and Sensor Noise On ICEPro 

Progress Report: Year 5 

Borja Martos, Principal Investigator 

University of Tennessee Space Institute (UTSI) 

411 B. H. Goethert Pkwy 

Tullahoma, TN 37388-9700 

and 

Ryan Oltman, David Gingras, and Billy Barnhart 

Bihrle Applied Research (BAR) 

81 Research Drive, 

Hampton VA 23666 

Cooperative Agreement: NNX07AD58A 
NASA/CR—2014-218320 54 

List of Symbols and Abbreviations 

Abbreviations 

AIAA American Institute of Aeronautics and Astronautics 

AIMMS-20 Atmospheric Integrated Meteorological Measurement System-20 

AOA angle of attack 

ATP Airline Transport Pilot 

BAR Bihrle Applied Research 

COTS commercial off the shelf 

D-ICES Dynamic Inversion Control Evaluation System 

D-Six Bihrle Applied Research’s Simulation Environment Software 

I0 no ice condition 

I2 20 min ice accumulation condition 

ICEFTD Ice Contamination Effects Flight Training Device 

ICEPro Icing Contamination Envelope Protection System 

ID identification 

ISP icing severity parameter 

IVHM Integrated Vehicle Health Monitoring 

KIAS knots indicated airspeed 

LIDAR light detection and ranging 

MIL military handbook 

MOS measure of success 

NASA National Aeronautics and Space Administration 

GRC NASA Glenn Research Center 

LaRC NASA Langley Research Center 

NASA TLX NASA Task Load Index 

OBES Onboard Excitation System 

PFD Primary Flight Display 

PFLF power for level flight 

RMS root mean square 

ROC rate of climb 

RT-PID Real Time Parameter Identification 

SFTE Society of Flight Test Engineers 

T0 no turbulence condition 

T1 light turbulence condition 

T2 moderate turbulence condition 

T3 severe turbulence condition 

TMS Turbulence Model Structure 

TOGA Take Off - Go Around 

UTSI University of Tennessee Space Institute 

VIAS velocity indicated airspeed 

VMC visual meteorological conditions 

VSST Vehicle Systems Safety Technology 

Symbols 

AX longitudinal acceleration 

AY lateral acceleration 

AZ vertical acceleration 
NASA/CR—2014-218320 55 

c mean aerodynamic chord 

ClE dihedral effect stability coefficient 

ClGa aileron effectiveness stability coefficient 

CmDG gust longitudinal static stability coefficient 

CmDS rigid body longitudinal static stability coefficient 

CmD longitudinal static stability coefficient 

CmO pitch stability coefficient 

Cmq pitch damping stability coefficient 

CmGe elevator effectiveness stability coefficient 

CND normal lift coefficient 

CNGr rudder effectiveness stability coefficient 

deU Theil inequality coefficient 

g gravitational acceleration constant 

HP pressure altitude 

Vg

G aircraft velocity relative to the earth 

Vt

G

 true airspeed 

Vw

G true wind velocity vector 

VD velocity down 

VE velocity east 

WD wind velocity, down 

WE wind velocity, east 

WN wind velocity, north 

VN velocity north 

VS stalling airspeed 

VT true airspeed 

p roll rate 

q dynamic pressure 

q pitch rate 

r yaw rate 

u longitudinal velocity 

v lateral velocity 

w vertical velocity 

ǻ change 

Į angle of attack 

Įm measured angle of attack 

ȕ angle of sideslip 

ȕm flow sideslip angle 

GF flap deflection 

Ge elevator deflection 

T pitch angle 

I roll angle 

ȥ yaw angle 
NASA/CR—2014-218320 56 

D.1 Abstract 

A team comprised of personnel from The University of Tennessee Space Institute, Bihrle Applied 

Research, Inc., NASA Glenn Research Center (GRC), and NASA Langley Research Center (LaRC), has 

successfully developed a prototype Icing Contamination Envelope Protection System (ICEPro). The ICEPro 

system facilitates flight envelope protection by making continuous real time vehicle state assessments, 

which are synthesized into flyable pilot cueing along with visual and aural alerts during in-flight icing 

conditions. The simulation could not duplicate all of the real-world conditions that could have first-order 

effects on system performance, such as those due to real atmospheric turbulence. Therefore, in order to 

assess system performance under these conditions and minimize the technical risk of an eventual flight-test 

validation program, the effects of atmospheric turbulence on ICEPro are studied in this extension. 

This extension focused on the impact turbulence has on the ICEPro system. Turbulence will degrade 

the quality of air data sensing. This study explored the accuracy of ICEPro’s ice detection and parameter 

estimation with perfect sensing and degraded sensing to determine an acceptable amount of degradation 

that would allow the ICEPro system to function correctly. Finally, after an acceptable error in air data 

collection was defined, it would be possible to determine whether the tolerance was obtainable in a 

commercially available air data sensor. 

D.2 Introduction 

D.2.1 Research Background 

D.2.1.1 Previous Studies and Results 

A team comprised of personnel from The University of Tennessee Space Institute (UTSI), Tullahoma, 

Tennessee, Bihrle Applied Research Inc (BAR), Hampton, Virginia, NASA Glenn Research Center (GRC), 

Cleveland, Ohio, and NASA Langley Research Center (LaRC), Hampton, Virginia, has successfully 

developed a prototype Ice Contamination Envelope Protection System (ICEPro). ICEPro was developed 

using a DH-6 Twin Otter simulation for development and evaluation. The aerodynamic math models of the 

Twin Otter were derived from a series of wind tunnel tests for both the clean and iced airplanes. The math 

models were then validated against NASA flight data for both the clean aircraft and the test aircraft flown 

with artificial ice shapes. The Twin Otter was chosen because of the availability of the flight test data. 

The ICEPro system facilitates flight envelope protection by making continuous real time vehicle state 

assessments, which are synthesized into flyable pilot cueing along with visual and aural alerts during in-flight 

icing conditions. Detection of degraded aircraft stability and control and performance due to icing is carried 

out by a dynamic inversion control evaluation system (D-ICES) that compares expected aircraft response from 

a-priori knowledge base with the current response. When differences reach defined thresholds, real time 

parameter identification (RT-PID) methods are invoked to estimate current state parameters, which 

continuously support pilot cueing and alerts. The development effort included simulation-based design, testing, 

and verification. A pilot in-the-loop study was conducted to gather pilot performance data and opinions of the 

utility of ICEPro during simulated icing encounters. Results of the study indicated that the system performed 

as expected and pilot performance benefited from the envelope protection cues. 

Figure D.1 presents a flow chart of the ICEPro system. (Some aspects of the system such as mode 

latching/delatching and logic for displaying messages, logic to force flushing the D-ICES and RT-PID 

buffers, etc. are not shown for clarity). The system starts in Monitor mode where D-ICES calculates, via 

an inversion technique using an a priori clean aircraft aerodynamic model, what the elevator deflection 

should be for the flight condition. This is compared with the actual elevator deflection using a statistical 

parameter called the Theil inequality coefficient. If this coefficient exceeds its threshold the system 

transitions to the ID mode. In the ID mode, the Real Time Parameter Identification (RT-PID) module is 

started and it predicts, based on flight state parameters, the linearized aerodynamic coefficients. These 

coefficients in conjunction with comparable clean and iced coefficients provided by D-ICES are used to 

calculate an Icing Severity Parameter (ISP). If the ISP becomes larger than its threshold, the system 

transitions to the Report mode where pilot cues and messages are displayed indicating the degraded 

aircraft’s condition and providing limits for safe operation of the aircraft. 
NASA/CR—2014-218320 57 

Figure D.1.—ICEPro System Flow Chart 

D.2.1.2 Current Year Five Study 

The simulation used for the development and evaluation did not duplicate all of the real-world 

conditions that could have first-order effects on system performance, such as those due to real 

atmospheric turbulence. Therefore, in order to assess system performance under these conditions and 

minimize the technical risk of an eventual flight-test validation program, the effects of atmospheric 

turbulence on ICEPro are studied in this extension. 

This extension focused on the impact turbulence has on the ICEPro system. Turbulence will degrade 

the quality of air data sensing. This study explored the accuracy of ICEPro’s ice detection and parameter 

Yes 

Yes 

Yes 

No 

a priori aero 

coefficients PID aerodynamics 

Yes 

No 

Monitor Mode 

IMOD=0 

Flight Data 

IMODE = 1 

Go to ID Mode 

IMODE = 0 

Stay in Monitor Mode 

ID Mode 

IMOD=1

Start RTPID 

Calculate ISP 

įe deviation) > 

threshold 

IMODE = 0 

Go to Monitor Mode 

IMODE = 1 

Stay in ID Mode 

IMODE = 2 

Go to Report Mode

Report Mode 

IMOD=2 

Update ISP 

IMODE = 1 

Go to ID Mode 

No

Yes 

ISP > 

Report 

Threshold 

No

No

ISP >Report

Threshold 

Deviation > 

Threshold 

Check individual Stab & 

Control values + ROC 

Update Display 

Cautions /Warnings 

įe deviation > 

threshold 

D-ICES Data 

(controls)
NASA/CR—2014-218320 58 

estimation with perfect sensing and degraded sensing to determine an acceptable amount of degradation 

that would allow the ICEPro system to function correctly. Finally, after an acceptable error in air data 

collection was defined, it would be possible to determine whether the tolerance was obtainable in a 

commercially available air data sensor. 

D.2.2 Research Objectives 

Four tasks and their measures of success (MOS) have been defined in the ICEPro atmospheric 

turbulence test plan (see Appendix E). This summary will describe the questions asked in each of the first 

three tasks and a brief introduction into the software tools developed to accomplish this work. The effort 

on Tasks 1 to 3 exhausted the allowed time and remaining funds, therefore no Task 4 results were 

included. 

D.2.2.1 Task 1: Simulating Real World RT-PID Performance in Turbulence 

Determine whether the scatter in the turbulence estimates arise from, 1) colored noise in the 

regressor(s), and if so what is the source of the colored noise and in which regressor(s), 2) insufficient 

model structure, and if so how should the model structure be adjusted or, 3) some other means. In the 

system identification process, regressor(s) are defined as the independent variables. 

D.2.2.2 Task 2: Effect of Sensor Noise and Turbulence 

Modify, test, and validate the use of RT-PID and D-ICES algorithms with atmospheric turbulence as 

an input for various levels of atmospheric turbulence, and to identify an acceptable error in the wind 

measurement during turbulence for RT-PID and D-ICES algorithms. Additionally, verify MOS during 

straight and level flight for varying atmospheric turbulence levels and icing conditions. 

D.2.2.3 Task 3: Effect of Turbulence for Different Flight Phases at Selected Noise Level 

Modify, test, and validate the use of RT-PID and D-ICES algorithms with atmospheric turbulence as 

an input. Select an acceptable atmospheric turbulence measurement error based on findings from Task 2 

and commercially available atmospheric turbulence sensors. Verify MOS during each phase of flight for 

varying atmospheric turbulence levels and icing conditions. 

D.2.2.4 Changes to ICEPro Logic 

Several specific features were added to the ICEPro module for the current effort. The new features 

included adding several input parameters within the DSix variable list. These parameters permitted 

overriding ICEPro’s logic in the Monitor mode forcing the system into identification mode. In addition, 

user specified surface buzzing times were created to override normal ICEPro buzzing logic. Buzzing 

refers to exercising the controls through a set of orthogonal multi-sine inputs to provide additional control 

content for RT-PID operation. Finally, the use of these new features was optional allowing ICEPro to 

operate in an unmodified logic set. 

Utilizing DSix scripting capabilities, a series of scripts were developed to facilitate batch testing. These 

scripts interfaced an Excel file containing the test matrix, set up the simulation initial conditions, stepped 

through the simulation using control inputs provided by virtual pilot logic contained in the scripts, recorded 

selected time history variables stopped the simulation, and, exported the recorded data to MATLAB. The 

script virtual pilot logic used the differences between the desired flight path angle, heading, and airspeed and 

the current actual values as inputs to the elevator, aileron, and throttles as control inputs. Sideslip feedback 

to the rudder was also used to coordinate turns. Equations (D.1) show the basic control laws used by the 

virtual pilot. The gains were set to low values that worked satisfactorily for the no turbulence cases, but 

were less than optimal with turbulence. These values were used for all cases to minimize excessive control 

movements during turbulent conditions. For some cases, the throttle commands were set to zero to eliminate 

excessive throttle movements encountered with turbulent conditions. 

e Kp    Kp dt 3 G command Jdesired  J  J  Jdesired  J   (D.1a) 
NASA/CR—2014-218320 59 

Gacommand K\   \desired  \  KIM (D.1b) 

Grcommand KEE (D.1c) 

K   V V K VV V   GThrottle desired    (D.1d) 

D.3 Discussion of Study and Results 

There are at least two ways to account for turbulence in system identification algorithms, 1) model the 

turbulence mathematically and estimate the corresponding parameters and 2) measure the turbulence. In the 

first approach, the identification of dynamic models containing turbulence is classified as a system with state 

and measurement noise. In the second approach, favored in this work, turbulence is measured indirectly 

since air turbulence is imbedded in the true wind velocity vector. This approach is favored because it makes 

it possible to use commercial off the shelf sensors to measure the atmospheric turbulence and treat it as a 

measured explanatory variable in the system identification problem. The true wind velocity vector, Vw

G

’

 of 

atmospheric air relative to the surface of the Earth is found from the vector subtraction: 

V VV wte  G GG (D.2) 

Where Vt

G the true aircraft velocity relative to the air mass and Ve

G

is the velocity of the aircraft with respect 

to the Earth. Since Vw

G is the difference between two large quantities it is essential that Vt

G

 and Ve

G

 are 

measured as accurately as possible in order to minimize errors in Vw

G . The details of these calculations 

can be found in flight test engineering handbooks (Refs. 1 to 3) The most obvious difficulty in measuring 

air turbulence from an aircraft is the removal of the aircraft’s velocity with respect to the Earth. 

Additional difficulties arise since the aircraft distorts the airflow around itself. This requires precise 

measurement of and V V t e

G G . Any inaccuracies in the measurements or calibrations will affect the derived 

wind, and consequently the aerodynamic estimates. It is only recently that COTS technology has been 

developed with sufficient measurement accuracy. 

A search was conducted to identify candidate COTS atmospheric turbulence sensors. Several sensors 

were identified using state of the art LIDAR technology with an airflow angle accuracy of 1.0°. In 

comparison, flight test air data booms with traditional air data technology are calibrated to an accuracy of 

0.1°. The cutoff accuracy for this work was set at 0.1°. Unfortunately, the accuracy of LIDAR based 

sensors requires an order of magnitude improvement to be useful for the present work. Several additional 

sensors were identified using traditional air data technology. The AIMMS-20 probe by Aventech 

Research was chosen because 1) It provides the most accurate wind determination out of the candidate 

probes and 2) it is already part of GRC’s inventory. Table D.1 summarizes the AIMMS-20 performance 

based on an error analysis. 

Turbulence was tested as calm, light, moderate, and severe. Turbulence implementation and levels 

were defined per MIL-F-8785C and were based on the Dryden Turbulence Spectrum. According to MILF-8785C, the probability of exceeding light turbulence levels is between 10–1 and 10–2, for moderate 

turbulence, the probability is approximately 10–3, and for severe turbulence it is approximately 10–5, as 

shown in Figure D.2 (Ref. 4) derived from MIL-F-8785C. Therefore, severe turbulence levels are not 

very likely to be encountered. The RMS turbulence wind speed values are also shown in Figure D.2 as a 

function of altitude. The turbulence velocity components were generated randomly within the guidelines 

of the Dryden spectrum and added to the aircraft velocity components to generate the total velocity 

components. A random number was used to generate the velocity components from the turbulence 

spectrum. By specifying a constant initial random number seed, the sequence of random numbers become 

reproducible allowing an “apples to apples” comparison of results. This method is referred to as a “fixed” 
NASA/CR—2014-218320 60 

seed. Likewise, using a varying random number seed results in unique sequences of random numbers 

from test to test. The three velocity components from the turbulence calculations were added to their 

appropriate wind frame velocity components (u, v, w) that were generated from the aircraft dynamics so 

that the velocity components used for the vehicle motion calculations included the components from the 

turbulence model. 

TABLE D.1.—AIMMS-20 PROBE PERFORMANCE 

Parameter 1 Sigma Error 

Longitudinal acceleration, AX, (g) ....................................... 0.003 

Lateral acceleration, AY, (g) ................................................. 0.003 

Normal acceleration, AZ, (g) .................................................. 0.05 

Angle of aWWDFNĮGHJ ........................................................ 0.10 

Angle of sLGHVOLSȕGHJ ..................................................... 0.10 

Pitch rate, q, (dps) .................................................................. 0.01 

Roll rate, p, (dps) ................................................................... 0.07 

Yaw rate, r, (dps) ................................................................... 0.01 

3LWFKDWWLWXGHșGHJ............................................................ 0.20 

Roll attitude, ࢥ,) deg) ............................................................. 0.20 

<DZDWWLWXGHȥGHJ ............................................................ 0.20 

GPS velocity down, VD, (mps) .............................................. 0.10 

GPS velocity north, VN, (mps) ............................................... 0.10 

GPS velocity east, VE, (mps) ................................................. 0.10 

True airspeed, VT, (mps) ........................................................ 0.20 

Wind velocity down, WD, (mps) ............................................ 0.30 

Wind velocity north, WN, (mps) ............................................ 0.30 

Wind velocity east, WE, (mps) ............................................... 0.30 

Figure D.2.—Turbulence Definition (derived from MIL-F-8785C). 
NASA/CR—2014-218320 61 

D.3.1 Task 1 

The following results are representative of the effect of turbulence on aircraft system identification. 

Data were collected using NASA’s AIRSTAR T2 model during atmospheric turbulence conditions. 

NASA’s T2 model is a 5.5 percent dynamically-scaled generic transport model. It is outfitted with typical 

flight test instrumentation and air data booms with flow vanes on each wingtip. In Figure D.3, each color 

represents a different run, and only the last estimate of each real-time parameter estimation run is shown. 

A total of nine runs are represented; six in high turbulence and three in low turbulence. 

Simulating the correct cause and effect due to atmospheric turbulence on aircraft parameter 

identification is an ongoing research topic. It is known that colored measurement noise causes scatter and 

bias (Refs. 5 and 6) in the parameter estimates, which is observed in Figure D.3 during repeated flight test 

maneuvers in atmospheric turbulence. However, it is unknown whether the scatter in the turbulence 

estimates arise from 1) colored noise in the regressor, and if so what is the source of the colored noise and 

in which regressor 2) inadequate model structure, and if so how should the model structure be adjusted or 

3) some other means. 

D.3.1.1 Hypothesis 

Colored AOA regressor noise and/or inadequate model structure is the cause of scatter in aircraft 

system identifcaiton estimates in atmospheric turbulence. 

D.3.1.2 Sources of Scatter in Estimates 

Based on the hypothesis a literature search was carried out to examine previous related work. The 

results are summarized in the following sections. 

D.3.1.2.1 Colored Regressor Noise 

Based on the literature search, the most likely causes of colored regressor noise are the wing 

aerodynamic response, the frequency response of long air data booms, and the frequency response of 

airflow angle vanes in atmospheric turbulence. 

Figure D.3.—Effect of Turbulence on Modeling Results

(Dr. Eugene A. Morelli, personal communication, April 24, 2011) 

 

 

ˆClD

ˆCmD

T-2 FLT 15 C21 WT04

(High Turbulence)

T-2 FLT 41 C18 WT02 

(Low Turbulence) 

ˆClD

ˆCmD
NASA/CR—2014-218320 62 

D.3.1.2.1.1 Flexure of AirData Boom 

During low-level flight tests over the terrestial and marine boundary layer, a Long EZ research 

aircraft was used to measure atmospheric turbulence (Ref. 7). Details of the aircraft and its 

instrumentation system are found in the cited reference. The boom motion with respect to the center of 

gravity (accelerometers were mounted at each location) is summarized in the following figure (Ref. 7) 

during light atmospheric turbulence. The phase trace compares fuselage and probe accelerometer sensors, 

whereas the magnitude traces represent each individual accelerometer sensor. 

Figure D.4 illustrates the complexity of motion for this particular boom-airframe combination. 

Significant difference is apparent in Figure D.4 between the vertical motion of the probe and the CG. 

Below 0.6 Hz the coherence between the two accelerations is 1.0 with zero phase shift. Between 0.6 Hz 

and 2.0 Hz the coherence decreases and the phase changers from zero to –45°. Beyond 6 Hz there is no 

significant relationship between the two accelerations. In other words, in the frequency range of interest 

for rigid body aircraft dynamics (0.1 to 2.0 Hz), the drop in coherence and phase indicates that the AOA 

LVGLIIHUHQWEHWZHHQORFDWLRQV7KLVLQWXUQUHVXOWVLQFRORUHGQRLVHLQWKHĮUHJUHVVRU1RGRXEWWKHUH

would be similarities for other boom-airframe combinations but with different frequency responses. 

In order to investigate the in-flight structural response of the air data boom, airflow angle flight data 

collected in a DHC-6 Twin Otter during multiple parameter etstimation maneuvers was analyzed in the 

frequecy domain using Fourier coefficients. Figure D.5 shows the frequency response data for the entire 

frequency range, up to the Nyquist frequency at 25 Hz. The longitudinal and lateral natural frequencies for 

the dominant structural mode of the air data boom are approximately 8 and 9 Hz. These natural frequencies 

are associated with the AOA and angle of sideslip measurements respectively. In Figure D.5, the Fourier 

coeffcients for the AOA near 8 Hz are at the noise floor. Therefore, the structural response of the nose boom 

on which the AOA sensor is mounted, and from fuselage bending, is not a source of colored noise in the 

AOA. Similar results were calculated for angle of sideslip but the results are not shown. 

Figure D.4.—Comparison between vertical accelerations of the boom 

and airplane center of gravity 
NASA/CR—2014-218320 63 

Figure D.5.—Fourier coefficients for measured AOA 

D.3.1.2.1.2 Wing Response to Turbulence 

During flight tests in the marine boundary layer, a Twin Otter research aircraft was used to study the 

aerodynamic effects on wind turbulence measurements(Ref. 8). Details of the aircraft and its 

instrumentation system are found in the cited reference. Although mounted on the wing the airdata boom 

was not used as the primary source of air data measurements. Instead a flush air data system was mounted 

on the nose of the Twin Otter. The flush air data system avoids the vibration and frequency response 

problems of a long boom mounted on the nose of the aircraft (Ref. 9). The wing response to atmospheric 

turbulence can be summarized in Figure D.6 during a 30 km flight in light atmospheric turbulence. Figure 

D.6 represents the estimated AOA scale upwash correction versus frequency. 

The frequency dependant upwash correction is caused by chordwise and spanwise upwash varations 

as a result of the wing vortex system response to atmospheric turbulence (Refs. 8 and 9). The frequency 

scale is the reduced frequency kb based on the half wing span where Z is the angular frequency, b is the 

wingspan, and U is the airspeed. The AOA upwash correction (1 + ku)

–1 is explained by examining a 

sensor model: 

D  D D M uT B   1 k (D.3) 

where 

DB = AOA bias 

DM = measured AOA 

DT = true AOA 

In Figure D.6, the vertical axis (1 + ku)

–1 represents the correction required to calculate the true AOA 

from measured data. 

   1

T u MB 1 k  D  D D (D.4) 

At low frequencies (representing flight in a calm atmosphere) the scale factor is approximately 0.75 

and increases to 0.90 as kb approaches 2.0. A kb value of 2.0 (using the DHC-6 Twin Otter wing span and 

110 kt cruise airspeed) corresponds to approximately 2.0 Hz. Therefore, over the frequency range of 
NASA/CR—2014-218320 64 

interest for rigid body aircraft dynamics (0.1 to 2.0 Hz), the scale factor changes from 0.75 to 0.90 due to 

atmospheric turbulence. It was also surmised that the frequency-dependent sidewash was a major source 

of colored noise but this was not tested. 

Figure D.7 depicts the time delay as a function of frequency. The time delay is caused by the 

separation between the nose boom mounted airflow vanes (measurement position) and the wing tip (lift 

and upwash generation), along which turbulence eddies travel. 

The magnitude of the time delay represents the correction required to calculate the true AOA from 

measured data. In the frequency range of interest for rigid body aircraft dynamics (0.1 to 2.0 Hz), the time 

delay is approximately 0.1 sec. 

Figure D.6.—Angle of attack upwash versus frequency

Figure D.7.—Time delay versus frequency 

(1+

ku)

–1

kb = Z*(b/2)/U
NASA/CR—2014-218320 65 

D.3.1.2.1.3 Frequency Response of Airdata Vanes 

Although References 10 to 12 were found to determine the frequency response of airdata vanes via 

natural frequency and damping ratio, no analysis was found confirming that the frequency response of 

airdata vanes was adequate in atmospheric turbulence. 

D.3.1.2.2 Model Structure 

It is conceivable that the aerodynamic coefficients are different in atmospheric turbulence that in 

smooth air although such differences have not been documented in flight (Ref. 13). The new model 

structure—referred to as the Turbulence Model Structure (TMS)—includes new coefficients CmDS and 

CmDG.These parameters represent the nondimensional SLWFKLQJPRPHQWGXHWRĮS (rigid body dynamics) 

and the nondimensional SLWFKLQJPRPHQWGXHWRĮG (atmospheric turbulence) respectively. Similar 

coefficients were implemented for the normal force. These coefficients represent different contributions 

due to the effect of atmospheric turbulence on the aircraft. Stated differently, knowing the effect of 

atmospheric WXUEXOHQFHRQĮVHQVRUGRHVQRWLPSO\WKDWWKHHIIHFWRIatmospheric turbulence on the 

aircraft response can be calculated by simply mXOWLSO\LQJĮE\WKHXVXDO ĮFRHIILFLHQWV,WLVDVVXPHGWKDW

WKHHIIHFWRIWXUEXOHQFHRQDLUFUDIWUHVSRQVHLVSURSRUWLRQDOWRWKHHIIHFWRIWXUEXOHQFHRQWKHĮVHQVRU

therefore, turbulencHFRHIILFLHQWVDUHHVWLPDWHGXVLQJWXUEXOHQWĮDVDUHJUHVVRUWRFDSWXUHLWVHIIHFW6R

LW¶VQRWUHDOO\DWXUEXOHQWĮFRHIILFLHQWDVPXFKDVLWLVDQHIIHFWRIWXUEXOHQFHXVLQJWKHEHVWPHDVXUHRI

turbulence. Both aerodynamic model structures are presented in Section D.3.1.3.1. 

D.3.1.3 Modeling 

For this study, a simplified Twin Otter aircraft simulation was generated outside the DSIX software 

environment using the MATHWORKS SIMULINK software. The simulation simplified the nonlinear 

equations of motion (Ref. 14) to 3 degrees of freedom (3-DOF) longitudinal dynamics and a linear 

aerodynamics model. The simulation rate was 50 Hz and the pertinent output variables were corrupted 

with gaussian noise with magnitudes that resulted in a signal to noise ratio of 50 for the measured outputs. 

The use of this simplified simulation was warranted to avoid changes to the high fidelity DSIX database 

as model structure changes were investigated. 

D.3.1.3.1 Model Equations 

For reference purposes, a detailed listing of the 3-DOF longitudinal equations of motion (Ref. 14) for 

a symmetric rigid aircraft is included. 

Force Equations: 

sin ( ) / X u qw g X T m   T  (D.5a) 

cos ( ) / w qu g Z T m  T  Z  (D.5b) 

Moment Equation: 

Y T Iq M M   (D.6) 

Kinematic Equation: 

T q  (D.7) 

Navigation Equations: 

cos sin Npu w  T T (D.8a) 
NASA/CR—2014-218320 66 

hu w T T sin cos  (D.8b) 

Atmospheric Turbulence Contributions: 

aerodynamics turbulence uu u  (D.9a)

ww w  aerodynamics turbulence (D.9b)

Where the atmospheric turbulence velocity components, uturbulence and wturbulence, were calculated using 

MIL-F-8785C specification and the Dryden Turbulence SIMULINK block for calm, light, moderate, and 

severe turbulence. Nondimensional aerodynamic force and moment coefficients for an aircraft can be 

computed from flight measurements as follows: 

C ma T qS X XX    (D.10a) 

C ma T qS Z ZZ    (D.10b) 

C I q M qSc mY T  > @  / (D.10c) 

The nondimensional aerodynamic force coefficients are transformed from the body axis to the wind 

axis as follows: 

cos sin CC C LZ X  D D (D.11) 

For local modeling over a short time period, the force and moment coefficients computed from 

Equations (D.10) can be modeled using linear expansions in the aircraft states and controls: 

LL SL L e O C C C eC D G 'D  'G  (D.12a) 

2 mm Sm m m q eO

qc C C C C eC

V D G

' 'D   'G  (D.12b) 

Atmospheric Turbulence Model Structure 

LL SL GL L G e O C C C C eC DD G 'D  'D  'G  (D.12c) 

2 mm Sm Gm m m G q eO

qc C C C C C eC

V DD G

' 'D  'D   'G  (D.12d) 

7KHǻQRWDWLRQLQGLFDWHVSHUWXUEDWLRQIURPD reference condition. In Equations (D.12b) and (D.12d), 

mO C represents the nondimensional pitching moment at a reference condition, and similarly for the other 

expansions. 

D.3.1.3.2 Wing Response to Turbulence 

In order to investigate the effects of upwash correction as a function of frequency and time delay as a 

function of frequency the recursive Fourier transform in the RT-PID algorithm was modified. The 

recursive Fourier transform is defined as: 

2 1   1 1 k j fn t X X xe n nn S '      (D.13) 
NASA/CR—2014-218320 67 

Where the frequency vector is defined as f = [0.1:0.04:2.0], the time delay is modeled as: 

 

 

2 *timedelay

2 1

1 1

k

j

j fn t

n nn

f e

X X xe

 S

 S '

 

I 

 I   (D.14) 

Where time delay, is a vector of time delays for each frequency f = [0.1:0.04:2.0]. The upwash correction 

1 + ku( f )is a vector of upwash corrections for each frequency f = [0.1:0.04:2.0]. The updated recursive 

Fourier transform is defined as: 

  2 1   1 1 1 k j fn t X X x kf fe n nn u  S '

   a  o aI o ¬ 1⁄4¬ 1⁄4   (D.15) 

D.3.1.3.3 Vane Sensor Model 

In order to investigate the frequency response of airflow angle vanes, a second-order airflow vane 

sensor dynamic model was studied. A natural frequency of Zn = 9 Hz and a damping ratio ] = 0.35 were 

chosen as representative values for a dynamic model of the airflow vanes (Ref. 10 to 12). Typical natural 

frequency and damping ratio values range from 5 to 20 Hz and 0.2 to 0.6, respectively. Equation (D.6) 

represents a second order continous-time transfer function: 

 

2

2 2 2

n

n n

G s

s s

Z  ]Z  Z

(D.16)

The transfer function representation does not support non-zero initial conditions; therefore, the vane 

sensor model was converted to a state space representation which supported a trim D initial condition. 

D.3.1.4 Testing 

The hypothesis was tested using the SIMULINK model described and the following test conditions: 

1. Baseline run with a perfect AOA measurement and the aerodynamic model structure defined in 

Equation (D.12). 

2. Based on findings in Section D.3.1.3.3, repeat test condition 1 with a representative frequency 

response model for the airflow angle vanes. 

3. Repeat test condition 2 with varying AOA magnitude and time delay as a function of frequency. 

4. Based on findings in Section D.3.1.2.2, add turbulence coefficients to the aerodynamic model 

structure and DSHUIHFWĮPHDVXUHPHQW

D.3.1.5 Results and Analysis 

The following parameters are common to all four test conditions. 

1. Aircraft trimmed at 10,000 ft and 140 kt 

2. 20 sec runs 

3. Orthogonal multi-sine inputs (Ref. 14) invoked from 1 to 11 sec 

4. Model parameter estimates taken at the end of each 20 sec run 

5. 200 runs per atmospheric turbulence intensity (calm, light, moderate, and severe) 

6. Fourier transform frequency spacing 0.04 Hz 

7. Fourier transform frequency range [0.1 2.0] Hz 

8. Signal-to-noise ratio of 50 

9. 50 Hz data rate 
NASA/CR—2014-218320 68 

The following comments are common to all of the figures associated with test conditions one to four: 

1. Green line—truth value of the parameter 

2. Red dashed line—confidence interval using ±10 percent of truth value of the parameter 

3. Red diamond—parameter estimate 

4. Blue bars—estimated two sigma error bounds 

In the dynamic modeling process, no attempt was made at estimating the drag coefficients, because 

the aircraft excitations did not appreciably changed airspeed, resulting in a low signal-to-noise ratio for 

the axial force coefficient parameter estimation. Additionally, contributions from the lift damping 

coefficient CLq were small and omitted from the model and the estimation process. In total, five 

coefficients were estimated, CLD, CLGe, CmD, Cmq, and CmGe. 

D.3.1.5.1 Test Condition 1 

As shown in Figure D.8, the baseline model estimates for CZD resulted in improved estimates with 

increasing atmospheric turbulence. This is not representative of actual flight data and is attributed to 

SHUIHFWĮmeasurement given the additional excitation caused by atmospheric turbulence. 

As shown in Figure D.9, the baseline model estimates for CmGe resulted in greater scatter as 

turbulence intensity increased. This is caused by noisy explanatory variables (Ref. 15) from the 

atmospheric turbulence implementation in Equation (D.9). Similar patterns were observed with the 

remaining coefficients CZq, CZGe, CmD, and Cmq, although the results are not shown. These simulations 

demonstrate that all if the AOA measurements are perfect, all model parameters can be estimated 

accurately for any level of turbulence. 

Figure D.8.—Test Condition 1—Effect on CZD

CZD CZD

CZD CZD
NASA/CR—2014-218320 69 

Figure D.9.—Test Condition 1—Effect on CmGe 

D.3.1.5.2 Test Condition 2 

Estimates for CZD using the second-order vane sensor model of Equation (D.6) (Zn = 9 Hz and 

] = 0.35) caused degradations and biasing in the estimates with increasing turbulence intensities, as 

shown in Figure D.10. This is representative of actual flight data. 

As shown in Figure D.11, the baseline model estimates for CmGe resulted in greater scatter as 

atmospheric turbulence intensity increased. This is caused by noisy explanatory variables (Ref. 15) from 

the atmospheric turbulence implementation in Equation (D.9), and the vane sensor model. Similar 

patterns were observed with the remaining coefficients CZq, CZGe, CmD, and Cmq, although the results are 

not shown. 

CmGe CmGe

CmGe CmGe
NASA/CR—2014-218320 70 

Figure D.10.—Test Condition 2—Effect on CZD

Figure D.11.—Test Condition 2—Effect on CmGe 

CZD

CZD CZD

CZD CmGe CmGe

CmGe CmGe
NASA/CR—2014-218320 71 

D.3.1.5.3 Test Condition 2—Expanded Testing 

Although no quantitative analysis was done, qualitatively, small differences were observed between a 

perfect and typical vane frequency response in test conditions 1 and 2. This is expected because the vane 

dynamics do not substantially contribute to the scatter in the estimates, since all of the added dynamics 

are outside the range of frequencies analyzed. 

To further validate this claim, additional testing was carried out using the same simulation. A total of 

121 different combinations of flow angle vane natural frequency and damping ratios were tested. The 

natural frequency values tested were 5 through 15 in 1 Hz increments. The damping ratio values tested 

were 0.01, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, and 0.6. Since the AIMMS-20 probe approaches 

a gain of one and zero phase shift, it is characterized by the results of test condition 1. The AIMMS-20 

probe offers superior frequency response performance to mechanical vanes by using 1) a 5 hole pressure 

probe , 2) short tube lengths (air data tubes are typically less than 5 in. long), and 3) the resonance 

frequency of the silicon pressure diaphragm is in the kilohertz region. The following figures depict the 

results of 121 combinations, times 4 turbulence levels, times 200 runs per case. This totaled 96,800 runs. 

Figure D.12 depicts the results of 96,800 runs for CZD. As an example, the bottom left corner block in 

severe turbulence is light blue. The color represents the number of exceedances of CZD from ±10 percent 

of truth. The remaining 120 blocks are dark blue representing less than 10 exceedances. Figure D.12 

suggests that any combination of natural frequency and damping ratio tested provides adequate 

performance. Similar results were obtained for CmD and Cmq results are not shown. 

Figure D.12.—Number of exceedances for CZD
NASA/CR—2014-218320 72 

Figure D.13 shows that independent of the natural frequency and damping ratio tested, accurate 

estimates of CmGe

 are feasible in atmospheric turbulence. This is expected due to strong elevator control 

authority. 

Results for CZq, not shown, demonstrate that independent of the natural frequency and damping ratio 

tested, exceedances less than 20 out of 200 or ±10 percent of truth are not feasible in any level of 

atmospheric turbulence except calm. For the remaining stability and control estimates, the results show 

that most combinations of natural frequency and damping ratio tested provide adequate performance in 

calm, light, moderate, and severe atmospheric turbulence. 

D.3.1.5.4 Test Condition 3 

As shown in Figure D.14, estimates for CZD resulted in degraded estimates with increasing 

atmospheric turbulence for varying AOA magnitude and time delay as a function of frequency 

implemented using the data in Figure D.6 and Figure D.7. The degradation is seen as biasing in the 

estimates and increased scatter and uncertainty estimate error bounds. This is representative of actual 

flight data. 

As shown in Figure D.15, the estimates for CZGe also resulted in degraded estimates with increasing 

atmospheric turbulence. The degradation is seen as biasing in the estimates and increased scatter and 

uncertainty estimates. The increase in biasing and uncertainty compared to CZD estimates is caused by the 

relatively small contribution of the elevator term to the normal force coefficient. 

Figure D.13.—Number of exceedances for CmGe
NASA/CR—2014-218320 73 

Figure D.14.—Test condition 3—Effect on CZD

Figure D.15.—Test condition 3—Effect on CZGe

CZD

CZD CZD

CZD CZGe

CZGe CZGe

CZGe
NASA/CR—2014-218320 74 

Figure D.16 shows that the estimates for CmD degraded with increasing atmospheric turbulence. The 

degradation is seen as biasing in the estimates and increased scatter and uncertainty estimates. This is 

representative of actual flight data. 

As shown in Figure D.17, the estimates for Cmq degraded with increasing atmospheric turbulence. The 

degradation is seen as biasing in the estimates and increased scatter and uncertainty estimates. The 

estimates for Cmq resulted in the largest biasing of all the aerodynamic estimates considered because 

angular velocity model parameters are most susceptible to time delays (Ref. 16). 

Figure D.18 shows that the estimates for CmGe degraded with increasing atmospheric turbulence. The 

degradation is seen as a scatter in the estimates and increased scatter and uncertainty estimates. This is 

representative of actual flight data and correlates with the flight results presented in Figure D.3. 

Corrupting the AOA measurement by a frequency-dependent magnitude and time delay affected all of 

the estimated stability and control coefficients. The estimates of CmD and CmGe exhibited moderate biasing 

and increased scatter and uncertainty, CZD and Cmq estimates had large biases and increased scatter and 

uncertainty, and CZGe estimates showed the largest biases and increases in scatter and uncertainty. 

Previous work on the effects of time-shifted data on flight determined stability and control estimates

(Ref. 16) using the output-error method in the time domain demonstrated similar trends in the biasing and 

increased uncertainty, compared to the results shown here due to atmospheric turbulence. 

Figure D.16.—Test condition 3—Effect on CmD

CmD

CmD CmD

CmD
NASA/CR—2014-218320 75 

Figure D.17.—Test condition 3—Effect on Cmq

Figure D.18.—Test condition 3—Effect on CmGe

Cmq

Cmq Cmq

Cmq CmGe

CmGe CmGe

CmGe
NASA/CR—2014-218320 76 

D.3.1.5.5 Test Condition 4 

Based on the findings in Sections D.3.1.2.2, additional turbulence coefficients were added to the 

model structure. All of the turbulence estimates for CLD except for the estimates in calm atmosphere 

resulted in large degradations. In fact, the degradations were so large that they were not captured by the 

standard plot range. A similar pattern was observed with the estimates of CmD. This is not representative 

of actual flight data. The estimates for CmGe resulted in greater scatter and an overall bias as turbulence 

intensity increased. The overall biasing effect is not representative of actual flight data. Similar patterns 

were observed with the remaining coefficients results are not shown. 

D.3.1.6 Task 1 Conclusions 

Several practical issues related to real-time parameter estimation for a linear longitudinal dynamics 

model in atmospheric turbulence using indirect turbulence measurements were examined and discussed. 

The frequency response of the airflow vanes, wing response to atmospheric turbulence, and the structural 

response of the air data boom were identified as sources of colored noise. The frequency response of the 

airflow vanes for a wide variety of combinations of natural frequency and damping ratio did not adversely 

impact parameter estimation results in turbulence, except for CZq and CZGe. This was attributed to noisy 

explanatory variables in the atmospheric turbulence implementation and relatively low aerodynamic 

contributions as compared to CZD. Airflow vane dynamics do not substantially contribute to the scatter in 

the estimates, because all of the added dynamics are outside the range of frequencies analyzed. The 

structural modes of the boom lead to similar conclusions, because the natural frequencies of the boom 

structural modes are outside the range of frequencies used for dynamic modeling. A major source of 

colored noise was identified as the frequency-dependent upwash and time delay induced by the wingbound vortex system and the longitudinal separation between the AOA measurement and the wing. It was 

also surmised that the frequency-dependent sidewash was a major source of colored noise. The 

frequency-dependent upwash and time delay appear to be significant contributors to biasing and increased 

scatter and uncertainty for parameter estimates in atmospheric turbulence. 

Practical issues were examined using data from a Twin Otter DHC-6 longitudinal linear simulation, 

with realistic noise sequences added to the computed aircraft responses. This allowed a clear view of the 

effect of each source of colored noise to the modeling problem, because the true values of the model 

parameters were known. This approach was used initially to show that real-time parameter estimation can 

be done accurately in all atmospheric turbulence conditions if the AOA measurement is accurate. Flight 

test data from the GRC DHC-6 Twin Otter aircraft was used to validate the effect of the identified colored 

noise sources. 

Based on these findings, several practical recommendations for flight testing in atmospheric 

turbulence are suggested, as well as areas for future study. First, mount the air data boom at the nose of 

the aircraft to minimize the effect of the wing-bound and wing-tip vortices. Second, if the air data boom is 

mounted on the nose of the aircraft, an upwash and time delay calibration as a function of frequency is 

required. 

The following Task 2 and Task 3 results were carried out assuming that these precautions and 

calibrations had been taken care of. This assumed that the frequency response was adequate (good 

assumption based on the results of this section for a well calibrated, stiff, nose mounted sensor). This left 

the sensor’s static performance to be evaluated in Task 2 and Task 3. This eliminated the need to update 

the D-Six Twin Otter simulation from previous studies. In other words, if these precautions are taken, the 

RT-PID estimates will behave similarly inside and outside of atmospheric turbulence. 

D.3.2 Task 2 

Through initial testing it was found that the Twin Otter at a typical cruise condition was not capable 

of controlled flight in severe turbulence with the virtual pilot used for this study. During runs in severe 

turbulence, the aircraft would consistently enter a wing stall. The data from stalled runs was not 
NASA/CR—2014-218320 77 

considered useful, because stalled flight does not produce reliable data. As a result, all severe turbulence 

test conditions have been omitted from Task 2 analysis. Additionally, non-ice test conditions were run, 

but in the context of RT-PID, were considered outside of the scope and omitted from the analysis. Both 

ice and no ice runs were considered when evaluating D-ICES performance. 

The purpose of Task 2 was to modify, test, and validate the use of RT-PID and D-ICES algorithms 

with atmospheric turbulence as an input for various levels of atmospheric turbulence, and to identify an 

acceptable error in the wind measurement during turbulence for RT-PID and D-ICES algorithms. Wind 

measurement error was varied from 0.0 to 1.0 kt. Eighty test runs were conducted with and without ice, 

turbulence, and wind measurement errors. 

D.3.2.1 D-ICES Study Results 

It was found that the application of turbulence and sensor noise on D-ICES had a minimal impact on 

performance without filtering. With the filtering scheme of state variables described in Section D.3.2.1.1, 

D-ICES performance allowed accurate detection of possible icing situations in the presence of turbulence 

and sensor noise. D-ICES used a Theil inequality coefficient, comparing the a-priori elevator deflection 

for a clean aircraft to that of the current flight condition, to statistically estimate the aircraft’s current 

state. The estimated clean elevator deflections were calculated using a flight model inversion technique to 

find a corresponding control deflection for the given flight condition. The Theil inequality coefficient 

provided a measure of how well the actual elevator deflection matched the calculated elevator deflection, 

where a value of 0 indicated a total match, and 1 indicated a total mismatch. If the coefficient exceeds a 

threshold of 0.18 with a 3 sec delay to latch true, D-ICES activates the IVHM ID mode. 

As a result, a metric for success with sensor noise and turbulence is determined by their effect on 

creating false positive or false negative mode switches. A false positive condition would be one where DICES latches the ICEPro ID mode in a non-iced flight condition. While the impact of an occasional false 

positive is a human factors exercise, the impact of a false negative where the pilot is told the aircraft is 

clean while accreting ice would have serious implications. 

As shown in Figure D.19 D-ICES allows one false positive indication without filtering. Adding 

sensor noise to selected parameters used in the ICEPro logic, as discussed in Section D.2 causes the Theil 

inequality coefficient to increase, growing larger as the severity of the sensor noise increases. When the 

largest sensor noise was used the Theil inequality coefficient exceeded the ID mode threshold, which 

would cause an IVHM mode switch to ID mode in normal operation. 

Looking at the iced aircraft results without filtering in Figure D.20, D-ICES fails to indicate iced 

conditions in severe turbulence. This system failure is due to the severity of the turbulence as the aircraft 

stalls and full elevator deflection has been reached by the virtual pilot. The virtual pilot, described in 

Equation (D.1), was unable to maintain airspeed or a level attitude. This is seen in Figure D.21. Due to the 

lack of aircraft control and severity of the turbulence in the severe turbulence scenarios these conditions 

will not be evaluated. As discussed in Section D.2.2, low virtual pilot gains were used for all conditions 

which worked well without turbulence, but were less than optimal in turbulence. The low virtual pilot 

gains most likely contributed to the aircraft stalling in severe turbulence. For Phase I tests, a fixed throttle 

setting was used. However, during Phase II testing, aircraft configuration changes required the throttles to 

be actively controlled. The deactivation of the throttles during Phase I was done to eliminate the excessive 

and frequent throttle movements during flight in turbulence where the control logic was attempting to 

hold velocity in dynamic conditions. 
NASA/CR—2014-218320 78 

Figure D.19.—Unfiltered D-ICES Performance with Clean Aircraft 

Figure D.20.—Unfiltered D-ICES Performance with Iced Aircraft 

0 2 4 6 8 10 12 14 16 18 20

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

D-ICES

Unfiltered State Vector

Clean Aircraft

Time, sec

Theil Inequality Coefficient

Ge (Flight vs Inversion)

No Turbulence

Light Turbulence

Med Turbulence

Sev Turbulence

ID Mode

Threshold

0 2 4 6 8 10 12 14 16 18 20

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

D-ICES

Unfiltered State Vector

Iced Aircraft

Time, sec

Theil Inequality Coefficient

Ge (Flight vs Inversion)

No Turbulence

Light Turbulence

Med Turbulence

Sev Turbulence

ID Mode

Threshold
NASA/CR—2014-218320 79 

Figure D.21.—Severe Turbulence Flight Conditions 

D.3.2.1.1 Results of 80 Runs With Fixed Random Seed 

The flight model inversion process in D-ICES is dependent on an input state vector derived from the 

current flight conditions. This state vector includes the following eight elements: 

x Angle of attack 

x Angle of sideslip 

x Roll rate 

x Pitch rate 

x Yaw rate 

x Elevator deflection 

x Aileron deflections 

x Rudder deflection 

Since D-ICES was only used in initial mode switching, it was proposed that filtering these inputs for 

the flight model inversion calculation could be done with a minimal delay on system mode switching. A 

second order low pass filter was chosen with a cut-off frequency of 2.5 Hz. 

The four turbulence scenarios were tested, no turbulence, light turbulence, medium turbulence and 

severe turbulence. As previously explained, the severe turbulence results were omitted from the analysis. 

In each turbulence scenario, sensor noise was added, set to either a nominal condition or off. Finally, a 

fixed random seed, or a reusable series of random numbers, was used to generate turbulence to allow 

direct comparison between the runs. 

Figure D.22 shows the filtered conditions for the no-ice or clean aircraft with a fixed random noise 

seed. Here the noise influence shown in Figure D.19 resulting in a false positive has been addressed. In 

addition the effect of atmospheric turbulence on the D-ICES algorithm is minimal. 

0 5 10 15 20

-30

-20

-10

0

10

Elevator Deflection

Ge, deg

Severe Turbulence Flight Condtions

0 5 10 15 20

-10

0

10

20

30

Angle of Attack

D, deg

0 5 10 15 20

-20

0

20

40

60

Time, sec

Downward

Wind Component

ft/sec
NASA/CR—2014-218320 80 

In the iced aircraft scenario, shown in Figure D.23, there were no false negatives with D-ICES. The 

sensor noise effects have been addressed as the scenarios with and without noise are almost over-plots. There 

is a banding effect due to the level of turbulence (i.e., all of the results for a common level of turbulence 

grouped tightly together but at different Theil inequality coefficient values); however, all conditions are well 

above the selected ID Mode switch threshold and result in mode switching at similar times. 

Figure D.22.—Filtered Clean Aircraft with Fixed Random Seed 

Figure D.23.—Filtered Iced Aircraft with Fixed Random Seed 

0 2 4 6 8 10 12 14 16 18 20

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

D-ICES

Filtered State Vector

Clean Aircraft

Time, sec

Theil Inequality Coefficient

Ge (Flight vs Inversion)

No Turbulence

Light Turbulence

Med Turbulence

ID Mode

Threshold

0 2 4 6 8 10 12 14 16 18 20

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

D-ICES

Filtered State Vector

Iced Aircraft

Time, sec

Theil Inequality Coefficient

Ge (Flight vs Inversion)

No Turbulence

Light Turbulence

Med Turbulence

ID Mode

Threshold
NASA/CR—2014-218320 81 

D.3.2.1.2 Results of 60 Runs Random Seed 

A total of 60 scenarios were tested (the 20 severe turbulence cases were dropped, as mentioned 

earlier), again at three turbulence levels with one condition without sensor noise and nine additional 

scenarios with sensor noise. In addition, the seed for the random number generation was selected 

randomly creating a unique set of random number for every run. 

In Figure D.24, the clean aircraft conditions continue to exhibit a very tight banding with only three 

turbulence conditions potentially exhibiting a false positive. Further examination revealed that in the 

immediate trim AOA region, small changes in AOA produced very little variation in pitching moment 

due to a specific elevator deflection. If the AOA went to higher values where there was a variation with 

AOA, the Theil inequality coefficient remained well below the threshold value. It was surmised that the 

D-ICES inversion process had some troubles in determining the correct elevator deflection near trim, in 

the presence of light turbulence with these pitch due to elevator characteristics. A real pilot would 

probably not chase the variations in speed and flight path in the same manner as the virtual pilot did in 

these runs, and the results could differ. Looking at only the initial 6 sec after buffer fill (t = 6 to t = 12) 

provides a snapshot of the typical D-ICES operation. 

Figure D.25 shows 60 separate runs in D-ICES performance under three turbulence conditions. Under 

light and medium turbulence D-ICES successfully detected the presence of icing conditions without 

degradation or delaying IVHM mode shifting. 

Figure D.24.—Filtered Clean Aircraft with Random Seed 

0 2 4 6 8 10 12 14 16 18 20

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

D-ICES

Filtered State Vector

Clean Aircraft

Time, sec

Theil Inequality Coefficient

Ge (Flight vs Inversion)

No Turbulence

Light Turbulence

Med Turbulence

ID Mode

Threshold
NASA/CR—2014-218320 82 

Figure D.25.—Filtered Iced Aircraft with Random Seed 

D.3.2.2 RT-PID Study Results 

It was found that the application of turbulence and sensor noise on RT-PID had a significant impact 

on performance. As previously mentioned, non-ice test conditions were run, but in the context of RT-PID 

were considered outside of the scope and omitted from the analysis. Therefore, only the 40 iced runs will 

be reported. Results from the 40 iced runs with fixed turbulence seed are depicted in Figure D.26. The 

error in flow angles was varied from 0.0° to 1.0° in 0.1° increments. All other errors were fixed using the 

probe specifications in Table D.1. 

The results of Figure D.27 illustrate the effect of flow angle error on the coefficient CmD. As 

expected, increased flow angle error results in increasing error in the estimate and increased error bars 

throughout each turbulence level. The severe turbulence cases are discarded from the analysis since the 

majority of the flight resulted in a stalled condition. 

Next, the MOS message criterion was applied with white Gaussian measurement noise. The following 

results for CmGe, representative of the pitch degrade message, are depicted in Figure D.27 with fixed 

turbulence seed. As the turbulence level increases so does the error in the estimate and the error bounds. 

 

0 2 4 6 8 10 12 14 16 18 20

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

D-ICES

Filtered State Vector

Iced Aircraft

Time, sec

Theil Inequality Coefficient

Ge (Flight vs Inversion)

No Turbulence

Light Turbulence

Med Turbulence

ID Mode

Threshold
NASA/CR—2014-218320 83 

Iced airplane CmD

Figure D.26.—Effect of flow angle error on CmD

 
NASA/CR—2014-218320 84 

Iced airplane CmGe

Figure D.27.—Effect of flow angle error on CmGe

D.3.2.3 Task 2 Measures of Success (MOS) 

The MOS message criterion is summarized below for convenience (Appendix E). 

1) Message accuracy 

Message error will be determined by the error bounds and accuracy of its associated stability and 

control coefficient for all ICEPro messages. Coefficients must meet or exceed the nonatmospheric turbulence error bound and accuracy in atmospheric turbulence. 

In general, the MOS for message accuracy is not achieved. As seen in Figure D.27, the estimates 

move away from the truth and error bounds increase with each increase in turbulence level. The same 
NASA/CR—2014-218320 85 

runs that were used to disqualify the MOS message criterion were used to evaluate the ISP MOS criterion 

summarized below for convenience (Appendix E). 

2) ISP accuracy 

ISP error will be determined by comparing the true ISP value with the estimated ISP value. ISP is 

a function of five stability and control coefficients: Longitudinal static stability, elevator control 

power, dihedral effect, aileron control power, and normal lift coefficient. The ISP error must meet 

or exceed its non-atmospheric turbulence error bound and accuracy in atmospheric turbulence. 

The following results for ISP, representative of all 40 ice runs with fixed turbulence seed, are 

depicted in Figure D.28.

Figure D.28 depicts the ISP time history for each simulated flow angle error. The ISP value is initially 

set to zero at the start of each run and ramps up as ice is detected. The D-ICES threshold for determining 

an ice condition is an ISP value of 0.5. This is depicted by dashed red lines in each subplot. Visual 

inspection confirms that the ISP is different in and out of turbulence. In order to satisfy the original MOS 

criteria, the ISP lines in and out of turbulence must be the same. As shown in Figure D.28 this is not the 

case. Additionally the ISP deteriorated greatly with increased flow angle error. In general, the MOS 

criterions for both messages and ISP, as originally defined, were not achieved. A new MOS criterion was 

formulated based on the results depicted in Figure D.27 and Figure D.28. The new criterion takes into 

account the degradation of RT-PID estimates and error bounds in turbulence. Errors in the estimates and 

in the error bounds with turbulence were tolerated as long as the ICEPro messages and AOA brackets are 

equivalently displayed inside and outside of turbulence. The criterion is summarized below. 

Figure D.28.—ISP for all turbulence levels and flow angle errors (D and E) 
NASA/CR—2014-218320 86 

1. Compare high and low AOA brackets across turbulence levels. Angle of attack brackets are 

strongly dependent on AOA and flap deflection and weekly dependent on ISP. 

ż Equal or conservative bracket location 

– Successful envelope protection 

ż Unconservative bracket location 

– Unsuccessful envelope protection 

2. Compare message state across turbulence levels as well as time to latch and time to delatch. 

ż Same state annunciation 

– Successful envelope protection 

ż Varying state annunciation 

– Unsuccessful envelope protection 

ż Latch / delatch within 10 sec of no turbulence results 

– Successful envelope protection 

The following results for message accuracy, representative of AIMMS-20 measurement errors for all 

turbulence levels, are depicted in Figure D.29 using the new MOS criterion. Only the results of the Roll 

degrade message are plotted since the remaining messages did not result in false positives or false 

negatives and were considered successful. In theory, the same messages are expected for all turbulence 

levels since the aircraft is flown about the same reference condition. In Figure D.29, this is represented by 

the average AOA and angle of sideslip during each run. In practice, for zero, light, and moderate 

turbulence the average flow angles are similar. In severe turbulence, the average flow angles are 

significantly different due to wing stall, but do not affect the message accuracy. As stated previously the 

severe turbulence runs are omitted from the analyses and all runs used a fixed turbulence seed for 

comparison purposes. 

In the zero turbulence case, the roll degrade message results in a false positive warning message. This 

is explained by the aileron control power estimate subplot. The estimate of the aileron control power 

quickly diverges from the truth. This difference causes the false positive indication. It should be noted 

that although a false positive condition occurs due to poor estimates the ICEPro logics latches at the 

correct levels. As a side note, this condition was repeated with higher buzzing amplitudes and no false 

positives occurred. In each of the turbulence cases, the additional aileron inputs provided by the virtual 

pilot improved the RT-PID estimates so that no false positives were annunciated. In general the new MOS 

message criterion is considered to be successful. 

Additional testing was conducted to access the new MOS AOA bracket criterion. The following 

results for AOA bracket accuracy, representative of AIMMS-20 measurement errors with conservative 

flow angle errors for all turbulence levels, are depicted in Figure D.30. A conservative flow angle error of 

0.3° versus the stated 0.1° was chosen because experience has proven the difficulty of calibrating up-wash 

and side-wash corrections on air data probes. 

As in the message MOS criterion, the same AOA bracket locations are expected for all turbulence 

levels since the aircraft is flown about the same reference condition. In Figure D.30, these are represented 

by the upper and lower bar delta plots using zero turbulence as the reference. For light and moderate 

turbulence the average deltas are similar. In severe turbulence, the average delta positions are 

significantly different due to wing stall and hence significantly affect the angle of attach bracket location 

and accuracy. The severe turbulence results are omitted from the analysis. 

As shown in Figure D.30, after 25 sec from start-up the ISP oscillates between 0.8 and 1.0. Similarly, 

after 25 sec the AOA brackets oscillate about the zero turbulence AOA brackets. Additional filtering was 

implemented to smooth the AOA brackets in turbulence. The filtered results are shown in Figure D.31. 

The results presented in Figure D.31 are filtered by a low pass fourth order Butterworth filter with a 

cut off frequency of 0.1 radians per second. Similar plots were generated to determine the effect of flow 
NASA/CR—2014-218320 87 

angle error versus ISP. It was determined that the AOA brackets are strongly dependant on AOA and 

weekly dependant on ISP. Even with perfect instruments the ISP was very similar to the ISP in Figure 

D.31 with 0.3° flow angle error. The trend in Figure D.31 is that the AOA bracket location in turbulence 

oscillates around the zero turbulence case. This agrees with the analysis in Figure D.29 were the average 

AOA and sideslip was approximately the same for all runs except severe turbulence. Closer inspection 

shows that the AOA brackets are conservative and unconservative throughout the runs. Looking at the 

period after 30 sec, where the ISP has stabilized after start-up, the oscillations are approximately 1° for 

light turbulence and 2° for moderate turbulence. These small changes are not considered to be discernible 

by the pilot. Additionally, since the oscillations are due to the variation in AOA throughout the run and 

not due to the ISP, the new MOS AOA bracket criterion is considered to be successful for this task. 

Figure D.29.—Roll degrade indication analyses 

 

Cl

Ga
NASA/CR—2014-218320 88 

AOA limit indication (0.3° flow vane error) 

Figure D.30.—Angle of attack bracket indication analyses 

 
NASA/CR—2014-218320 89 

AOA limit indication (0.3° flow vane error) 

Figure D.31.—Filtered AOA bracket indication analyses 
NASA/CR—2014-218320 90 

D.3.2.4 Task 2 Conclusions 

RT-PID 

x The original RT-PID MOS (Appendix E) for message accuracy were not achieved. A new MOS 

criterion was formulated due to results depicted in Figure D.27 and Figure D.28. 

x A new MOS was defined were errors in the estimates and in the error bounds with turbulence are 

tolerated as long as the ICEPro messages and AOA brackets are equivalently displayed inside and 

outside of turbulence. 

x The average flow angles are similar for zero, light, and moderate turbulence. In severe turbulence, 

the average flow angles are significantly different due to wing stall, but do not affect the message 

accuracy. 

x With the exception of the Roll degrade message, there were no false positives or false negatives. 

The roll degrade message was caused during a period of low control activity. In each of the 

turbulence cases, the additional aileron inputs provided by the virtual pilot improved the RT-PID 

estimates so that no false positives were annunciated. 

x In general the new MOS message criterion is considered to be successful. 

D-ICES 

x It was found that the application of turbulence and sensor noise on D-ICES had minimal impact 

on performance without filtering. 

ż Only one condition was found that yielded a false positive that would have caused the system 

to enter ID Mode when no icing was present. 

ż There was one condition where the elevator deflection Theil inequality coefficient for an iced 

airplane failed to exceed the threshold and two where the value originally exceeded the 

threshold but with time decreased until it was below the threshold. 

x When the state vectors going to the D-ICES inversion routine were filtered to remove higher 

frequency disturbances due to turbulence and noise, D-ICES accurately detected all icing 

conditions in the presence of turbulence and sensor noise and thus operated correctly. 

x Three potential false positive cases were found that showed the elevator Theil Coefficient built up 

over time to exceed the threshold when no ice was present. An examination of the data in this 

region led to supposition that the D-ICES inversion process had some troubles in determining the 

correct elevator deflection with the pitch due to elevator characteristics at those flight conditions. 

Overall it was found that the objective to identify an acceptable error in the wind measurement was 

not required and that the performance of the AIMMS-20 probe was satisfactory for Task 2. 

D.3.3 Task 3 

The purpose of Task 3 was to evaluate four representative phases of flight (Appendix E) with the 

performance results of Task 2. Twenty four test runs were conducted with and without ice, turbulence, 

and wind measurement errors over four phases of flights. Turbulence was tested as calm, light, moderate, 

and severe. Turbulence levels are defined per MIL-F-8785C. All measurement errors were biased in a 

random time history with limits defined by the condition matrix and sensor specifications using the probe 

specifications in Table D.1 and the same conservative flow angle errors. 

Whereas Task 2 focused on straight and level flight in the cruise configuration, Task 3 involved 

additional flight phases including, for a reference, a repeat of flaps up straight and level flight, straight 

and level flight flap transition, straight and level flight heading capture, and –3° flight path angle with a 

flap transition. This last phase is representative of the instrument approach task studied with the ICEFTD 

and thirty subject pilots at Embry-Riddle Aeronautical University during the summer of 2009. 

Through initial testing it was found that the Twin Otter in approach configuration (20° flap) was not 

capable of controlled flight in moderate and severe turbulence. During some runs in moderate and all runs 

in severe turbulence, the aircraft entered a wing stall with the virtual pilot. The data from stalled runs was 
NASA/CR—2014-218320 91 

not considered useful. As a result, all moderate and severe turbulence test conditions have been omitted 

from the RT-PID Task 3 analysis. The non-ice test conditions were run, but in the context of RT-PID, 

were considered outside of the scope and omitted from analysis. Additionally, the severe turbulence test 

conditions have been omitted from the D-ICES Task 3 analyses. 

D.3.3.1 D-ICES Study Results 

D.3.3.1.1 Results of 24 Runs Fixed Seed 

The first phase matched the Task 2 conditions. The aircraft was level at 2500 ft, 110 kt with the flaps 

retracted. As expected, results in Figure D.32 match Figure D.22 and Figure D.23 in Task 2. 

In the second phase of flight, the aircraft was slowed from an initial velocity of 110 to 80 kt and the 

flaps were then extended from 0° to 20°. D-ICES experienced no false positives during this scenario as 

seen in Figure D.33. The clean aircraft tests experienced little change due to the dynamic flight 

conditions. 

The next phase of flight examined a heading change from 090° to 045°. Here a false negative was 

seen for the iced airplane with no turbulence, as shown in Figure D.34. The value of the Theil inequality 

coefficient for elevator deflection is a measure of how well the actual elevator time history matches the 

predicted time history for an uniced airplane. For an uniced airplane, the elevator deflection time history 

should be virtually identical to the D-ICES predicted time history. This will produce very low values of 

the deU Theil inequality coefficient. A sample of the elevator and corresponding Theil inequality 

coefficients for the uniced airplane with light turbulence is shown in Figure D.35. In the lower plot, 

INV_PitchCmd is the predicted elevator deflection, while Ge is the actual deflection. 

Task 2 

Level flight, 2500 ft, 110 kt, Gf : 0°

Figure D.32.—D-ICES Level Flight Scenario 

0 2 4 6 8 10 12 14 16 18 20

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Time, sec

Theil Inequality Coefficient

Ge (Flight vs Inversion)

Clean/No Turb

Clean/Light Turb

Clean/Med Turb

Ice/No turb

Ice/Light Turb

Ice/Med Turb

ID Mode

Threshold
NASA/CR—2014-218320 92 

Task 2 

Level flight, 110 kt o 80 kt, Gf : 0° o 20°

Figure D.33.—D-ICES Flap Transition Scenario 

Task 2 

Level flight, left turn 090° o 045° 80 kt, Gf : 20°

Figure D.34.—D-ICES Heading Change Scenario 

For an iced airplane, there should be some differences between the expected (uniced) elevator 

deflection and the actual deflection. The closer the iced and uniced CmGe values are, the less this will be. 

In Figure D.36, one can see that the actual and predicted elevator deflection were starting to separate 

0 5 10 15 20 25 30 35

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Time, sec

Theil Inequality Coefficient

Ge (Flight vs Inversion)

Flap

Transition

Clean/No Turb

Clean/Light Turb

Clean/Med Turb

Ice/No turb

Ice/Light Turb

Ice/Med Turb

ID Mode

Threshold

0 5 10 15 20 25 30

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Time, sec

Theil Inequality Coefficient

Ge (Flight vs Inversion)

Heading

Change

Clean/No Turb

Clean/Light Turb

Clean/Med Turb

Ice/No turb

Ice/Light Turb

Ice/Med Turb

ID Mode

Threshold
NASA/CR—2014-218320 93 

resulting in the larger Theil inequality coefficient values. However, because the iced CmGe was not very 

different from the uniced values at this flight condition, the differences without turbulence were not 

enough to exceed the threshold. With the addition of turbulence, the elevator deflection separation 

became greater and resulted in Theil inequality coefficient values that exceeded the threshold. This is 

probably because the turbulence oscillations caused changes in alphas where there was greater difference 

between the iced and uniced CmGe’s. 

In the fourth phase of flight, the aircraft was flown down a 3° glide slope from 3000 ft at 75 kt. As the 

descent was established, the airspeed was reduced from 75 to 70 kt and flaps were then extended to 30°. 

While there were no false indications, Figure D.37 shows that the D-ICES results remained close to the 

singular condition seen in the level turning flight case condition. However, the Theil inequality coefficient 

values are still sufficient to exceed the threshold for transitioning to ID mode for all of the iced 

conditions. 

Figure D.35.—Uniced Elevator Deflection Theil inequality Coefficient Analysis 

Figure D.36.—Iced Elevator Deflection Theil inequality Coefficient Analysis 
NASA/CR—2014-218320 94 

Task 2 

3° descent from 3000 ft, 75 kt o 70 kt, Gf : 20° o 30°

Figure D.37.—D-ICES Descent with Flap Change Scenario 

Figure D.38.—D-ICES Task 2 with Random Seed 

D.3.3.1.2 Results of 24 Runs Random Seed 

Task 3 was rerun using a random seed. Figure D.38, summarizes the results with all four scenarios 

plotted. Due to occasional low speed stall issues with medium turbulence with the random seed, these 

conditions were excluded. As expected, the condition that caused the iced aircraft with no turbulence to 

exhibit a false positive during the heading capture scenario still produced similar results here. Also, with 

0 5 10 15 20 25 30

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Time, sec

Theil Inequality Coefficient

Ge (Flight vs Inversion)

Airspeed

Change

Flap

Transition

Clean/No Turb

Clean/Light Turb

Clean/Med Turb

Ice/No turb

Ice/Light Turb

Ice/Med Turb

ID Mode

Threshold

0 5 10 15 20 25

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Clean/No Turb

Clean/Light Turb

Ice/No turb

Ice/Light Turb

ID Mode

Threshold
NASA/CR—2014-218320 95 

the random seed, the same scenario light turbulence condition was delayed by 10 sec before latching into 

ID Mode. 

D.3.3.2 RT-PID Study Results 

In general, the original MOS criterion for both messages and ISP are not achieved in Task 3 and 

results are similar to Figure D.27 and Figure D.28 in Task 2. As in Task 2 the new MOS criterion based 

on AOA brackets and messaging was applied to each flight phase. The results for this task are broken up 

by respective flight phase. 

D.3.3.2.1 Phase 1—Flaps Up Straight and Level Flight 

As expected from Task 2 results, Phase 1 runs had no pitch degrade, climb limit, or flap limit 

messages for zero, light, and moderate turbulence levels. 

As shown in Figure D.39, the light turbulence roll degrade message appears before the zero 

turbulence roll degrade message. Since this is within the 10 sec defined in the MOS this is considered 

successful. Like Figure D.29 in Task 2, Figure D.39 shows false positives for light and no turbulence 

cases. This condition was repeated with higher buzzing amplitudes and no false positives occurred. The 

MOS criterion for AOA bracket was applied to Phase 1 with similar results as shown in Task 2 Figure 

D.31 and considered to be successful. 

Figure D.39.—Phase 1 Roll degrade indication analyses 

Cl

Ga
NASA/CR—2014-218320 96 

D.3.3.2.2 Phase 2—Straight and Level Flight Flap Transition 

Phase 2 runs had no pitch degrade, yaw degrade, climb limit, or flap limit messages for zero, light, 

and moderate turbulence levels. As shown in Figure D.40 the roll degrade message performance in 

turbulence was better than outside of turbulence. In light and moderate turbulence the system did not 

annunciate any caution or warning messages. The increase in virtual pilot activity in turbulence during the 

flap transition provided improved RT-PID performance to that shown in Figure D.40. In Figure D.40 the 

bottom plot shows poor RT-PID performance in zero turbulence with respect to the truth values due to 

lack of information content. 

As in Task 2, the same AOA bracket locations are expected for all turbulence levels since the aircraft 

is flown about the same reference condition. In Figure D.41, these are represented by the upper and lower 

bar delta plots using calm atmosphere as a reference. After approximately 35 sec the flap transition from 

zero to 20° flap is complete. Incidentally it takes approximately the same amount of time for the ISP to 

stabilize from start-up as shown at the bottom of Figure D.41. It should be noted that the AOA brackets 

exhibit similar behavior to that found in Task 2 from 35 to 60 sec and therefore considered to be 

successful. There are no pitch degrade messages generated during this time because the current ICEPro 

logic for pitch degrade messaging is based on the elevator effectiveness, CmGe, which is well within the 

thresholds such that no messages are generated. Data after 60 sec should be discarded because the virtual 

pilot is unable to adapt near the tail stall boundary and enters into a pilot coupling event between repeated 

incipient tail stalls and wing stalls causing the large oscillations shown. Along these lines it should be 

noted that the virtual pilot gains were set as low as practical while obtain reasonable performance without 

excessive control activity. A human pilot who is familiar with the aircraft and its flying characteristics 

may be better suited to control these situations. It is recommended that additional piloted runs be 

conducted to verify this conclusion by the virtual pilot. 

Figure D.40.—Phase 2 Roll degrade indication analyses 

Cl

Ga
NASA/CR—2014-218320 97 

Figure D.41.—Phase 2 AOA bracket analyses 

D.3.3.2.3 Phase 3—Straight and Level Flight Heading Capture 

Phase 3 runs had no pitch degrade, climb limit, or flap limit messages for zero, light and moderate 

turbulence levels. As shown in Figure D.42 the roll degrade message performance in turbulence was 

better than out of turbulence. In light turbulence the system annunciates a caution message and in 

moderate turbulence the system did not annunciate any caution or warning messages. The increase in 

virtual pilot activity during the heading capture due to turbulence provided improved RT-PID 

performance. In Figure D.42 the bottom plot shows poor RT-PID performance in zero turbulence with 

respect to the truth values due to lack of information content. 

Figure D.43 shows a typical run for Phase 3, heading capture. Ice did not greatly affect the rudder 

effectiveness for this airplane, so it was anticipated that there would be few, if any yaw degrade messages 

as is shown to be the case in Figure D.43 no messages were generated for any of the heading capture 

conditions investigated. 
NASA/CR—2014-218320 98 

Figure D.42.—Phase 3 Roll degrade indication analyses 

Figure D.43.—Phase 3 Yaw degrade indication analyses 

CNG

ClGa
NASA/CR—2014-218320 99 

The MOS criterion for AOA bracket was applied to Phase 3 with similar results as shown in Task 2, 

Figure D.31 and Task 3, Figure D.41 and considered to be successful. Similar to Task 3 Figure D.41 the 

last 20 sec should be discarded because the virtual pilot is unable to adapt near the tail stall boundary and 

enters into a pilot coupling event between repeated incipient tail stalls and wing stalls causing large 

oscillations. 

D.3.3.2.4 Phase 4—Descending Flight Flap Transition—3° Glide Slope 

Phase 4 runs had no pitch degrade or yaw degrade messages for zero, light or moderate turbulence 

levels. As shown in Figure D.44 the roll degrade message performance in turbulence was better that out 

of turbulence. The increase in virtual pilot activity during the heading capture due to turbulence provided 

improved RT-PID performance. In Figure D.44 the bottom plot shows poor RT-PID performance in zero 

turbulence with respect to the truth values due to lack of information content. 

As shown in Figure D.44 the zero turbulence roll degrade caution and warning messages annunciate 

before the light turbulence messages. This is considered to be conservative and successful based on MOS 

criterion. The toggle in the warning message in moderate turbulence is discarded because as before the 

virtual pilot is unable to adapt near the tail stall boundary and enters into a pilot coupling event between 

repeated incipient tail stalls and wing stalls. Because the elevator effectiveness during this maneuver stays 

within the thresholds for caution and warning messages, no pitch degrade messages are generated. 

Figure D.44.—Phase 4 Roll degrade indication analyses 
NASA/CR—2014-218320 100 

As shown in Figure D.45 the zero turbulence climb limit warning message annunciated at the same 

time as in light turbulence. This is considered to be successful based on the MOS criterion. In moderate 

turbulence the warning message came on for approximately 7 sec. The ICEPro message logic requires 

messages to persist for 10 sec before they can unlatch, however, when a stall is detected by ICEPro, it 

declutters the PFD by removing all other messaging for 20 sec to allow the pilot to regain normal flight. 

After that time, ICEPro returns to normal messaging. In the situation shown in Figure D.45, a stall 

condition occurred which terminated the messaging. 

As shown in Figure D.46 the zero turbulence flap limit warning message annunciates at the same time 

as in light turbulence. This is considered to be successful based on the MOS criterion. As in the climb 

limit message, in the moderate turbulence case the warning message came on for approximately 7 sec. 

The ICEPro message logic requires messages to persist for 10 sec before they can unlatch. The stall 

condition resulted in decluttering the PFD and removed the message after 7 sec, as discussed above. 

The MOS criterion for AOA bracket was applied to Phase 4 and shown in Figure D.47. After 

approximately 30 sec the flap transition from 20 to 30° flap is complete. Incidentally it takes 

approximately the same amount of time for the ISP to stabilize from start-up as show at the bottom of 

Figure D.47. It is believed that the increased oscillations in Phase 4 AOA brackets over previous phases is 

because of the narrow envelope between wing stall and tail stall at flaps 30°. The inability to adapt by the 

virtual pilot in this reduced stability regime causes larger oscillations. Applying the MOS criterion would 

be unsuccessful in this phase since a pilot is able to discern the unconservative variation in the AOA 

brackets. It is recommended that additional piloted runs be conducted to verify this conclusion by the 

virtual pilot. 

Figure D.45.—Phase 4 Climb Limit indication analyses 
NASA/CR—2014-218320 101 

Figure D.46.—Phase 4 Flap Limit indication analyses 
NASA/CR—2014-218320 102 

Figure D.47.—Phase 4 AOA bracket analyses 

D.3.3.3 Task 3 Conclusions 

D-ICES 

x The first scenario was a repeat of Task 2 conditions and the results were similar. 

x For the level flight flap extension scenario, the aircraft was slowed from an initial velocity of 110 

to 80 kt and the flaps were then extended from 0° to 20°. D-ICES experienced no false positives 

during this scenario as seen in Figure D.33 

x The next scenario examined was a heading change from 090° to 045°. Here a false negative was 

seen for the iced airplane with no turbulence but correctly predicted positives for the low and mid 
NASA/CR—2014-218320 103 

turbulence levels. The false positive was believed to be due to a singular condition where there 

was little difference between the iced and clean airplane’s control effectiveness at the chosen 

flight condition. 

x For the 3° glide slope descent scenario there were no false indications, even though the results 

remained close to a singular condition seen in the level turning flight case. However, the Theil 

inequality coefficient values are still sufficient to exceed the threshold for transitioning to ID 

mode for all of the iced conditions. 

x In most cases, D-ICES performed as designed in the presence of turbulence and noise. However, 

the exceptions noted above, especially the false negatives, do indicate that some further 

development of the system is warranted. One possible area to examine would be utilizing other 

terms related to the Theil inequality coefficient, namely the bias, variance and co-variance terms 

in some way to develop a more robust detection scheme. 

RT-PID 

x Phase 1 runs had no pitch degrade, climb limit, or flap limit messages for zero, light, and 

moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 1 and 

considered to be successful. 

x Phase 2 runs had no pitch degrade, yaw degrade, climb limit, or flap limit messages for zero, 

light, and moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 

2 and therefore considered to be successful. 

x Phase 3 runs had no pitch degrade, climb limit, or flap limit messages for zero, light, and 

moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 3 and 

considered to be successful. 

x Phase 4 runs had no pitch degrade or yaw degrade messages for zero, light, and moderate 

turbulence levels. As shown in Figure D.46 the calm atmosphere flap limit warning message 

annunciates at the same time as in light turbulence. This is considered to be successful based on 

the MOS criterion. The MOS criterion for AOA bracket was applied to Phase 4: Applying the 

MOS criterion would be unsuccessful in this phase since a pilot is able to discern the 

unconservative variation in the AOA brackets. 

x It is recommended that additional piloted runs be conducted to verify this conclusion by the 

virtual pilot. 

In general, it was found that the performance of the AIMMS-20 probe was satisfactory for Task 3. 

D.4 Conclusions and Recommendations 

Summaries are provided of the results of each task. For Task 2 and Task 3 the summaries are divided 

into D-ICES and RT-PID. 

D.4.1 Summary of Results 

Task 1 

Several practical issues related to real-time parameter estimation for a linear longitudinal dynamics 

model in atmospheric turbulence using indirect turbulence measurements were examined and discussed. 

The frequency response of the airflow vanes, wing response to atmospheric turbulence, and the structural 

response of the air data boom were identified as sources of colored noise. The frequency response of the 

airflow vanes for a wide variety of combinations of natural frequency and damping ratio did not adversely 

impact parameter estimation results in turbulence, except for CZq and CZGe. This was attributed to noisy 

explanatory variables in the atmospheric turbulence implementation and relatively low aerodynamic 

contributions as compared to CZD. Airflow vane dynamics do not substantially contribute to the scatter in 

the estimates, because all of the added dynamics are outside the range of frequencies analyzed. The 
NASA/CR—2014-218320 104 

structural modes of the boom lead to similar conclusions, because the natural frequencies of the boom 

structural modes are outside the range of frequencies used for dynamic modeling. A major source of 

colored noise was identified as the frequency-dependent upwash and time delay induced by the wingbound vortex system and the longitudinal separation between the AOA measurement and the wing. It was 

also surmised that the frequency-dependent sidewash was a major source of colored noise. The 

frequency-dependent upwash and time delay appear to be significant contributors to biasing and increased 

scatter and uncertainty for parameter estimates in atmospheric turbulence. 

Practical issues were examined using data from a Twin Otter DHC-6 longitudinal linear simulation, 

with realistic noise sequences added to the computed aircraft responses. This allowed a clear view of the 

effect of each source of colored noise to the modeling problem, because the true values of the model 

parameters were known. This approach was used initially to show that real-time parameter estimation can 

be done accurately in all atmospheric turbulence conditions if the AOA measurement is accurate. Flight 

test data from the GRC DHC-6 Twin Otter aircraft was used to validate the effect of the identified colored 

noise sources. 

Based on these findings, several practical recommendations for flight testing in atmospheric 

turbulence are suggested, as well as areas for future study. First, mount the air data boom at the nose of 

the aircraft to minimize the effect of the wing-bound and wing-tip vortices. Second, if the air data boom is 

mounted on the nose of the aircraft, an upwash and time delay calibration as a function of frequency is 

required. 

Task 2 

D-ICES 

x It was found that the application of turbulence and sensor noise on D-ICES generally had a 

minimal impact on performance without filtering. 

ż Only one condition was found that yielded a false positive that would have caused the system 

to enter ID Mode when no icing was present. 

ż There was one condition where the elevator deflection Theil inequality coefficient for an iced 

airplane failed to exceed the threshold and two where the value originally exceeded the 

threshold but with time decreased until it was below the threshold. 

x When the state vectors going to the D-ICES inversion routine were filtered to remove higher 

frequency disturbances due to turbulence and noise, D-ICES accurately detected all icing 

conditions in the presence of turbulence and sensor noise and thus operated correctly. 

x Three potential false positive cases were found that showed the elevator Theil Coefficient built up 

over time to exceed the threshold when no ice was present. An examination of the data in this 

region led to supposition that the D-ICES inversion process had some troubles in determining the 

correct elevator deflection with the pitch due to elevator characteristics at those flight conditions. 

RT-PID 

x In general, the (Old) RT-PID MOS for message accuracy was not achieved. A new MOS criterion 

was formulated due to results depicted in Figure D.27 and Figure D.28. 

x Errors in the estimates and in the error bounds with turbulence are tolerated as long as the ICEPro 

messages and AOA brackets are equivalently displayed inside and outside of turbulence (New 

MOS). 

x In practice, for zero, light, and moderate turbulence the average flow angles are similar. In severe 

turbulence, the average flow angles are significantly different due to wing stall, but do not affect 

the message accuracy. 

x With the exception of the Roll degrade message, there were no false positives or false negatives. 

x In each of the turbulence cases, the additional aileron inputs provided by the virtual pilot 

improved the RT-PID estimates so that no false positives were annunciated. 
NASA/CR—2014-218320 105 

x In general the new MOS message criterion is considered to be successful. 

Overall it was found that the objective to identify an acceptable error in the wind measurement was 

not required and that the performance of the AIMMS-20 probe was satisfactory for Task 2. 

Task 3 

D-ICES 

x For the level flight flap extension scenario, the aircraft was slowed from an initial velocity of 110 

to 80 kt and the flaps were then extended from 0° to 20°. D-ICES experienced no false positives 

during this scenario as seen in Figure D.33. 

x The next scenario examined was a heading change from 090° to 045°. Here a false negative was 

seen for the iced airplane with no turbulence but correctly predicted positives for the low and mid 

turbulence levels. The false positive was believed to be due to a singular condition where there 

was little difference between the iced and clean airplane’s control effectiveness at the chosen 

flight condition. 

x For the 3° glide slope descent scenario there were no false indications, even though the results 

remained close to a singular condition seen in the level turning flight case. However, the Theil 

inequality coefficient values are still sufficient to exceed the threshold for transitioning to ID 

mode for all of the iced conditions. 

RT-PID 

x Phase 1 runs had no pitch degrade, climb limit, or flap limit messages for zero, light, and 

moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 1 and 

considered to be successful. 

x Phase 2 runs had no pitch degrade, yaw degrade, climb limit, or flap limit messages for zero, 

light, and moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 

2 and therefore considered to be successful. 

x Phase 3 runs had no pitch degrade, climb limit, or flap limit messages for zero, light, and 

moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 3 and 

considered to be successful. 

x Phase 4 runs had no pitch degrade or yaw degrade messages for zero, light or moderate 

turbulence levels. As shown in Figure D.46 the calm atmosphere flap limit warning message 

annunciates at the same time as in light turbulence. This is considered to be successful based on 

the MOS criterion. The MOS criterion for AOA bracket was applied to Phase 4: Applying the 

MOS criterion would be unsuccessful in this phase since a pilot is able to discern the 

unconservative variation in the AOA brackets. 

x It is recommended that additional piloted runs be conducted to verify this conclusion by the 

virtual pilot. 

In general, it was found that the performance of the AIMMS-20 probe was satisfactory for Tasks 2 and 

3. Acceptable error is therefore defined by current specification of AIMMS-20 probe. As long as booms are 

not used and the calibration for wing response to turbulence has been taken care of the system will work. 

D.4.2 Additional Questions Raised by Research 

1. What is the calibration accuracy required of the wing response to turbulence for RT-PID estimates to 

be within ± 10 percent? 

2. What is the boom stiffness required for RT-PID estimates to be within ± 10 percent 

3. How can the other terms related to the Theil inequality coefficient, namely the bias, variance and covariance terms be used to develop a more robust detection scheme. 
NASA/CR—2014-218320 106 

D.4.3 Recommended Future Studies 

1. Execute Task 4. This task evaluates the effectiveness of manual pilot inputs alone for making high 

confidence vehicle state estimates when performing typical maneuvering flight tasks under varying 

levels of atmospheric turbulence. 

2. Flight test to validate RT-PID findings based on colored AOA regressor findings. 

3. Flight test to validate ICEPro. 

D.5 References 

1. Lawless, Al, “Inertial-Based Instrumentation for Performance Flight Testing,” SFTE Symposium XX 

Proceedings, 2008. 

2. Olson, Wayne, “Aircraft Performance Flight Testing,” AFFTC-TIH-99-01. 

3. Advisory Group for Aeronautical Research and Development, “Flying Qualities Flight Testing of 

Digital Flight Control Systems,” Flight Test Techniques Series – RTO AG-300, Volume 21, pp. 58-

60, Dec, 2001. 

4. Mathworks, “Dryden Wind Turbulence Model (Discrete)” 

[http://www.mathworks.com/help/toolbox/aeroblks/drydenwindturbulencemodeldiscrete.html. 

Accessed 1/20/12.] 

5. Maine, R. E. and Iliff, K. W., “Formulation and Implementation of a Practical Algorithm for 

Parameter Estimation with Process and Measurement Noise,” SIAM Journal of Applied 

Mathematics, Vol. 41, No. 3, 1981, pp. 558-579. 

6. Maine, R.E. and Iliff, K.W., “Identification of Dynamic Systems,” AGARD AG-300 Vol. 2, Jan. 

1985. 

7. Crawford, T.L., and R. J. Dobosy, 1992: A sensitive fast-response probe to measure turbulence and 

heat flux from any airplane. J. Boundary Layer Meteorology, 59, 257–278. 

8. Kalogiros, John A., Qing Wang, 2002: Aerodynamic Effects on Wind Turbulence Measurements 

with Research Aircraft. J. Atmos. Oceanic Technol., 19, 1567–1576. 

9. Kalogiros, J. A., and Q. Wang, 2002: Calibration of a radome-differential GPS system on a Twin 

Otter research aircraft for turbulence measurements. J. Atmos. Oceanic Technol., 19, 159–171. 

10. Lenschow, D.H., 1971: Vanes for sensing incidence angles of the air from an aircraft. Journal of 

Applied Meteorology 1989-2005 (After 2005 - Journal of Applied Meteorology and 

Climatology), 10, pp. 1339–1343. 

11. Sakamoto, G.M., “Aerodynamic Characteristics of a Vane Flow Angularity Sensor System Capable 

of Measuring Flight Path Accelerations for the Mach Number Range from 0.40 to 2.54,” NASA TND8242, 1976. 

12. Foster, G.W, “The Identification of Aircraft Stability and Control Parameters in Turbulence,” Ph.D 

Thesis, Cranfield Institute of Technology, 1982. 

13. Iliff, K.W., Maine, R.E., and Montgomery, T.D., “Important Factors in the Maximum Likelihood 

Analysis of Flight Test Maneuvers,” NASA TM-1459, 1979. 

14. Klein, V and Morelli, E.A., Aircraft System Identification Theory and Practice, AIAA Education 

Series, Reston, VA, 2006. 

15. Morelli, E.A., “Practical Aspects of the Equation-Error Method for Aircraft Parameter Estimation,” 

AIAA-2006-6144. 

16. Steers, S.T. and Iliff, K.W., “Effects of Time-Shifted Data on Flight-Determined Stability and 

Control Derivatives,” NASA TN D-7830, 1975. 
NASA/CR—2014-218320 107 

Appendix E.—ICEPro Atmospheric Turbulence Test Plan 

VSST – 1 Year Extension

Borja Martos 

Aviation Systems and Flight Research 

University of Tennessee Space Institute 

MAY 2011 
NASA/CR—2014-218320 108 

E.1 Abstract 

The effects of atmospheric turbulence on the RT-PID and D-ICES algorithms are studied by 

modeling representative atmospheric turbulence. Atmospheric turbulence is measured directly as an input 

to both methods. The performance of these algorithms is evaluated using reasonable assumptions for 

measurements from currently-available turbulence sensors in operational flight. Methods for improving 

the performance of these algorithms in these conditions are implemented and studied. 

E.2 Introduction 

E.2.1 Background 

A team comprised of personnel from The University of Tennessee Space Institute (UTSI), Tullahoma, 

Tennessee, Bihrle Applied Research Inc (BAR), Hampton, Virginia, NASA Glenn Research Center 

(GRC), Cleveland, Ohio, and NASA Langley Research Center (LaRC), Hampton, Virginia, has 

successfully developed a prototype Ice Contamination Envelope Protection System (ICEPro). The ICEPro 

system facilitates flight envelope protection by making continuous real time vehicle state assessments, 

which are synthesized into flyable pilot cueing along with visual and aural alerts during in-flight icing 

conditions. Detection of degraded aircraft stability and control and performance due to icing is carried out 

by a dynamic inversion control evaluation system (D-ICES) that compares expected aircraft behavior 

from a-priori knowledge base with current measures of those behaviors. When differences reach defined 

thresholds, real time parameter identification (RT-PID) methods are invoked to estimate current state 

parameters, which continuously support pilot cueing and alerts. The development effort included 

simulation-based design, testing, and verification. A pilot in-the-loop study was conducted to gather pilot 

performance data and opinions of the utility of ICEPro during simulated icing encounters. Results of the 

study indicated that the system performed as expected and pilot performance benefited from the envelope 

protection cues. However, the simulation could not duplicate all of the real-world conditions that could 

have first-order effects on system performance, such as those due to real atmospheric turbulence. 

Therefore, in order to assess system performance under these conditions and minimize the technical risk 

of an eventual flight-test validation program, the effects of atmospheric turbulence on ICEPro are studied. 

E.2.2 Participating Organizations 

The following organizations will provide simulator flight test support and facilities. 

UTSI: Is one of the primary organizations and facilities for simulator flight testing and algorithm 

development of the ICEPro system. The basis for the simulator flight tests will be a COTS 

desktop simulator running BAR’s D-six software and representative flight controls, visuals, and 

virtual aircraft instruments. 

BAR: Is one of the primary organizations for algorithm and software development of the ICEPro 

system. The basis for algorithm and software development will be a COTS desktop simulator 

running BAR’s D-Six software and representative flight controls, visuals, and virtual aircraft 

instruments. 

GRC: Will provide technical monitoring and serve as the NASA focal point for this grant. 

LaRC: Will provide technical assistant with real-time parameter identification algorithms. 

E.2.3 Overview of Test Concepts 

Simulator Flight Test 

The ICEPro systems software will be tested in Birhle’s D-six software environment. Software will be 

tested both at UTSI and BAR. In support of these tests, BAR will provide the following: 

1. Update the ICEPro software so that desktop simulations at BAR and UTSI match the ICEFTD. 
NASA/CR—2014-218320 109 

2. Update the ICEPro system logic diagram. 

3. Synchronize system software between UTSI and BAR as software changes are made. 

4. Author release notes documents with each software revision and update the system logic diagram 

were appropriate. 

E.2.4 Outline of Document 

The remainder of this document is divided into two parts. Section E.3 contains the simulator flight 

test plan and Section E.4 contains the simulator flight test tasks. 

E.3 Simulator Flight Tests 

E.3.1 Test Objectives 

The primary objectives of the simulator flight tests are to assess the performance of RT-PID and DICES algorithms in the presence of atmospheric turbulence and determine methods for improving 

algorithm performance in these conditions. The simulations will be based on events that cannot be 

repeatedly duplicated during flight testing and benefit from a controlled environment. 

The secondary objective is to access the performance of RT-PID and D-ICES algorithms in the 

absence of an onboard excitation system (OBES) with and without atmospheric turbulence. 

E.3.2 Test Guidelines 

E.3.2.1 Primary Objectives 

E.3.2.1.1 Performance Testing in Turbulence 

The performance of the RT-PID and D-ICES algorithms will be assessed by flying a low gain and a 

high gain pilot-in-the-loop task in the presence of atmospheric turbulence. The low gain task is straight 

and level unaccelerated flight with the flaps retracted. The high gain task is the final segment of an 

instrument approach. This represents a descent from the final approach fix to the inner marker and 

requires tight control of the glide slope and localizer. Both tasks will be performed under Visual 

Meteorological Conditions (VMC) conditions. Prior to these tasks, a UTSI pilot will train to ensure 

proficiency in basic instrument flying skills and can perform a precision approach within Airline 

Transport Pilot (ATP) standards. 

E.3.2.1.2 Development of Algorithms for Atmospheric Turbulence 

To develop RT-PID and D-ICES algorithms in the presence of atmospheric turbulence this study will 

utilize the previous pilot in-the-loop test scenario. A copy of this scenario is found in Figure E.1. 

However, to coordinate testing and algorithm and software development between UTSI and BAR, the test 

scenario will be broken up into a series of segments or phases of flight. These will consist of straight and 

level unaccelerated flight, flap transition, heading capture, descent, and glide slope and localizer tracking. 

The use of autopilots will provide a standardized pilot performance model for batch testing at UTSI and 

BAR. 

E.3.2.2 Secondary Objectives 

E.3.2.2.1 Preliminary Feasibility Study Without OBES 

Finally, the updated RT-PID and D-ICES algorithms will be assessed in the in the presence of 

atmospheric turbulence, with a pilot-in-the-loop, with and without an on board excitation system. The 

algorithms will be assessed by the same pilot-in-the-loop low gain and high gain tasks as in the initial 

task. Additionally, a UTSI pilot will be under the same training and flying standards as previously 

mentioned. 
NASA/CR—2014-218320 110 

E.3.3 Performance Parameters to be Measured 

All flight data will be recorded at 50 Hz. The following parameters will be recorded and calculated. 

1. OBES frequency and duration 

2. ISP value 

3. Error in ISP parameters 

a. CmĮ Longitudinal Static Stability 

b. Cmįe Elevator Control Power 

c. Clȕ Dihedral Effect 

d. Clįa Aileron Control Power 

e. CNĮ Normal Lift Coefficient 

4. Error in Message parameters 

5. Glide slope and localizer performance 

6. Airspeed performance 

7. Number of AOA exceedances 

a. Stick shaker and stick pusher firings 

b. STALL message postings 

c. Tail stall events 

E.3.4 Measures of Success (MOS) 

To ensure that test objectives are attained, MOS’s have been formulated. A MOS is defined as a 

comparative requirement which must be satisfied by the results of testing. The overall measure of success 

for the primary and secondary objectives will be that the error bounds and accuracy in RT-PID and Theil 

inequality coefficients in D-ICES algorithms meets or exceeds its non-atmospheric turbulence 

performance in atmospheric turbulence. This success criterion focuses on the performance of these two 

algorithms and not on ICEPro’s utility to mitigate a potentially hazardous icing encounter or on the pilot’s 

performance during such encounter. It assumes that if the RT-PID and D-ICES algorithms perform 

equally in and out of atmospheric turbulence the ICEPro system will perform equally. The following 

MOS are defined for the primary objective. 

RT-PID Algorithm 

5. Message accuracy 

Message error will be determined by the error bounds and accuracy of its associated stability and 

control coefficient. For example, the red pitch degrade logic is a function of elevator control power 

CmGe t 0.25CmGe _ no ice

The elevator control power must meet or exceed its non-atmospheric turbulence error bound and 

accuracy in atmospheric turbulence. This is repeated for all ICEPro messages. 

6. ISP accuracy 

ISP error will be determined by comparing the true ISP value with the estimated ISP value. ISP is a 

function of five stability and control coefficients: Longitudinal static stability, elevator control power, 

dihedral effect, aileron control power, and normal lift coefficient. The true ISP value is known since 

the five true stability and control parameters are known to the simulation. The ISP error must meet or 

exceed its non-atmospheric turbulence error bound and accuracy in atmospheric turbulence. 
NASA/CR—2014-218320 111 

D-ICES Algorithm 

2. Control deflection accuracy 

Control deflection error is determined by comparing the estimated deflection Theil inequality 

coefficient with the baseline no turbulence case. The control deflection error must meet or exceed its 

non-atmospheric turbulence error in atmospheric turbulence. 

E.3.5 Organizational Responsibilities 

The organizational responsibilities for each task are stated in Section E.4. Through each task the 

responsible party’s name is highlighted by parenthesis. 

E.4 Simulator Flight Test Tasks 

E.4.1 Task 1 (UTSI) 

Determine whether the scatter in the turbulence estimates arise from, 1) colored noise in the 

regressor(s), and if so what is the source of the colored noise and in which regressor(s), 2) insufficient 

model structure, and if so how should the model structure be adjusted or, 3) some other means. 

E.4.2 Primary Objective—Performance Testing in Turbulence 

Accomplished in Task 2 by baseline turbulence 

E.4.3 Primary Objective—Development of Algorithms for Atmospheric Turbulence 

E.4.3.1 Task 2 (UTSI and BAR) 

Modify, test, and validate the use of RT-PID and D-ICES algorithms with atmospheric turbulence as 

an input for various levels of atmospheric turbulence. Verify MOS during each phase of flight for varying 

atmospheric turbulence levels and icing conditions. UTSI and BAR will use the full 6-DOF nonlinear 

twin otter database. 

Autopilot Task (Batch Testing) 

1. Isolate the RT-PID problem from D-ICES and executive logic (UTSI) 

a. Flaps up straight and level flight for 30 sec at 110 KIAS and 2500 ft 

(1) Perfect Turbulence measurement—Optimal Inputs 

(a) Turbulence levels 

i. Baseline (no turbulence) 

ii. Light 

iii. Moderate 

iv. Severe 

(b) Ice 0 and 2 

(2) Degraded Turbulence measurement—Optimal Inputs 

(a) Turbulence levels 

i. Baseline (no turbulence) 

ii. Light 

iii. Moderate 

iv. Severe 

(b) Ice 0 and 2 

(c) Vary turbulence measurement error 0.3 to 1.5 kt in increments of 0.3 kt 

2. Isolate the D-ICES problem from RT-PID and executive logic (BAR) 

a. Repeat steps outlined in 1 
NASA/CR—2014-218320 112 

Software Requirements (Task 2 plus) 

1. D-six Light, Moderate, and Severe Turbulence Models (BAR) 

2. D-six level flight autopilot with OBES hook outside of the autopilot loop (BAR) 

3. D-six software switch providing isolation of D-ICES or RT-PID algorithms (BAR) 

4. Example script file that calls D-six within MATLAB, trims linear model to 110 KIAS, isolates an 

algorithm, selects a turbulence model, selects and icing state, runs the model for a given time, and 

saves one example parameter (BAR) 

5. Batch script file for RT-PID and D-ICES testing (UTSI and BAR respectively) 

6. Update RT-PID and D-ICES algorithms as required (UTSI and BAR respectively) 

E.4.3.2 Task 3 (UTSI and BAR) 

Modify, test, and validate the use of RT-PID and D-ICES algorithms with atmospheric turbulence as 

an input. Select an acceptable atmospheric turbulence measurement error based on findings from Task 2 

and commercially available atmospheric turbulence sensors. Verify MOS during each phase of flight for 

varying atmospheric turbulence levels and icing conditions. UTSI and BAR will use the full 6-DOF 

nonlinear twin otter database. 

Autopilot Task (Batch Testing) 

Phases of flight—Autopilot with Optimal Inputs 

1. Flaps up straight and level flight at 110 KIAS and 2500 ft—30 sec duration 

2. Flaps up straight and level flight at 110 KIAS and 2500 ft followed by a constant altitude flap 

transition (0° to 20°) to 80 KIAS – 1 min duration 

3. Heading capture at 80 KIAS, 2500 ft, and flaps 20° from 090° to 360° – 30 sec duration 

4. Descent from 3000 ft at 75 KIAS and flaps 20° followed by a flap transition (30°) to 70 KIAS with 

localizer and glide slope tracking – 2 min duration 

Ice Condition 

1. Ice 0 

2. Ice 2 in zero time 

Turbulence Levels 

1. Baseline (no turbulence) 

2. Light 

3. Moderate 

4. Severe 

Software Requirements (Task 1 plus) 

1. D-six autopilots for Phases 2 to 4 (BAR) 

2. Update RT-PID and D-ICES algorithms as required (UTSI and BAR respectively) 

E.4.4 Secondary Objective—Preliminary Feasibility Study without OBES 

E.4.4.1 Task 4 (UTSI and RR) 

Research Objectives 

1. Evaluate the effectiveness of manual pilot inputs alone for making high confidence vehicle state 

estimates when performing typical maneuvering flight tasks under varying levels of atmospheric 

turbulence. 

a. Determine the portion or portions of the maneuvers, which provide frequency content at the 

natural frequency of the short period, dutch-roll, and roll modes and thereby support the 

achievement of high confidence parameter estimates from RT-PID: 

(1) Without use of the OBES. 
NASA/CR—2014-218320 113 

(2) With OBES. 

(3) Optionally, acquire pilot workload assessment for (1) and (2) 

Assumptions 

1. Accomplishment of this objective assumes that the goal of system performance as stated in the 

primary objectives has been met. 

Test Design 

1. This test will be designed and executed to acquire data that will enable the researcher to compare the 

differences between the effect of manual only pilot control inputs and pilot inputs that are augmented 

by the operation of the OBES on RT-PID vehicle state estimates. This comparison will be made for 

four atmospheric turbulence conditions, which were previously defined under Task 1. The required 

data will come from maneuver time histories of pilot control inputs with and without OBES 

“buzzing” and from RT-PID derived confidence intervals of key stability and control coefficients in 

each of the aircraft’s three rotational axes. The a priori truth versus the current estimate for each 

stability and control coefficient in the ISP formulation will be used to determine the accuracy in the 

estimate and will be reflected in the magnitude error bars for each coefficient of interest. This will 

form the basis for pilot alerting when the state estimate needs to be improved. A power spectral 

density analysis of control activity with and without OBES, for each part of the maneuver where a 

poor vehicle state estimate exists will be used to determine the amount of energy in the inputs at the 

desired frequency. Coherence plots will then be developed will facilitate the comparison between the 

non-OBES and OBES control input condition. 

Method of Test 

1. Task 4 will be divided into two subtasks. 

a. The first subtask will be an initial evaluation of normal manual pilot inputs for achieving vehicle 

state estimates while performing basic flight maneuvers. These maneuvers will include steady 

level flight, climbs and descents, changes in airspeed, turns to headings, wing flap 

extensions/retractions, and power transitions. These flight maneuvers are described in the 

following section, and will be performed with and without OBES operation. This subtask will be 

performed in smooth atmospheric conditions, and later, under various levels of atmospheric 

turbulence. The goal of this testing is to acquire data that establishes the relationship between 

pilot inputs alone, and pilot inputs with OBES when performing basic aircraft maneuvers, and 

their effect on achieving the required level of confidence in vehicle state estimation. The results 

of this initial testing will be used to verify system functionality, ensure readiness for more 

extensive data collection as described in the second subtask below, and to verify data collection, 

processing, and analysis methods In addition, this testing will identify and address any anomalies 

found with pilot cueing and messaging such nuisance alerts and toggling. 

b. The second subtask will be an extensive series data collection tests flown by a test pilot to 

evaluate manual pilot control input gain on vehicle state estimates under various conditions of 

atmospheric turbulence. This will involve three repeats of each maneuver after the pilot has 

practiced and reached a consistent level of performance. These tests will be performed both with 

and without OBES operation. Two means of varying pilot control activity and gain will be 

employed during this subtask. The first is by increasing desired maneuver performance 

requirements, which involves specifying high and low error tolerances during each set of repeat 

maneuvers. The large tolerance will result in lower gain and control activity, and the small 

tolerances should result in higher gain and control activity. A second means of affecting pilot gain 

and control activity will be to simply alert the pilot via TBD methods (voice or possibly 

messaging) that more aggressive inputs are required in any or all of the pitch, roll, and yaw axes 

until the alerting requirements no longer exist. It is also expected that some maneuvers may not 
NASA/CR—2014-218320 114 

provide sufficient vehicle excitation in all the aircraft axes, e.g., level climbs and descents may 

not provide lateral directional excitation, and it may be necessary to combine the two methods to 

obtain a satisfactory state estimate. In this case, the pilot may be alerted to make inputs in the 

affected axes if a full state estimate is desired for that test point. As a final evaluation of manual 

pilot inputs, an operationally representative flight profile consisting of an instrument approach 

procedure as shown in Figure E.1 will be performed under each of the turbulence conditions 

defined in this plan. In addition to evaluating the effectiveness of manual pilot inputs on vehicle 

state estimation, the data derived from this subset of testing will also assess the effect of manual 

inputs and OBES operation on pilot workload. 

c. Pilot tasks will be performed under simulated IMC conditions. 

Data Requirements 

1. Test data will be sampled at a minimum of 10 Hz. The sampled parameters include time histories of 

the following: 

a. Pressure altitude Hp 

b. Airspeed VIAS 

c. Angle of attack (AOA) Į

d. Angle of sideslip (AOS) ȕ

e. Heading angle Ȍ

f. Pitch angle ș

g. Bank angle ij

h. Control and wing flap position įeįaįrįf

i. Body rates and accelerations p, q, r, AX, AY, AZ, 

j. Localizer and GS errors Localizer and glide slope Theil inequality coefficient 

k. State coefficient estimates CmD,Cmįe, Clȕ, Clįa, CND, Cnįr estimate, error, and truth 

Pilot Performance Task Standards 

1. To ensure consistency, flight standards are provided, which will cause pilot gain to be either high or 

low. Low gain will result from the larger error tolerance values, and high gain will result from the 

lower tolerance values. The appropriate standard will be specified on the test card for the maneuver 

event. 

a. Altitude control for level flight: ± 150 and ± 20 ft 

b. Heading control: ± 5 ° and ± 1 ° 

c. Climb/descent rate: full power, idle power, power as required per test card 

d. Airspeed: ± 5 kt and ± 1 kt 

e. Roll rates: ± 10 °/sec and ± 5 °/sec 

f. Localizer course intercept angle: 45 ° and 30 ° 

g. Localizer error: ± 1 dot and zero lateral deviation + 10 kt X wind 

h. Glide slope error: ± 1 dot and zero vertical deviation + 10 kt X wind 

Maneuver Descriptions 

1. The descriptions of each maneuver below are generalized. Detailed test cards will provide specific 

instructions as to how each maneuver is to be accomplished. Each evaluation will be conducted in 

four levels of turbulence. Icing configuration will be ICE 0 for the initial and final evaluation and ICE 

02 for the instrument approach procedure as shown in Figure E.1. The 10 kt cross wind in addition to 

the turbulence will be applied for only the instrument approach procedure. Repeat each procedure 3 

times using low gain standards, and then repeat 3 times using high gain standards. Initialization 

conditions will be specified for each maneuver on the test card. 

2. Basic flight maneuvers 

a. Steady level flight at constant heading. 
NASA/CR—2014-218320 115 

(1) Perform at 125 KIAS and 1.3 versus (90 KIAS) with PFLF 

(2) Stabilize on condition with PFLF and call “mark” when stable 

(3) Hold heading and altitude within specified parameters and take data for 3 min 

b. Level turns to headings 

(1) Perform at 125 KIAS and 1.3 versus (90 KIAS) 

(2) Stabilize on condition with PFLF and call “mark” when stable 

(3) Remain steady level for 15 sec. Using power as required maintaining constant airspeed and 

altitude while performing the following: Turn right to a heading of 090 and maintain steady 

level flight for 15 sec. Turn left to 045 and maintain steady flight for 15 sec. Turn left to 030 

and maintain steady level flight for 1 min. 

c. Constant airspeed and vertical speed climbs/descents on a steady heading 

(1) Perform at 125 KIAS and 1.3 versus (90 KIAS) 

(2) Stabilize on condition with PFLF and call “mark” when stable. 

(3) Using power as required, maintain constant airspeed and climb at specified climb rate. Level 

off at 6000 ft and remain steady for 15 sec. Using power as required, begin a constant 

airspeed descent at the specified rate and level off at 5000 ft. Maintain steady level flight for 

15 sec. 

d. Constant airspeed climbing and descending turns to headings 

(1) Perform at 125 KIAS and at 1.3 versus (90 KIAS) 

(2) Remain steady level for 15 sec. Select maximum power and turn right to a heading of 090. 

Maintain constant airspeed in climb and level off at 6000 ft. Stabilize on airspeed and 

heading of 090 for 15 sec. Select idle power and turn left to 360. Maintain constant airspeed 

in the descent and level off at 5000 ft. Stabilize on airspeed and heading for 15 sec. 

(3) Repeat (ii) and use power as required to establish a 300 fpm climb and descent rate 

e. Wing flap transitions 

(1) Initialize on heading 360, PFLF at 85 KIAS, wing flaps up. When stable, call “mark” and 

hold steady condition for 15 sec. Select wing flaps DOWN and use power as required to 

maintain airspeed and altitude while holding heading constant. After wing flaps reach 

DOWN, stabilize and hold the steady condition for 15 sec. 

(2) Stabilize for 1 min with wing flaps full down at 85 KIAS. When stable, call “mark” and 

select wing flaps to UP and use power as required to maintain airspeed while keeping altitude 

and heading constant as the flaps retract to the fully up condition. Stabilize and hold the 

steady condition for 15 sec. 

f. Power transitions: 

(1) Initialize on a 360 heading, at 85 KIAS, wing flaps up, PFLF. When stable, call “mark” and 

rapidly apply full power while maintaining heading and level flight until reaching the 

maximum speed attainable. When stable at maximum speed, hold condition for 15 sec. 

(2) Initialize on a 360 heading, at maximum level flight speed, wing flaps up, PFLF. When 

stable, call “mark” and rapidly reduce power to idle while maintaining heading and altitude. 

When approaching 90 KIAS, add power as required to maintain constant heading and 

airspeed. When stable at 90 KIAS, hold condition for 15 sec. 

g. Operational evaluation—Instrument Approach Procedure 

(1) Initialize with wing flaps up, 3000 ft, heading 360, and 125 KIAS, ICE02 

(2) Conduct the approach procedure per Figure E.1 without OBES, and provide pilot alerting 

when state estimates need to be refreshed with additional control activity 

(3) Evaluate workload using NASA TLX 

(4) Repeat the procedure with OBES functioning 

(5) Evaluate workload using NASA TLX 
NASA/CR—2014-218320 116 

Turbulence Levels as Defined by MIL-F-8785C 

1. Smooth air, no turbulence 

2. Light turbulence 

3. Moderate turbulence 

4. Severe turbulence 

Test Cards 

1. Detailed test cards will be developed for each of the two sub tasks from the maneuver descriptions 

above. These test cards will be provided in a standard UTSI test template format and retained as part 

of the data package for each test session. Any deviations from initially planned test procedures will be 

noted on the test cards. 

Software Requirements 

1. D-six executive logic software switch enabling / disabling OBES functionality (BAR) 

2. Alerting the pilot that control excitations are required (BAR) 

a. Update excite button functionality 

b. Apply stop light color scheme 

(1) Tie color scheme to RT-PID confidence bounds 

(a) Parameters in ISP calculation define axis color 

a. Differentiate scheme depending on which axis or axes require excitation 

Figure E.1.—Test and evaluation profile. Total duration of test is approximately 15 min. Quotes 

are clearances read to the EP, italics are required EP actions per the test protocol.