NASA/TM—2014–216640

 

Complexity and Pilot Workload Metrics for 

the Evaluation of Adaptive Flight Controls 

on a Full Scale Piloted Aircraft 

Curt Hanson, Jacob Schaefer, John J. Burken, and David Larson

Dryden Flight Research Center, Edwards, California

Marcus Johnson

Ames Research Center, Moffett Field, California

 

Click here: Press F1 key (Windows) or Help key (Mac) for help 

February 2014
NASA STI Program ... in Profile

Since its founding, NASA has been dedicated 

to the advancement of aeronautics and space 

science. The NASA scientific and technical 

information (STI) program plays a key part in 

helping NASA maintain this important role.

The NASA STI program operates under the 

auspices of the Agency Chief Information Officer. 

It collects, organizes, provides for archiving, and 

disseminates NASA’s STI. The NASA STI 

program provides access to the NASA 

Aeronautics and Space Database and its public 

interface, the NASA Technical Reports Server, 

thus providing one of the largest collections of 

aeronautical and space science STI in the world. 

Results are published in both non-NASA channels 

and by NASA in the NASA STI Report Series, 

which includes the following report types:

TECHNICAL PUBLICATION. Reports of 

completed research or a major significant 

phase of research that present the results of 

NASA Programs and include extensive data 

or theoretical analysis. Includes compilations of significant scientific and technical 

data and information deemed to be of 

continuing reference value. NASA counterpart of peer-reviewed formal professional 

papers but has less stringent limitations on 

manuscript length and extent of graphic 

presentations.

TECHNICAL MEMORANDUM. 

Scientific and technical findings that are 

preliminary or of specialized interest, 

e.g., quick release reports, working 

papers, and bibliographies that contain 

minimal annotation. Does not contain 

extensive analysis.

CONTRACTOR REPORT. Scientific and 

technical findings by NASA-sponsored 

contractors and grantees.

CONFERENCE PUBLICATION. 

Collected papers from scientific and 

technical conferences, symposia, seminars, 

or other meetings sponsored or 

co-sponsored by NASA.

SPECIAL PUBLICATION. Scientific, 

technical, or historical information from 

NASA programs, projects, and missions, 

often concerned with subjects having 

substantial public interest.

TECHNICAL TRANSLATION. 

English-language translations of foreign 

scientific and technical material pertinent to 

NASA’s mission.

Specialized services also include organizing 

and publishing research results, distributing 

specialized research announcements and 

feeds, providing information desk and personal 

search support, and enabling data exchange 

services.

For more information about the NASA STI 

program, see the following:

Access the NASA STI program home page 

at http://www.sti.nasa.gov

E-mail your question to help@sti.nasa.gov

Fax your question to the NASA STI 

Information Desk at 443-757-5803

Phone the NASA STI Information Desk at 

443-757-5802

Write to:

STI Information Desk

NASA Center for AeroSpace Information

7115 Standard Drive

Hanover, MD 21076-1320

This page is required and contains approved text that cannot be changed.
NASA/TM—2014–216640

 

Complexity and Pilot Workload Metrics for 

the Evaluation of Adaptive Flight Controls 

on a Full Scale Piloted Aircraft 

Curt Hanson, Jacob Schaefer, John J. Burken, and David Larson

Dryden Flight Research Center, Edwards, California

Marcus Johnson

Ames Research Center, Moffett Field, California

Insert conference information, if applicable; otherwise delete 

Click here: Press F1 key (Windows) or Help key (Mac) for help 

National Aeronautics and

Space Administration

Dryden Flight Research Center

Edwards, CA 93523-0273

February 2014
 Enter acknowledgments here, if applicable. 

Click here: Press F1 key (Windows) or Help key (Mac) for help 

Available from:

Click here: Press F1 key (Windows) or Help key (Mac) for help 

NASA Center for AeroSpace Information

7115 Standard Drive

Hanover, MD 21076-1320

443-757-5802
1

Abstract

Flight research has shown the effectiveness of adaptive flight controls for improving aircraft 

safety and performance in the presence of uncertainties. The National Aeronautics and Space 

Administration’s (NASA)’s Integrated Resilient Aircraft Control (IRAC) project designed and 

conducted a series of flight experiments to study the impact of variations in adaptive controller 

design complexity on performance and handling qualities. A novel complexity metric was 

devised to compare the degrees of simplicity achieved in three variations of a model reference 

adaptive controller (MRAC) for NASA’s F-18 (McDonnell Douglas, now The Boeing Company, 

Chicago, Illinois) Full-Scale Advanced Systems Testbed (Gen-2A) aircraft. The complexity 

measures of these controllers are also compared to that of an earlier MRAC design for NASA’s 

Intelligent Flight Control System (IFCS) project and flown on a highly modified F-15 aircraft

(McDonnell Douglas, now The Boeing Company, Chicago, Illinois). Pilot comments during the 

IRAC research flights pointed to the importance of workload on handling qualities ratings for 

failure and damage scenarios. Modifications to existing pilot aggressiveness and duty cycle 

metrics are presented and applied to the IRAC controllers. Finally, while adaptive controllers 

may alleviate the effects of failures or damage on an aircraft’s handling qualities, they also have 

the potential to introduce annoying changes to the flight dynamics or to the operation of aircraft 

systems. A nuisance rating scale is presented for the categorization of nuisance side-effects of 

adaptive controllers.

Nomenclature

Acronyms

ACAT Autonomous Collision Avoidance Technology

ESA European Space Agency

FAST full-scale advanced systems testbed

Gen-1 first-generation IFCS adaptive flight control system

Gen-2 second-generation IFCS adaptive flight control system

Gen-2A update to the second-generation IFCS adaptive flight control system

IFCS intelligent flight control system

IRAC integrated resilient aircraft control

MRAC model reference adaptive control 

NASA National Aeronautics and Space Administration

NDI nonlinear dynamic inversion

onMRAC MRAC with normalization and optimal control modification

onMRAC+ onMRAC with an additional adaptive parameter

PIO pilot-in-the-loop oscillation

RFI request for information

sMRAC simple MRAC

SOHO Solar Heliospheric Observatory

Mathematical Symbols

 matrix of input coefficients

 error feedback compensator



 estimate of applied aerodynamic and inertial moments

 inertia matrix

	 measure of complexity

	 measure of pilot aggressiveness


 reference model input
2

 reference model

s Laplace operator

 time


 weighting terms

 state vector

 state derivative vector

 state error vector

 output vector

 surface command vector

 pilot roll stick input



 steady state component of pilot roll stick input

 pilot pitch stick input



 steady-state component of pilot pitch stick input

 quantity measure for a category of complexity elements

 scalar, time-varying disturbance uncertainty estimate

 time index

 elapsed time between data points

 vector of body-axis rotational rates

Subscripts

0 initial

 adaptive augmentation

 compensator augmentation

 ! command

"

 NDI reference model tracking error

 final

/# input / output

$ tunable parameter

%& lookup table

 MRAC reference model

'( mode discrete 

)% nonlinear


" reference model

*  integration

 differentiation

+, inversion

Superscripts

  maximum available control input

 - minimum available control input

Introduction

This report represents a proposed set of metrics to quantify the relative complexity of 

multiple flight control system designs and for evaluating changes in pilot workload due to the 

behavior of an adaptive controller. Over the past 50 years, flight research has demonstrated that 

adaptive flight controls can be an effective technology for improving aircraft safety and

performance in the presence of uncertain and changing environments (refs. 1–9). However, the 
3

nonlinear, time-varying nature of adaptive algorithms continues to challenge traditional methods 

for the verification and validation of safety-critical flight control systems.

In April of 2009, The National Aeronautics and Space Administration’s (NASA)’s Integrated 

Resilient Aircraft Control (IRAC) project disseminated a request for information (RFI) to the 

adaptive controls community seeking ideas for potential flight experiments (ref. 10). A workshop 

was held in Chicago in August of 2009 with representatives from industry, academia, and other 

government agencies to discuss the wide variety of RFI responses received by NASA. Three 

focus areas were identified through this process.

1. Simple, yet effective, adaptive control algorithms should be investigated to help address 

the issue of verification and validation of adaptive flight controls to a safety-critical level.

2. The appropriate level of pilot awareness and interaction with adaptive control systems 

should be studied.

3. Techniques should be matured for incorporating feedback information, both static and 

dynamic, from the structure of an aircraft into the flight control system to ensure structural 

integrity during adaptation.

The first two of these focus areas prompted the development of a low complexity, 

textbook-like direct model reference adaptive control (MRAC) scheme (ref. 11) for the NASA 

Full-Scale Advanced Systems Testbed (FAST) (refs. 8–9). FAST is a highly modified F-18 

(McDonnell Douglas, now The Boeing Company, Chicago, Illinois) aircraft that contains a 

research flight control system capable of housing advanced flight controls experiments. 

Full-scale piloted in-flight experimentation has been shown to be an effective method for 

uncovering implementation issues of adaptive systems (refs. 1, 4–9). A philosophy of “simpler is 

better” motivated the development of the low complexity MRAC formulation under the 

assumption that simplification leads to lower implementation and verification costs, and 

ultimately to a greater likelihood of acceptance by aircraft manufacturers, operators, and 

certification authorities (refs. 12–13). Reduced complexity can also lead to a safer design by 

reducing the potential for software implementation errors and unexpected interactions, both 

within the aircraft systems and between the controller and the human pilot (refs. 14–17).

Pilot interaction with adaptive systems can be either beneficial or have unwanted

consequences. Beneficial interactions include decisions made by the pilot based upon 

information regarding the performance and status of the system. For example, the pilot may 

configure the controller specifically to adapt to a failed control surface, poor pitch damping or for 

severe cross-axis coupling. While altering the closed-loop dynamics of the aircraft to 

accommodate failures or damage, adaptive control also has the potential to adversely affect the 

handling qualities and pilot-in-the-loop oscillation (PIO) characteristics of the system. The 

potential for adverse interactions is compounded by simultaneous changes in piloting technique 

as the pilot also adapts to the altered aircraft’s flight characteristics. Additional adverse 

interactions between the pilot and adaptive controller include increased workload for the pilot 

and nuisance side effects caused by the adaptation.

The purpose of this report is to suggest potential metrics that will aid the designer in 

selecting and evaluating an appropriate adaptive control design for their application. Other 

important metrics not addressed in this report include measures of the system’s robustness and 

performance, which have been addressed previously by other authors (ref. 18).
4

Complexity Metric

Any flight control design theory must buy its way onto an aircraft by meeting performance 

and robustness specifications without being overly burdensome to design, implement, and 

qualify for flight. Control theories that are complicated to implement and difficult to prove the 

safety of will be bypassed in favor of those that may not perform as well, but can meet the 

project’s cost and schedule requirements. Such a trade study requires the ability to evaluate a 

priori the difficulty in bringing a particular control design to flight. By documenting the 

implementation complexity for future researchers, these metrics begin to construct a 

knowledge-base of experience for the controls community.

Traditionally, metrics used for evaluating control methods are often nebulous in nature and a 

result of “tribal knowledge” within the controls community. The necessity of full-state feedback, 

the number of generated controller states, and the difficulty of gain scheduling are some

examples of potential drawbacks that are often cited when considering various modern control 

methods. Implementation of adaptive control is relatively new to the flight controls community,

and engineers may not have the requisite experience to draw from when making a 

determination of an adaptive algorithm’s suitability for a given application.

Formal complexity metrics are generally designed for application to the software 

implementation of a system (ref. 19). Examples include the Halstead complexity metrics, which 

include concepts such as volume, difficulty, and effort derived from the number and variety of 

operators and operands in a software program (ref. 20). However, no attempt is made within 

these metrics to differentiate between the relative complexity of different operations. The 

cyclomatic complexity measures of McCabe are based upon the number of linearly independent 

paths through a software program (ref. 21). Neither of these approaches accounts for 

differences in complexity between different elements of control system design.

Complexity metrics can be used to characterize the impacts of theory selection on the costs 

of designing, implementing, and flight qualifying a control system. The metrics are designed to 

evaluate both fundamental elements of the control theory and any additional elements that,

while required for implementation, are often omitted from theoretical descriptions. While 

motivated by the need to evaluate adaptive control theories, these metrics are applicable to any 

control theory.

Quantitative Complexity Metric Description

Quantitative metrics provide an objective means for comparison between competing control 

methodologies prior to selection and can measure the impact of modifications to the complexity 

of the algorithm during implementation. A proposed quantitative metric for measuring the 

complexity of a control system is given in equation (1). The metric is similar in approach to 

Halstead’s complexity metrics (ref. 20) in that it derives a measure of complexity by counting the 

number of different control system elements used in the design. The proposed method differs 

from Halstead’s primarily in that it accounts for variations in complexity between different types 

of elements.

	=
,./1 + 
34* 5 + 5 + 567 + 89: + 
;< + 
>9? + 
@AB (1)

The scalar J is a measure of complexity for the controller. Each component of the control 

design is categorized and tallied, and category totals represented by the series of variables
5

CDE. Weighting terms 
, F 
@ C GE are used to adjust for inherent differences in the impact 

to complexity of the various component categories.

Relative magnitudes for the weighting terms are not proposed in this report. Their

determination will require a rigorous and detailed tracking of resource utilization during multiple 

control law development programs. The complexity metric function in equation (1) is a linear 

approximation that does not contain element interdependencies and other non-linear 

relationships such as those of the Halstead metrics (ref. 20). For example, an increase in the 

number of tunable parameters may cause the complexity of the controller to rise at a 

faster-than-linear rate due to their inter-dependencies. Identification of any higher-order terms

would be subject to the same empirical investigation as the weighting terms.

Input and Output Plane Dimensionality 4HI/K:

The dimensions of the input and output planes impact control system development in 

several ways. Linear robustness analysis is typically performed at each feedback loop and each 

controller output. Noise, latency and sampling rate characteristics for each input and actuator 

dynamics for each output, must be identified and modeled. During testing, the failure modes of 

each input and output must be understood and the effects evaluated. Some examples of the 

importance of understanding possible failure modes of feedback signals include the crashes of 

the NASA X-31 (Enhanced Fighter Maneuverability (EFM) demonstrator) (ref. 22) and Air 

France A330-200 Flight 447 (ref. 14), both cases in which Pitot tube icing was a contributor.

Integrators, Derivatives, Inverses and Nonlinear Elements 4H* L, HL

, HL6N, HOP:

Integrators introduce complications to the control system primarily through the need to 

ensure their proper initialization on startup and during mode changes. Examples of incidents 

caused by improper handling of integrators include the premature retirement of NASA’s

Demonstration for Autonomous Rendezvous Technology (DART) spacecraft (ref. 23), and the 

mission interruption of the joint NASA and European Space Agency (ESA) Solar Heliospheric 

Observatory (SOHO) (ref. 24). Derivatives of signals introduce noise into the system, which can 

ultimately lead to numerical instability and may require filtering. The number of scalar divides 

and matrix inversions add to the complexity of the system by requiring special handling to avoid 

numerical errors. Nonlinear elements such as integrator limits, saturation and rate limits, 

nonlinear mathematical operations, and dead-bands typically complicate linear robustness 

analysis and create additional test scenarios. These four elements share a common weighting 

coefficient because they are estimated to have roughly the same impact on the cost of control 

system development.

Tunable Parameters (HR)

Tunable parameters are elements of the controller such as gains, thresholds, and filter 

coefficients that can be adjusted to affect controller performance. These parameters tend to 

complicate the design and verification of the controller considerably due to the potential for 

unforeseen interactions between them. The fewer tunable parameters a system has, the easier 

it is for the designer to understand the impacts of adjusting the design to achieve the desired 

result. Tunable parameters also have an impact on the “fly-fix-fly” process of flight testing as 

further adjustments must be made to correct for a lack of model fidelity. In 1998, American 

Airlines flight 1340 crashed just short of the runway due to excessive pitch oscillations caused 

by a gain in the autopilot of the Boeing 727 that was not properly designed for the approach 

speed used by the pilots (refs. 17 and 25).
6

Lookup Table Dimensionality (HPT)

A lookup table encodes a function for use in the control system. Examples of look-up tables 

include gain schedules and coefficient tables. This metric captures their total dimensionality by 

summing together the number of independent input variables. Ensuring proper interpolation 

within the boundaries of the table and extrapolation or limiting outside the table complicates the 

testing effort. These types of tables often also include additional complicating features such as 

pre-lookup calculations and index memory for efficient operation.

Mode Discretes (HUV)

Switchable modes within a control system, such as flap and gear settings, and autopilot 

modes require considerable effort to test, which is especially true when it comes to evaluating 

the interactions of multiple modes and the effects of switching from one mode to another. One 

example of the unexpected complexity effects of mode transitions is the crash of TWA Flight 

843, in which the combination of a state transition from ground to flight and a faulty 

angle-of-attack sensor contributed to a false stall warning immediately after takeoff (ref. 16).

Complexity Comparison between Multiple Adaptive Controllers

The complexity metrics are applied to three variations of the IRAC experiment model 

reference adaptive controllers (refs. 8–9) designed for the FAST aircraft, and to the Gen-2A

controller from the Intelligent Flight Control System (IFCS) experiment (refs. 6–7). Both 

experiments were conducted under the NASA Aviation Safety program. The IFCS flight 

research program was conducted on a highly modified F-15 aircraft (McDonnell Douglas, now 

The Boeing Company, Chicago, Illinois) and concluded approximately 2 years prior to the 

beginning of the IRAC flights on the FAST F-18 aircraft. Despite implementation on different 

avionics systems and different aircraft, the two adaptive controls experiments shared a similar 

set of top-level design requirements. Both experiments were designed to evaluate direct 

adaptive model-reference algorithms integrated with nonlinear dynamic inversion (NDI) baseline 

controllers as pilot-initiated emergency systems. The simulated failures used to evaluate the 

controllers were also similar, including a destabilized pitch axis scenario and a failed stabilator.

Controller Descriptions

Figure 1 shows the top-level block diagram of the integrated MRAC and NDI controllers from 

the IRAC experiment. Three variations of the MRAC were implemented and tested to explore 

the relationships between adaptive controller complexity and performance, robustness, and 

handling qualities. The three controllers differed from each other by adding successive levels of 

complexity rather than implementing three entirely independent designs. The positive and 

negative effects of the additional complexity could then be studied. The simplest design,

referred to as sMRAC, consisted of a single adaptive parameter in each of the pitch and roll 

axes, computed using a basic gradient-based update law. A second design, onMRAC,

increased the complexity of the update law, while the third, onMRAC+, included one additional 

adaptive parameter in each axis. A detailed description of the IRAC controllers is given in

reference 26.
7

Figure 1. IRAC NDI and MRAC controller architectures.

The objective of the IFCS research effort was to develop, through flight research, adaptive 

algorithms to enhance control during primary control surface failures and other aerodynamic 

changes resulting from damage. The first flight phase, referred to as Generation 1 or Gen-1 for 

short, focused on an indirect adaptive aerodynamic parameter identification method (ref. 27).

The second generation, Gen-2, used neural networks to provide direct adaptive command 

augmentation signals. In principle, this approach does not require information on the nature of 

the failure or the extent of the damage. During flight evaluation with simulated failed control 

surfaces, the Gen-2 controller exhibited poor handling qualities. The goal of Gen-2A, shown in 

figure 2, was to modify the Gen-2 control architecture to improve command tracking and 

handling qualities with control surface failures (ref. 7).

Figure 2. IFCS Gen-2A controller architecture.
8

Complexity Metric Application

The complexity metric measures for the IRAC and IFCS controllers are listed in table 1.

Values are given separately for the pitch and roll axes of each controller. Elements of the 

system that existed solely to facilitate flight testing or accommodate research hardware were not

included in the determination of complexity. None of the controllers that were evaluated 

contained derivatives or lookup tables, so those categories are eliminated from table 1 in the 

interest of brevity. Integrators and nonlinear elements which contain tunable parameters were 

counted separately from their tunable parameters, and calculations that were used more than 

once were only counted once wherever possible.

Table 1. Complexity measures.

sMRAC onMRAC onMRAC+ IFCS Gen-2A

I/O dimensions

./1

pitch 3 3 3 7

roll 3 3 3 7

integrators

* 5

pitch 5 5 6 1

roll 1 1 2 1

inverses

567

pitch 0 1 2 4

roll 0 1 2 4

non-linear terms

89

pitch 10 18 20 61

roll 11 16 19 61

tunable gains

<

pitch 14 16 22 47

roll 12 14 20 47

mode discretes

AB

both 3 3 3 2

One of the goals of the IRAC adaptive controls experiment was to design three controllers 

with varying levels of complexity. From table 1 it is apparent that the difference in complexity 

between the IRAC controllers is primarily due to the number of non-linear elements and tunable 

parameters. These differences are mostly a result of increases in the complexity of the adaptive 

update laws.

By four of the six measures in table 1, all three of the IRAC controllers are less complex than 

the IFCS Gen-2A control law. The key difference in complexity between the IRAC controllers 

and the IFCS controller is the number of inputs, identified by the measure, ./1. The larger 

number of inputs used by the IFCS controller drives the substantial increases in the number of 

nonlinear elements, 89, and in the number of tunable parameters, <.

The seemingly incongruous decrease in the number of integrators required by the IFCS 

Gen-2A pitch axis when compared to the IRAC controllers is due to the inclusion of the integral 

of pitch rate as a controlled variable in the IRAC architecture. As described in reference 26, the 

influence of the integral of pitch rate is essentially zeroed out within the controller, but related 

calculations were included in the IRAC controller’s complexity measures for completeness.
9

The complexity metric in equation (1) was applied twice to each of the four adaptive 

controllers in table 1, using two different sets of arbitrary weightings. The resulting complexity 

measures are plotted in figure 3 along with handling quality ratings (ref. 28) gathered during 

research flights with simulated failures. All five controllers were evaluated with a failed stabilator. 

The NDI and IRAC controllers were also tested with unstable pitch dynamics, unstable roll 

dynamics, and significant roll-to-pitch coupling; these failures were not included in the IFCS 

project. Handling quality ratings for the NDI and IRAC controllers were taken from references 8 

and 9, while the IFCS controller ratings are previously unpublished.

Figure 3. Complexity measure versus handling qualities rating for simulated failures.

The complexity of the NDI controller is defined as a baseline of zero since it does not 

contain an adaptive controller. The complexity metric was first applied using a weighting set of 

unity. The metric was then applied a second time, in which the tunable parameter weighting was 

set to ten to reflect the difficulty of adaptive control tuning, while the input-output weighting and 

mode discrete weightings were both set to five, under the assumption that they are greater 

contributors to complexity than the mathematical elements, but not as significant as tunable 

parameters.

By either measure of complexity, the three IRAC controllers are significantly less complex

than the IFCS Gen-2A controller, while providing approximately the same level of handling 

qualities performance in the presence of failures. Although the onMRAC+ controller is the most 

complex of the three IRAC designs, it exhibits the lowest and most compact set of handling 

qualities ratings, indicating that the additional complexity may provide improved performance.

Pilot Workload Metrics

Pilot comments during the IRAC research flights indicated that workload played a significant 

role in the pilot’s handling qualities ratings. Traditionally, Cooper-Harper handling qualities 
10

ratings (ref. 28) and the PIO rating scales are used to assess pilot workload. The 

Cooper-Harper scale, in particular, requires the pilot to assess the perceived level of 

compensation required to accomplish the desired task. A more qualitative measure of pilot 

workload can be calculated by cross-plotting measures of the pilot’s aggressiveness and duty 

cycle against each other, as proposed by Shepherd (ref. 29).

Aggressiveness and Duty Cycle Metrics

Pilot aggressiveness is a measure of dynamic control deflection, defined by equation (2) as 

the time-averaged summation of the unbiased pilot roll and pitch stick inputs during the 

maneuver. The roll and pitch control bias, 

() and () respectively, is found by forward 

and backward low-pass filtering of the original signal. Filtering the data in both directions 

eliminates any filter-induced phase shift. Each unbiased control measure is normalized by the 

total allowable deflection range of the appropriate control inceptor.

	 = 100%

Y Z [

\ ]

^() Z 

()^



_5 Z _`a + ^() Z 

()^



_5 Z _`a b c

de

fgdh

(2)

The aggressiveness metric proposed in equation (2) differs from that of Shepherd (ref. 29)

by introducing a subtraction of the bias terms 

() and (). This modification avoids the 

errant categorization of steady, non-zero pilot deflections as highly aggressive commands.

Equation (2) also combines pitch and roll control deflections into a single metric to support the 

analysis of multi-axis maneuvers.

Pilot duty cycle is a measure of how frequently the pilot is making a significant adjustment to 

his or her input. Duty cycle is calculated by counting the number of peaks per second in both the 

pitch and roll control stick inputs. A peak is arbitrarily defined as a change in the signal 

magnitude greater than 0.5 percent of the inceptor’s total displacement range and in the 

direction opposite of the signal’s change identified in the previous time step. Two test pilots, 

referred to as Pilot A and Pilot B, participated in the IRAC adaptive controls experiment. Figure 

4 shows example results of the peak detection algorithm for the pitch stick inputs of Pilot A 

during a 2g air-to-air tracking maneuver with a simulated reduced pitch damping failure and the 

sMRAC controller.

Figure 4. Duty cycle peaks for Pilot A pitch inputs during 2g air-to-air tracking with a reduced 

pitch damping failure and sMRA.
11

This calculation of duty cycle diverges from the method proposed by Shepherd (ref. 29) in 

that it is a determination of the frequency with which the pilot reverses control direction, whereas 

Shepherd measures the percentage of duration of the maneuver during which the pilot’s 

compensation is non-steady. Application of the non-steady compensation technique to the IRAC 

flight data failed to provide sufficient distinction between the different controllers as the pilots, 

regardless of the control system, tended to almost continuously vary their control inputs.

Significant differences were seen, however, in the frequency of control motion reversals 

between controllers.

Figure 5 shows a plot of pilot aggressiveness versus duty cycle for two pilots during 2g 

air-to-air tracking maneuvers with a simulated reduced pitch damping failure. The metrics were 

computed for the IRAC non-adaptive NDI controller with no failures or damage, for the NDI 

controller with a simulated failure and for all three IRAC adaptive controllers with the same 

simulated failure. When no failure is present (Unfailed NDI), both Pilot A and Pilot B exhibit very 

similar workload metrics. Following the failure (NDI), with no adaptation, both pilots experience 

an increase in the aggressiveness and duty cycle of their control inputs, although the workload 

increase for Pilot B is significantly higher than that of Pilot A. The simplest adaptive controller

(sMRAC) restores the duty cycle of Pilot A to approximately the same value as that of the 

unfailed scenario. The two controllers with higher level of complexity (onMRAC, onMRAC+)

have almost no effect on duty cycle but reduce the aggressiveness of Pilot A, with the most 

complex controller (onMRAC+) displaying nearly the same workload metrics as the unfailed 

case.

Figure 5. Aggressiveness versus duty cycle metrics for Pilot A and Pilot B during 2g air-to-air 

tracking with a reduced pitch damping failure.

The workload of Pilot B is also reduced with adaptive control in the presence of the failure, 

but in this case both aggressiveness and duty cycle are affected. There is no significant 

difference between the three adaptive controllers on workload for Pilot B. It is interesting to note 

that even with adaptive control, the workload of Pilot B is much higher than in the unfailed 

scenario. One possible explanation is that despite the improvement in reference model tracking 
12

attributable to adaptive augmentation, the gain of Pilot B remains elevated from his initial 

response to the failure. Consequently, even after the adaptive controller restores good pitch 

damping, Pilot B continues to operate at a higher workload than is necessary. This situation is a

case where information provided to the pilot on the status of the adaptive controller might prove 

beneficial.

Nuisance Rating Scale

A nuisance rating scale was developed as part of the Autonomous Collision Avoidance 

Technology (ACAT) project. The nuisance scale, shown in figure 6, is designed to categorize 

the nuisance effects of an autonomous system using a format similar to that of the 

Cooper-Harper rating scale for handling qualities (ref. 28) and the PIO rating scale (ref. 30). An 

adaptive flight control system, while restoring good aircraft handling qualities in the event of a 

failure or damage, may exhibit undesirable side effects ranging from annoying to unsafe.

Application of the nuisance rating scale provides a consistent, systematic framework for the

classification of undesirable side effects caused by adaptive flight control systems.

Figure 6. Nuisance rating scale for autonomous systems on piloted aircraft.
13

During the MRAC research flights, the nuisance scale was initially applied during tests of the 

adaptive controllers with no simulated failures present. The absence of simulated failures 

allowed for a clearer identification of any nuisance characteristics inherent in the controllers.

Pilot A was the only pilot to apply the nuisance scale to all four controllers with no failures during 

the 2g air-to-air tracking task. Pilot A gave the NDI controller a nuisance rating of 1, which 

served as a baseline for evaluation of the adaptive controllers. The onMRAC+ controller was 

rated as a 4 on the nuisance scale by Pilot A due to objectionable characteristics in the pitch 

axis. The following are excerpts from Pilot A’s comments on 2g air-to-air tracking with the 

onMRAC+ controller and no simulated failures:

A lot more bobble in gross acquisition than I would have expected . . . had to back out of 

the loop a little bit when doing the gross acquisition, let everything kind of die out; it dies 

out relatively quickly, but little bit bigger oscillations than I would have thought . . . might 

have been a little sensitive in pitch for the gross acquisition and that may have been what 

caused some of the stuff.

As described in reference 8, an adverse interaction occurred during this maneuver between 

the pilot and the onMRAC+ controller involving the  adaptive parameter. A minor PIO

resulted, for which a simple corrective adaptive parameter update law modification term has 

subsequently been proposed (ref. 8). The time-varying pitch sensitivity of this controller was 

also revealed in the Cooper-Harper handling qualities ratings and PIO rating scale evaluations

by Pilot A, making the nuisance scale a consistent metric, but in this particular case potentially 

redundant. Although no qualities were discovered for any of the MRAC controllers through the 

use of the nuisance scale that weren’t also revealed by the other rating systems, this may be 

more a function of the nature of the controllers’ behavior than a weakness of the nuisance scale 

itself.

Summary

Realization of the objectives for the IRAC experiment necessitated the development of 

non-traditional evaluation metrics. A complexity metric was created for measuring progress 

toward a simple yet effective adaptive control method. Control system complexity metrics can 

be useful when selecting a design method, or when making modifications to an existing design, 

in order to better account for the lifecycle impacts of complicating controller structures or 

features. However, significant work remains in determining, for the complexity metric presented 

here, the appropriate term weightings and potential applicability of higher-order term 

combinations.

Application of the complexity metric to the three IRAC adaptive controllers confirmed the 

expected trend of increased complexity with incremental design enhancements. A significant 

reduction in complexity was demonstrated between all three IRAC designs relative to an earlier

similar adaptive controller, the IFCS Gen-2A, validating the IRAC project objective of achieving 

simplified adaptive controls.

Pilot workload metrics were developed and refined to better understand the interactions 

between the pilot and the adaptive controller. Workload metrics can help the control system 

designer evaluate the impact of control system performance changes, and better understand 

pilot handling qualities evaluations. Pilot workload metrics for aggressiveness and duty cycle in 

the presence of a simulated aircraft failure were computed and cross-plotted for flight data from 

three IRAC adaptive controllers and a non-adaptive baseline controller, as well as for the 

baseline controller with no simulated failure. The metrics indicated that the adaptive controllers 
14

reduced the pilot workload under a failure scenario, but the degree of reduction was different 

between the two pilots.

A novel nuisance scale metric was applied to adaptive controllers with no simulated aircraft 

failures or damage. The metric successfully identified a low-level pilot-in-the-loop oscillation as a 

nuisance. The results warrant further application and evaluation of the nuisance scale as a 

pilot-interaction metric for adaptive controllers.
15

References

1. Whitaker, H. P., “Massachusetts Institute of Technology Presentation,” Proceedings of the 

Self Adaptive Flight Control Systems Symposium, Wright Air Development Center, 

January 13-14, 1959, pp. 58-78.

2. Schuck, H. O., “Honeywell’s History and Philosophy in the Adaptive Control Field,” 

Proceedings of the Self Adaptive Flight Control Systems Symposium, Wright Air 

Development Center, January 13-14, 1959, pp. 123-145.

3. National Aeronautics and Space Administration Flight Research Center Staff, “Experience 

with the X-15 Adaptive Flight Control System,” NASA TN D-6208, 1971.

4. Thompson, Milton O., and James R. Welsh, “Flight Test Experience with Adaptive Control 

Systems,” Advanced Control System Concepts, AGARD CP No. 58, AGARD Joint 

Symposium of the Guidance and Control Panel, and the Flight Mechanics Panel of 

AGARD, Oslo Norway, September 3-5, 1968, pp. 141-147, 1970.

5. Monaco, J., D. Ward, R. Barron, and R. Bird, “Implementation and Flight Test Assessment 

of an Adaptive, reconfigurable Flight Control System,” AIAA 97-3738, 1997.

6. Bosworth, John T., and Peggy S. Williams-Hayes, “Stabilator Failure Adaptation from 

Flight Tests of NF-15B Intelligent Flight Control System,” Journal of Aerospace 

Computing, Information, and Communication, Vol. 6, March, 2009, pp. 187-206.

7. Burken, John J., Curtis E. Hanson, James A. Lee, and John T. Kaneshige, “Flight Test 

Comparison of Different Adaptive Augmentations to Fault Tolerant Control Laws for a 

Modified F-15 Aircraft,” AIAA 2009-2056, 2009.

8. Hanson, Curt, Jacob Schaefer, John J. Burken, Marcus Johnson, and Nhan Nguyen, 

“Handling Qualities Evaluations of Low Complexity Model Reference Adaptive Controllers 

for Reduced Pitch and Roll Damping Scenarios,” AIAA 2011-6607, 2011.

9. Schaefer, Jacob, Curt Hanson, Marcus A. Johnson, and Nhan Nguyen, “Handling 

Qualities of Model Reference Adaptive Controllers with Varying Complexity for Pitch-Roll 

Coupled Failures,” AIAA 2011-6453, 2011.

10. Krishnakumar, Kalmanje, “A Request for Information in Flight Validation of Adaptive 

Control to Prevent Loss-of-Control Events,” Solicitation Number: NNH09ZEA002L, 

National Aeronautics and Space Administration Headquarters Office of Procurement, April 

2009.

11. Ioannou, Petros A., and Jing Sun, Robust Adaptive Control, Prentice Hall Inc., Upper 

Saddle River, New Jersey, 1996.

12. Younossi, Obaid, David E. Stem, Mark A. Lorell, and Frances M. Lussier, “Lessons 

Learned from the F/A-22 and F/A-18E/F Development Programs,” RAND Corporation, 

2005, pp. 39-44.

13. Decadal Survey of Civil Aeronautics: Foundation for the Future, Steering Committee for 

the Decadal Survey of Civil Aeronautics, National Research Council, 2006.
16

14. “Final Report on the Accident on 1st June 2009 to the Airbus A330-203 registered 

F-GZCP operated by Air France flight AF 447 Rio de Janeiro–Paris,” BEA, July 2012, 

pp. 199-201.

15. Harris, Jeffrey Jay, and G. Thomas Black, “F-22 Control Law Development and Flying 

Qualities,” AIAA 96-3379, 1996.

16. National Transportation Safety Board, “Aircraft Accident Report: Aborted Takeoff Shortly 

After Liftoff, Trans World Airlines Flight 843 Lockheed L-1011, N11002, John F. Kennedy 

International Airport, Jamaica, New York, July 30, 1992,” NTSB/AAR-93/04, 

March 31, 1993.

17. National Transportation Safety Board, “Aircraft Accident Brief DCA98MA023, American 

Airlines 1340,” NTSB/AAB-01/01, May 14, 2001.

18. Stepanyan, Vahram, Kalmanje Krishnakumar, Nhan Nguyen, and Luarens Van Eykeren, 

“Stability and Performance Metrics for Adaptive Flight Control,” AIAA 2009-5965, 2009.

19. Weyuker, Elaine J., “Evaluating Software Complexity Measures,” IEEE Transactions on 

Software Engineering, Vol. 14, No. 9, September 1988.

20. Halstead, Maurice H., Elements of Software Science. Elsevier North-Holland, Inc., 

New York, 1977.

21. McCabe, Thomas, J., “A Complexity Measure,” IEEE Transactions on Software 

Engineering, Vol. SE-2, No. 4, December 1976.

22. Merlin, Peter W., Gregg A. Bendrick, and Dwight A. Holland, “Breaking the Mishap Chain,” 

NASA SP-2011-594, 2011, pp. 3-21.

23. NESC Review of Demonstration of Autonomous Rendezvous Technology (DART) Mission 

Mishap Investigation Board Review (MIB), RP-06-119, December 2006.

24. SOHO Mission Interruption Joint NASA/ESA Investigation Board Final Report, August 31, 

1998. http://umbra.nascom.nasa.gov/soho/SOHO_final_report.html, accessed January 9, 

2013.

25. National Transportation Safety Board, “Autopilots on Boeing 727 Aircraft and Other 

Aircraft,” Joint Flight Standards Information Bulletin for Air Transportation, General 

Aviation, and Airworthiness, NTSB A-00-43/45, June 13, 2001.

26. Hanson, Curt, Jacob Schaefer, Marcus Johnson, and Nhan Nguyen, “Design of Low 

Complexity Model Reference Adaptive Controllers,” NASA/TM-2012-215972, May 2012.

27. Hageman, Jacob J., Mark S. Smith, and Susan Stachowiak, “Integration of Online 

Parameter Identification and Neural Network for In-Flight Adaptive Control,” 

NASA/TM-2003-212028, October 2003.

28. Cooper, George E., and Robert P. Harper Jr., “The Use of Pilot Rating in the Evaluation of 

Aircraft Handling Qualities,” NASA TN D-5153, April 1969.
17

29. Shepherd, Michael J., Adam MacDonald, William R. Gray III, and Richard G. Cobb, 

“Limited Simulator Aircraft Handling Qualities Evaluation of an Adaptive Controller,” IEEE 

Aerospace Conference Paper 1292, March 7-14, 2009.

30. U.S. Department of Defense, Flying Qualities of Piloted Aircraft, MIL-HDBK-1797, 

December 19, 1997.