Technology Readiness Levels and Maturation Approaches

**for ISHM Technologies**

*Ryan Mackey, Advanced Computing & ISHM Technologies Group,*

*Jet Propulsion Laboratory*

**Abstract**

Integrated System Health Management is a necessary component for NASA's
new near-term goals. New technologies are needed to improve safety,
reliability, affordability, and ultimately capability of upcoming and
increasingly complicated exploration systems. Many of these new
technologies are needed in the short term, and must be matured from
concept to space-proven applications within five to ten years.
Furthermore, the technologies themselves are often inherently complex
and difficult to validate. This paper focuses on these issues of ISHM
development, with the aim to facilitate technology maturation and
certification.

Much of the difficulty arises because ISHM technologies are commonly
associated with complex software and modeling components. To understand
the challenges in more detail, we begin by considering equivalent
efforts to bound and understand software and information technologies,
and expand these conventions to include the broader scope of ISHM.
Previous studies of TRL for autonomy-related information technology
provide us a solid foundation to evaluate ISHM maturity.

Another issue hindering maturation is that rigorous testing of ISHM
generally involves rare situations that are risky or impossible to
simulate. One required function of ISHM in space applications is
automatic crew escape in case of catastrophic vehicle failure, a
situation that is difficult to test. Virtually all ISHM functions are
triggered by failures, events that are random, unusual, and often poorly
understood if they are known at all. The expense and time delay
associated with spaceflight makes in-space validation of ISHM
technologies a daunting prospect. However, as we will argue, a careful
analysis of the true ISHM maturation needs substantially reduces
dependence on in-space testing, permitting cost-effective alternatives
in many cases.

We will conclude this paper with a brief example, drawing from our
efforts to validate two specific ISHM technologies, both in the mid-TRL
range, using an F-18 aircraft as a test platform. In this discussion, we
will illustrate the specific challenges of ISHM maturation, where these
needs were met by our aircraft testbed, where further validation effort
remains, and how we could adapt our testbed to address as many
validation needs as possible. We will also discuss how other facilities
and other testbeds associated with the F-18 were leveraged to simplify
and improve ISHM validation. Finally, we will compare our experience to
other categories of ISHM functions, in an effort to propose optimal
means of technology maturation on an individual basis.

**Introduction**

Integrated System Health Management is a blanket of technologies that
covers any approach to sense, interpret, predict, correct, and optimize
the health of any given physical system. Without going through a
complete enumeration of ISHM components, let it suffice to say that ISHM
is an emerging discipline, and new functions are still being added.
There have only been a few successful applications in aerospace, with
considerable technology maturation required. There is also a new and
significant focus on ISHM for spaceflight as a means to improve system
safety and, by consequence, overall mission capability.

Our recent experience in NASA missions -- both successful and
unsuccessful -- describes the need for new ISHM technologies, as well as
improvement in related fields of software, modeling, and methods of
certification. The cost, time, and difficulty inherent in returning the
Space Shuttle to flight, responding to fault modes that went
unappreciated for two decades of service, highlight the importance and
the complexity of ISHM. This need is acknowledged in new programs, such
as the CEV, where ISHM is listed as a critical need for mission safety
and assurance.

While most operational ISHM developments within NASA have been of a
retrofitting nature, gradually improving monitoring capabilities within
existing systems rather than implementing a systems-level capability,
there have been isolated episodes of ground-breaking ISHM development.
Two well-known examples are the Remote Agent Experiment (RA/X) (Nayak
1999), flown on the Deep Space One (DS-1) spacecraft in 1999, followed
by the Autonomous Sciencecraft Experiment (ASE) on Earth Observing One
(EO-1) in 2004 (Chien 2004). These experiments demonstrated the
feasibility of on-board reasoning technologies to handle mission
execution as well as mission replanning, both functions that are central
to ISHM. Another example is the upcoming Shuttle Advanced Health
Monitoring System (AHMS) (Jue 2002) (Fiorucci 2000), adding new
monitoring capabilities to the Space Shuttle Main Engines (SSMEs),
including for the first time detection and correction of sensor data
inside the control loop. In the aircraft industry, the Boeing 777
(Ramohalli 1992) and its on-board model-based diagnostic reasoner have a
long and successful history. More recently, the Boeing C-17 transport
includes a pair of on-board and ground-based reasoners, and the Lockheed
Martin Joint Strike Fighter (JSF) is the first aircraft to attempt a
thorough application of prognostic technologies.

Given the apparent interest, value, and body of research in ISHM, it is
remarkable that ISHM technology remains at a relatively immature level.
A closer analysis of the examples listed above reveals that, in many
cases, maturation of ISHM is unusually difficult. For example, the
estimated cost to certify the Shuttle AHMS is in the realm of \$100
million, and the C-17 health monitoring performance falls well below its
requirements in terms of false alarms. The Remote Agent Experiment has
yet to be adapted to any "normal" spacecraft, despite being demonstrated
six years prior to the time of writing. For these reasons, ISHM
technology appears to have lagged progress in other aerospace sciences,
leaving us with significant unanswered questions about the capabilities
of next-generation spacecraft.

In the interest of overcoming these difficulties, we will re-examine
technology readiness for ISHM, attempting to expose and understand the
special issues germane to ISHM development. We will outline how the
Technology Readiness Levels differ for ISHM as opposed to other
technology categories. Finally, we will consider an example of an
alternate ISHM development path, one currently in use by the author, as
a means to illuminate the special needs of ISHM and to meet them in a
repeatable fashion.

**ISHM Overview**

To begin, let us briefly consider what is required and what is typical
of ISHM, and therefore what abilities ISHM technology must demonstrate
in the course of its development. ISHM differs from other technologies
in several important aspects, all of which bear upon the process of
technology maturation.

ISHM exists to treat or prevent unusual, unexpected, and hazardous
behavior of systems. Because of this fact, ISHM needs are always
intertwined with the system to which it is applied. This also has strong
implications on the way ISHM performance is evaluated. In the ideal
case, a system that functioned perfectly would have no need of ISHM.
More realistically, requirements placed upon ISHM -- reliability,
false-alarm rate, latency, and so on -- depend on the frequency,
severity, and consequences of the failure modes it must manage. In
aerospace, one frequently hears of ISHM "buying its way" onto the
system, meaning that ISHM must demonstrably be the best solution to a
given problem. Since ISHM tends to be highly complex, it must justify
itself with substantial improvements to system performance.

By necessity, ISHM must be a system-level implementation. There are
numerous ISHM functions that for our purposes can be broadly classified
into *detecting*, *understanding*, and *reacting* to abnormal
conditions. While a significant fraction of ISHM is hardware,
specifically sensors and computing hardware at a minimum, many needed
ISHM technologies are software technologies. Furthermore, with few
exceptions, the sensors and computing hardware required for ISHM are
identical to sensors and computers needed for control. The exceptions
include sensors that are only used by ISHM, such as strain gages or wear
detectors. However, unlike the hardware, much of the ISHM software is
ISHM specific. This emphasis on software drives technology maturation to
a large degree.

The "understanding" components of ISHM are generally referred to as
*reasoners*, meaning software that interprets information to hypothesize
about the state of the system. Reasoners, and many other ISHM components
besides, are dependent upon models. These models are generally separable
from the reasoners themselves, but are often at least roughly equal in
complexity to the reasoner algorithms. While the model may not need
"development" as it is usually not a distinct technology, models do
require certification.

Any discussion of ISHM technology also invariably leads towards
automation and autonomy. The majority of ISHM functions can be described
as either providing some form of active response to off-nominal
conditions, or partially or fully replacing the judgment of human
pilots, maintainers, or controllers. Either of these processes is a step
towards automation by definition. Autonomy brings with it special
challenges, because in general, tasks that are traditionally performed
by people are difficult to define in rigorous terms. Just as a
conventional "autopilot" is in no manner equivalent to an actual pilot,
ISHM technologies that replace functions such as inspections and
certification for flight require a substantial leap in capability and
complexity.

All of these factors germane to ISHM contribute to special difficulties
in maturation. In the next section, we will review the standard ladder
of Technology Readiness Levels as a starting reference, and then modify
this view to accommodate ISHM technologies.

**Review: Technology Readiness Levels**

Technology Readiness Levels (TRLs) are a commonly used yardstick of
technology maturity, which is understood to be a separate concept from
the maturity of any specific application. TRL is an abstract scale,
arbitrarily chosen to range between 1 and 9, that reflects the
development high-water mark of a given technology. An example of NASA
TRL guidelines (Mankins 1995) is given below in Figure 1.

Since comprehensive ISHM integrated at the system level has never been
demonstrated in space, with the exceptions of only a select few
technologies and functions (such as fault protection in deep space
probes), we are most concerned with the intermediate TRLs, between about
TRL 3 and 7. This includes the range of technology development after the
formulation of the method, from first laboratory experiments and
prototypes, up to and including first flight validation.

There are a few focal points contained within the TRL guidelines that
merit special consideration. The most significant element is the
*relevant environment*, in which technologies must be tested to reach
TRL 5. A relevant environment is a carefully arranged laboratory
environment that accurately simulates the key difficulties and stresses
of operation in space. It must sufficiently stress the technology to
permit an accurate, defensible estimate of how the technology will
perform in its intended application. Because of this definition, the
relevant environment differs and is entirely dependent on the technology
itself. For the special case of ISHM, we can improve this loose
definition with some specific guidelines.

Two other important points are the precise meaning of *validation*, and
consequently the *performance model* of the technology. Both terms
appear at TRL 4, the first stage at which a prototype of the technology
must exist, and must be demonstrated to function correctly. Validation,
in the context of technological maturity, means obtaining sufficient
knowledge of a technology to predict its behavior over a significant
range of operating conditions and inputs. The "performance model" is the
embodiment of this prediction, meaning performance can be reduced to a
formula based on one or more inputs. As TRL advances from 4 through 7,
the performance model is expanded and proven on increasingly large
"envelopes" of inputs, from the laboratory through actual flight
environments. Without this model, without defining boundaries for which
the model is usable (i.e. "valid"), and without demonstrating the
model's accuracy within those boundaries, a technology cannot be said to
have been validated.

![](media/image1.wmf){width="7.0in" height="5.25in"}Questions of what
constitutes the relevant environment, and how to describe a performance
model, are significantly easier for hardware technologies. Specifically,
the relevant environment is defined in terms of only a few tangible
dimensions where physical devices are concerned -- temperature,
vibration, radiation, and so forth -- whereas software-enabled
technology invites a host of new environmental criteria, such as data
interchange and operating system. We should point out that nearly all
space technologies have some software component, and few technologies
can be unambiguously classified as hardware or software only.

Many ISHM technologies are largely or completely software-based, or more
accurately can be described as information technologies (IT), though the
full spectrum of ISHM includes both extremes. For example, a new type of
sensor without on-board processing would be strictly hardware, while an
algorithm to correlate faults using a Computer-Aided Design schematic of
the system would be strictly information technology. The bulk of ISHM
technologies lie somewhere in between, with considerable emphasis on IT.

For this reason, we will also examining special issues for IT technology
maturation. In (Mackey 2003), a revised list of TRL guidelines, centered
on the all-important 3-7 range, is as follows:

*TRL 1* **--** Identified/invented and documented a useful information
technology with a qualitative estimate of expected benefit. Basic
functional relationships of a potential application formulated and shown
to be compatible with reasonable processing constraints.

*TRL 2* **--** Completed a breakdown of information technology into its
underlying components and analyzed requirements and interactions with
other systems. Defined and documented the relevant IT execution
environment. Preliminary design assessment confirmed compatibility with
the expected IT environment.

*TRL 3* -- Key components of IT prototyped to prove scientific
feasibility. Successful preliminary tests of critical functions
demonstrated and documented. Experiments with small representative data
sets conducted. IT development environment and development tools
required to complete prototype defined and documented.

*TRL 4* -- Prototype completed on laboratory hardware and tested in a
realistic environment simulation. Experiments conducted with full-scale
problems or data sets in a laboratory environment and results of tests
documented. IT development environment completed as needed for the
prototype. A model of IT performance, adequate for prediction of
performance in the intended space environment, must be documented as a
result of these tests.

*TRL 5* -- Prototype refined into a system and tested on simulated or
flight-equivalent hardware. Interaction environment, including
interfaces to other systems, defined and included in the testing
environment. Rigorous stress testing completed in multiple realistic
environments and documented. Performance of the IT in the relevant
environment must be documented and shown to be consistent with its
performance model.

*TRL 6* **--** System ported from breadboard hardware testbeds to flight
hardware and tested with other systems in realistic simulated
environment scenarios. IT tested in complete relevant execution
environment. Engineering feasibility fully demonstrated.

*TRL 7* **--** Information technology validated in space. Adequate
documentation prepared for transfer from developers to full operations
engineering process team.

*TRL 8* -- Development environment completed and validated. Approved by
an independent verification and validation (IV+V) team.

*TRL 9* **--** Documentation of the information technology completed,
approved and issued.

Operational limits of the software are understood, documented and
consistent with the operational mission requirements

As we have noted above, development for information technologies
requires careful thought about the relevant environment and the
performance model. ISHM is similar in this regard, although there are a
few additional factors unique to ISHM that must be addressed. We will
investigate these issues in the following section.

**Difficulties of ISHM**

There is a fundamental difficulty with any ISHM technology, whether it
is hardware or otherwise, regardless of function or relevant
environment. This difficulty is the simple fact that ISHM is intended to
deal with system failures, situations that are typically unusual and
unwanted, as well as expensive and dangerous. As a result, ISHM as a
class requires a rethink of technology maturation practices.

In an ideal ISHM maturation effort, we would begin with a complete
understanding of all anticipated states of the system, both nominal and
faulty. We would be able to describe these states in terms of measurable
quantities and environmental conditions that lead to such effects. We
would have past examples of each of these states for laboratory testing
of our technologies, followed by a full-scale series of tests. We would
then be able to examine the technology's performance in the situations
for which it was intended.

In general, few of these desires can be fully met. A list of specific
testing difficulties is given below:

1.  *Enumerability of States.* Since the relevant environment for ISHM
    generally has many dimensions, it becomes difficult or impossible to
    test technologies in every plausible set of conditions. This applies
    both to the number of states defining test cases, and the ISHM
    technology or architecture's internal representation of states. For
    example, a monitoring technology designed to sense faults in a
    system with ten different known failure modes cannot usually be
    tested for every combination of faults, since there are over 1,000
    combinations.

2.  *Defining Nominal.* Implicit in the assumption that one can define
    failures is the converse, namely that a well-defined nominal state
    exists. The nominal state may be difficult to establish, due to
    manufacturing variation between identical spacecraft and components,
    effects of normal wear and usage, and other long-term effects of the
    environment. There may be uniqueness due to specific payloads,
    configurations, or interfaces with launch vehicles. Establishing
    nominal is particularly difficult if technology and the intended
    application are developed concurrently, as is often the case for
    space systems. Many properties of nominal operation are only evident
    after a long period of operational experience.

3.  *Inability to perform full-scale test of faults.* Certification of a
    fault-reactive system -- not to mention operator confidence in the
    system -- requires unit testing as well as full-scale tests of the
    assembled system. This is not practical due to the danger and
    expense. Even component testing of actual faults is difficult,
    particularly in such areas as propulsion and structures. It is also
    assumed that the ISHM system will improve with flight experience,
    implying that ISHM is not fully complete, mature, or tested before
    the first flight.

4.  *Access to data and implementation schedule.* The ISHM system cannot
    be built without a thorough understanding of system behavior. Needed
    domain knowledge includes test data and engineering models wherever
    possible. If ISHM is developed concurrently, it will necessarily
    evolve as system trades are made and more detailed description
    becomes available. These data requirements are much more stringent
    than for most other system components. High quality data are rarely
    available without a determined data capture effort.

5.  *Certification of algorithms vs. models.* Many ISHM algorithms use
    embedded models, and these models are often updated to include new
    information. This raises the important question of how to certify
    ISHM, as performance is dependent upon both the algorithm and the
    model. In many cases, the model is more difficult to create than the
    algorithm itself.

In legacy systems, designers have reacted to these difficulties by
keeping ISHM out of the safety-critical path. To date, in all aircraft
examples, ISHM has been entirely decoupled from flight controls, only
affecting maintenance cycles and other ground-based activities. In
spaceflight, only straightforward ISHM functions (often intricate, but
lacking any detailed reasoning) have been embedded in control systems,
except where absolutely necessary to achieve mission needs, such as
on-board "behaviors" governing locomotion of the Mars Exploration
Rovers. We must therefore find an alternate solution to these problems
if ISHM is to have a meaningful impact.

**Approaches to Mitigate Problems of ISHM**

While the difficulties listed above seem nearly intractable, it is
important to keep in mind that the difficulty may be greatly reduced if
an elegant system is chosen for development. From the standpoint of TRL,
we need only perform space validation *in the relevant environment.* We
require only a single example that is suitably detailed and relevant to
the broader class of space missions. It is therefore reasonable to
assume that we can find a case where these problems are not so severe.

There are several promising approaches to solve the problems listed
above. We will list a few of the more important ones below:

1.  *Design for data capture.* Few programs currently realize the
    long-term value of system data, including that gathered during
    assembly and initial tests, but such information is invaluable to
    ISHM. Additional sensors are rarely required; a program must only
    perform the incremental effort of collecting and managing data
    through all phases of construction, and checkout. Furthermore, past
    experience in aircraft shows that ISHM can help find and correct
    errors during vehicle assembly and test, offsetting the additional
    cost.

2.  *Prototype ISHM using mission or vehicle analogues*. In general, a
    meaningful surrogate system can be found that has higher re-flight
    rates and greater flexibility in terms of fault injection and
    mission variation than the target application. Mission analogues
    still are capable of adding "flight realism" that may not be present
    in other simulations.

3.  *Work towards more realistic interpretations of flight-critical
    software specifications.* Since not every fault can be anticipated,
    let alone invoked for testing purposes, it is not practical to
    insist that ISHM software be tested for every possible combination
    of input conditions, as is the standard practice for flight-critical
    code. A more realistic approach would be to independently verify
    that the models used are correct and operable using iterative
    "pathfinder" techniques, and that the ISHM algorithms will not
    inject dangerous inputs to the control system whether or not a
    correct model is loaded. Highly critical, reflexive ISHM, such as
    emergency escape mechanisms, should be separable and testable
    independently as well.

4.  *Take maximum advantage of autocoding techniques and model-based
    engineering (MBE) approaches.* Since ISHM is a reasoning process and
    is therefore model-dependent, accuracy, completeness, and
    interoperability of models are paramount. Autocoding, automatic
    model abstraction, and model checking will greatly reduce the effort
    needed to certify models and the overall ISHM system. This benefit
    accumulates as models are improved over the system's lifecycle.

5.  *Construct and maintain a centralized "meta-environment" combining
    system, subsystem, and component models into a unified simulation.*
    A unified simulation provides three primary benefits for our
    purposes: First, it maintains data and domain knowledge. Second, it
    serves as a platform to test and certify ISHM technologies. Third,
    it provides a rigorous description of "Relevant Environments" for
    different ISHM components, by encapsulating the various interfaces
    and operating conditions found in different parts of the system.
    This closes the circle of system knowledge that began with data
    gathering above.

The five approaches above are not an exhaustive list, but do cover the
major difficulties that face ISHM. In brief, Table 1 below describes
specific responses to the various difficulties.

  ------------ ------------- ------------ ----------------- -------------- --------------------
               Design for    Analogue     Negotiated        Autocoding and Meta-Environmental
               Data Capture  Missions     Flight-Critical   MBE            Models
                                          Requirements                     

  Difficulty   Gradual       Full-scale   More realistic    Organize unit  Provide end-to-end
  of Full      testing as    tests of     testing           testing        simulation at
  Scale        system is     equivalent   requirements                     various levels of
  Testing      assembled     system                                        detail

  Poor Access  Capture data  Access to                                     Playback and
  to Data      and domain    extensive                                     integration of
               knowledge     data of                                       captured data,
               during        analogous                                     synthesis with
               assembly and  system                                        simulated data
               test                                                        

  Imprecise    Mechanism to  Realistic                                     Collect known
  Definition   gather        flight                                        variation within
  of Nominal   nominal data  testing,                                      nominal
               from system   including                                     
               as it is      nominal                                       
               built         variation                                     

  Algorithms   Test models   Test ability Permit            Automated      
  vs. Models   against       to generate  certification of  generation and 
               actual data   and include  models without    checking of    
               as system is  model        recertifying      models         
               constructed   updates      entire ISHM                      

  State                                   Permit separation Allow optimal  Provide and maintain
  Explosion                               of ISHM to allow  model testing  a meaningful
                                          abstraction,      using          abstraction of state
                                          independent       exhaustive or  space
                                          certification     branch/bound   
                                                            methods        
  ------------ ------------- ------------ ----------------- -------------- --------------------

Table 1: Matching Approaches to Maturation Difficulties

**TRL for ISHM**

Having examined the difficulties posed by ISHM as well as some
solutions, we now revisit the notion of Technology Readiness Levels. The
TRLs are intended as a guideline for technology development, and should
explicitly reference the most difficult achievements at every stage. The
slight modifications below are intended to highlight issues of
particular importance to ISHM technologies.

> *TRL 1* **--** Identified/invented and documented a useful ISHM
> technology with a qualitative estimate of expected benefit. Basic
> functional relationships of a potential application formulated *and
> shown to be compatible with reasonable ISHM architectures and testing
> requirements*.
>
> *TRL 2* **--** Completed a breakdown of the ISHM technology into its
> underlying functions and components, and analyzed requirements and
> interactions with other systems. Defined and documented requirements
> for operation, interfaces, and relevant mission phases. *Preliminary
> design assessment confirmed compatibility with the expected ISHM
> architecture*.
>
> *TRL 3* -- Major functions of ISHM technology prototyped to prove
> scientific feasibility. Successful preliminary tests of critical
> functions demonstrated and documented, leading to a preliminary
> performance estimate. Experiments with small representative data sets
> conducted. *Execution environment and development tools required to
> conduct these tests, such as modeling tools, defined and documented*.

Aside: The key additional component in early TRL is an understanding of
the ISHM architecture for which the technology is intended. This
architecture defines inputs, outputs, issues of timing, and performance
requirements of the technology, and is thus an integral part of the
relevant environment.

> *TRL 4* -- Prototype completed on laboratory hardware and tested in a
> realistic environment simulation. Experiments conducted with
> full-scale problems or data sets in a laboratory environment and
> results of tests documented. *Development ISHM infrastructure
> completed as needed for the prototype*. A model of ISHM technology
> performance, adequate for prediction of performance in the intended
> application, must be documented as a result of these tests.
>
> *TRL 5* -- Prototype refined into a system and tested on simulated or
> flight-equivalent hardware. Interaction environment, including
> interfaces to other systems, defined and included in the testing
> environment. Rigorous stress testing completed in multiple realistic
> environments and documented. Performance of the technology in the
> relevant environment must be documented and shown to be consistent
> with its performance model.
>
> *TRL 6* **--** System ported from breadboard hardware testbeds to
> flight hardware and tested, along with all other needed components, in
> realistic simulated environment scenarios. ISHM technology tested in
> complete relevant execution environment. Engineering feasibility fully
> demonstrated.

Aside: The "relevant environment" in this case means a prototype or
skeleton ISHM architecture, conforming to the envisioned final
application. Sensors, computing hardware, message passing, etc. are all
defined by that architecture. Stress-testing, for purposes of ISHM,
means injection of faults -- either simulated or real -- that are
considered limiting cases, either in terms of sensitivity, timing, or
severity. Stress-testing also should include overlapping or concurrent
faults. As part of performance model validation, *all* technologies, not
just ISHM technologies, should be tested to failure and beyond.

> *TRL 7* **--** ISHM technology validated in space. Adequate
> documentation prepared for transfer from developers to full operations
> engineering process team.
>
> *TRL 8* -- Development environment completed and validated. Approved
> by an independent verification and validation (IV+V) team.
>
> *TRL 9* **--** Documentation of the ISHM technology completed,
> approved and issued.
>
> Operational limits of ISHM are understood, documented and consistent
> with the operational mission requirements.

**An Example: Ongoing ISHM Maturation on Aircraft**

In 2005, NASA-JPL, NASA-DFRC, and NASA-ARC jointly undertook an effort
to mature two specific ISHM technologies using a Dryden F-18 aircraft as
the carrier. The two technologies in question were BEAM (Beacon-based
Exception Analysis for Multimissions) (James 2001) and IMS (Inductive
Monitoring System) (Iverson 2004). Both are software technologies
designed to process raw sensor data and sense anomalies, including
system novelty and incipient faults. The F-18 aircraft served as a
spacecraft analogue. This project was intended to demonstrate TRL 6 for
both technologies. Results from this experiment will be published
shortly.

For a proper validation, the F-18 testbed must contain all necessary
elements of the relevant environment. These are as follows:

ISHM Architecture: Asynchronous or synchronous detection capability

> Must operate with no feedback from higher-level reasoners

Hardware: Performance sensors, environmental sensors, state variables

Real-time data sampling at rates of 10 Hz and up

Interfaces: IEEE-1553 databus

Computing hardware: Flight-qualified processor (Geode 686 at 300 MHz)

Operating system: Flight-like

Scenarios: Wide range of nominal conditions and simulated faults

Other flight issues: On-board power, expected sensor anomalies, pilot
override,

> computer hardware errors, model data errors

Development Environment: Automatic model generators and model testing
tools

This list of requirements is typical for ISHM validation. As should be
obvious, the requirements above can be met by an aircraft as easily as a
spacecraft, with the exception of the specific fault behavior.
Significantly, these ISHM algorithms can be expected to perform
identically in atmosphere or microgravity, provided the testbed
accurately describes a spacecraft computing and sensor environment. For
these two technologies, the only change between this F-18 test and a
spaceflight test would be in the models used by the algorithms. During
this validation experiment, we also exercised the development tools to
generate models from captured data, and included their performance as
part of the experiment.

To date, through six months of designing and conducting this experiment,
all of these validation needs have been met with two exceptions. The
first is the software operating system; Linux-RT and VX/Works were both
considered but not adopted due to cost limitations. The second is in the
scope of simulated faults, which has been limited to unusual pilot
inputs -- specifically, manual shutdown of engines -- and the few
unexpected events that have happened to occur during our test flights.
Both of these shortcomings can be easily solved. The first simply
requires purchasing and incorporating a proven spaceflight operating
system. We can address the second through more creative use of flight
controls, inclusion of failed (redundant) hardware onboard, or playback
of captured fault data over the 1553 bus during flight.

Using the F-18 as a surrogate system confers several dramatic
advantages. First, numerous flights are now available for testing -- 18
flights were performed in the space of a single month, and we were able
to effect several model updates and code changes between flights.
Second, this particular aircraft also comes with realistic simulators,
both pure software and hardware-in-the-loop, permitting rigorous
checkout of flight software prior to flight experiments. Third, the F-18
is far more amenable to modification and testing in degraded mode than
an actual spacecraft, such as our experiments with one of two engines
deliberately failed in flight. From the standpoint of ISHM performance,
the F-18 experiment allows us to estimate algorithm sensitivity,
latency, fault coverage, suitability to different sensor types, model
building requirements, processor and memory requirements, and adequacy
of the associated development tools. These are all of the ingredients
necessary to validate our "performance model" of ISHM technologies, and
therefore legitimately claim TRL 6.

This "surrogate spacecraft" approach, as it stands today, is sufficient
for many ISHM technologies -- fault detection, diagnostic reasoning,
prognostics, planners, etc. The major element missing is one of
feedback. In our present system, there is no direct route from the
experiment computer back to the flight controls. This is not surprising
given the austere nature of our experiment. However, given standard
interfaces and a suitable ISHM architecture, virtually every step of the
ISHM path except the "executive" component can be tested in this
configuration. Yet the aircraft paradigm is not limited in this regard.
Other aircraft experiments, such as the Intelligent Flight Control
System project, have included complete feedback. There are similar
opportunities available on UAVs, among others, where experimental
technology intrusion into active controls is of lesser concern.

**Conclusions**

We now close this discussion by recalling the original intent of
Technology Readiness Levels in general. The value of TRL is in the
ability to extrapolate whether or not a technology can be applied to any
given application, and estimate the effort required to do so, from
experiences with different applications. TRL is therefore particularly
important for emerging and complex technologies. ISHM has both of these
characteristics.

The central concept of TRL is in the performance model. As one develops
and matures a technology, it is essential to create, update, and verify
the performance model at every stage. In essence, this performance model
is the developer's "radius of confidence" in the technology, describing
how well one can predict technology performance *in a totally new
application,* given the characteristics of that application. The
technology developer should have a good understanding of the technology
limitations and should be able to describe them in the context of the
performance model. If the model is constructed with care, describes
operating conditions in a manner that can be understood by other
developers, has been thoroughly tested "at the corners" of performance
and shown to be consistent with anticipated performance, then it is a
simple matter for another developer to evaluate the technology's
suitability.

High TRL can also be described as "trust" in a technology's behavior --
specifically, trust that the technology will function as expected after
being handed from its inventors to an entirely different development
team, based on a comprehensive series of tests. The issue of trust takes
on an entirely new significance in the context of ISHM. Given the low
flight rates of spacecraft in the foreseeable future, virtually every
response of ISHM to a major fault mode is likely to be the first and
only time that fault is ever encountered. For this reason, trust is
essential, and at the same time difficult to earn.

For ISHM to be successful, and to have any real impact in system
performance, this trust has to be earned. We therefore must be aware of
the special difficulties pertaining to ISHM. We must understand the
details of TRL to make certain that advanced TRLs are justified. We must
also explore alternate forms of technology maturation. TRL is a
particularly important tool for ISHM, simply because no single project
will ever be able to afford the time or effort to exhaustively prove
ISHM. Developers must look to past performance on simpler systems to
build trust in ISHM.

NASA is now reaching for a new generation in space vehicles, cognizant
of the need for safety, reliability, and sustainability. ISHM is a key
component to reaching these goals, but the ISHM system must be ready --
on-board, functional, and operating with the confidence of mission
controllers on the first flight. This is not an impossible task. Careful
attention to the architecture, interfaces, and environment that governs
ISHM; thorough testing of ISHM components and end-to-end functions on
surrogate vehicles; systematic capture and usage of operational data as
components are developed and integrated; and coordinated engineering,
modeling, and simulation are all achievable and will be sufficient to
meet this challenge.

**References**

> P. Nayak, D. Bernard, *et. al.*, 1999. "Validating the DS1 Remote
> Agent Experiment." In *Proceedings of the 5^th^ International
> Symposium on Artificial Intelligence, Robotics and Automation in Space
> (iSAIRAS-99)*, June 1999.
>
> S. Chien, R. Sherwood, *et. al.*, 2004. "The EO-1 Autonomous Science
> Agent." In *Proceedings of the 2004 International Conference on
> Autonomous Agents and Multi-Agent Systems (AAMAS-2004)*, July 2004.
>
> F. Jue and F. Kuck, 2002. "Space Shuttle Main Engine (SSME) Options
> for the Future Shuttle." In *Proceedings of the 38^th^
> AIAA/ASME/SAE/ASEE Joint Propulsion Conference*, July 2002.
>
> T. Fiorucci, D. Lakin, T. Reynolds, 2000. "Advanced Engine Health
> Management Applications of the SSME Real-Time Vibration Monitoring
> System." In *Proceedings of the 36^th^ AIAA/ASME/SAE/ASEE Joint
> Propulsion Conference*, July 2000.
>
> G. Ramohalli, 1992. "The Honeywell On-board Diagnostic and Maintenance
> System for the Boeing 777." In *Proceedings of the 11^th^ IEEE/AIAA
> Digital Avionics Systems Conference*, October 1992.
>
> J. Mankins, 1995. "Technology Readiness Levels." NASA White Paper,
> April 1995.
>
> R. Mackey, R. Some, A. Aljabri, 2003. "Readiness Levels for Spacecraft
> Information Technologies." In *Proceedings of the 2003 IEEE Aerospace
> Conference*, March 2003.
>
> M. James, R. Mackey, H. Park, M. Zak,2001. "BEAM: Technology for
> Autonomous Self-Analysis." In *Proceedings of the 2001 IEEE Aerospace
> Conference*, March 2001.
>
> D. Iverson, 2004. "Inductive System Health Monitoring." In
> *Proceedings of the 2004 International Multiconference in Computer
> Science and Computer Engineering*, June 2004.