Human Factors of Integrated Systems Health Management on Next-Generation
Spacecraft

Robert S. McCann, Ph.D., & Lilly Spirkovska, Ph.D.

NASA Ames Research Center

Moffett Field, CA

# Abstract

A shuttle crew's ability to manage the health of the spacecraft systems
is compromised by the limited capabilities of the onboard health
management technologies, many of which date from the 1970s and 1980s.
Most notably, the Caution and Warning system does little more than
generate auditory alerts and fault messages in response to out-of-limit
sensor readings. Today's health management technologies have much more
extensive capabilities that range from detecting off-nominal trends and
data patterns to executing fault isolation and recovery procedures. On
next-generation spacecraft, these technologies could be harnessed to
replace the traditional Caution and Warning system with a decision and
action support system that assists the crew with all aspects of
real-time health management operations. We discuss several aspects of
the design of such a system, including human-machine functional
allocations; user interfaces to enable and support human-machine
interaction, and methodologies for testing and evaluating collaborative
operational concepts and associated user interfaces.

# Introduction

Sending people into space ranks among the riskiest and most challenging
endeavors of our time. Much of the challenge stems from the fact that
human-rated spacecraft consist of very complex and often highly
interconnected engineering systems, including propulsion systems;
electrical and mechanical power generation and distribution systems;
guidance, navigation, and control (GN&C) systems; data processing
systems; life support systems; and communications systems. Particularly
during the dynamic phases of a space mission, such as launch, ascent,
and entry, these systems must perform to precise operational
specifications in very harsh environments, whose cumulative effects on
system functioning are often poorly understood. As a result, systems
malfunctions are an ever-present threat to mission success and crew
safety.

To minimize this threat, a significant fraction of real-time mission
operations is devoted to monitoring, managing, and maintaining the
health of the onboard systems. Each system is heavily instrumented with
sensors that provide real-time numeric readings of critical operating
parameters, such as temperatures, pressures, accelerations, and flow
rates. If the values from any one of these sources move outside a range
consistent with normal systems operations, the "out-of-limits" condition
must be detected, the cause must be identified and, if the cause is
determined to be a system malfunction, the appropriate remedial actions
must be taken.

Figure 1 captures the series of events that accompany a systems
malfunction from the perspective of a crewmember in the shuttle cockpit
(for simplicity, the figure excludes ground-based activities and
crew-ground communications). The vehicle's Caution and Warning (C&W)
System, described in greater detail below, detects an off-nominal sensor
reading, sounds an alarm, and generates a flashing fault message on one
or more cockpit displays. The crewmember first silences the alarm by
pressing a master alarm button, and then stabilizes the fault message by
pressing the "acknowledge" key on the cockpit keyboard. He or she then
reads the fault message (or several messages if the problem has
"daughter" faults associated with it), and determines which message
pertains most closely to the root cause of the problem. Using the
selected message as an index, the crewmember then locates the
appropriate fault management procedures in a paper "flight data file"
(FDF) or cockpit cue card. These procedures typically take the form of
one or more manual switch throws that change the operating mode of the
system.

# 

# Complicating Factors

Although these activities may sound straightforward, each one is
impacted to greater or lesser degrees by a variety of complicating
factors. These factors include:

*Data availability*. The most basic requirement for effective fault
management is adequate situation awareness of systems mode and systems
functioning. Sensor data is the primary source of this awareness. On the
shuttles, however, only a small subset of the sensed data is included in
the data bus connecting the orbiter instrumentation system with the
cockpit. Furthermore, in the original version of the cockpit, much of
the data could only be viewed on one of three cathode-ray tube (CRT)
screens, meaning that an even smaller subset of the data was visible at
any one time. If a crewmember wanted to view all available data for a
particular system, he or she had to navigate through several successive
display formats. Starting in 2000, the CRT's on some of the orbiters
were replaced by liquid-crystal display (LCD) screens as part of a
"glass cockpit" upgrade. The cumbersome display navigation requirements
for accessing systems data were largely unaffected by the upgrade,
however.

*Cockpit display formats*. For the most part, the display formats
available for viewing on the new LCD screens are replicas of the
original display formats designed for the CRTs. These "legacy" formats
are often poorly organized and highly cluttered, taking the form of
closely spaced tables of monochromatic (green) alphanumerics. An example
is the "BFS GNC SYS SUM 1" display on the left side of Figure 2. The
format makes it difficult to localize and process key sources of
information, such as off-nominal sensor readings. More importantly, as
shown in the "zoomed-in" section in the lower left section of the
figure, digital values are a notoriously inefficient way to convey
information about systems mode and systems functioning. Crewmembers are
not so much interested in the actual values themselves as in what the
values convey about the operational status of the system ("Is the system
functioning nominally or off-nominally?") and its current configuration
(e.g., "Is Isolation Valve "A" on Leg "B" open or closed?"). To build up
such situation awareness, the "raw data" on the screen has to be
translated into a form that matches the crewmember's mental model of the
system architecture and possible configurations.

*C&W System Limitations.* By today's standards, the shuttles' data
processing system has only limited capabilities. During dynamic flight
phases, the general-purpose computers are almost fully occupied with
computations relating to vehicle guidance and flight control. The C&W
system software, also housed on these computers, performs simple
real-time comparisons between sensor readings and preset upper and lower
values. If a set number of successive readings fall outside of these
values, the out-of-limits condition is signaled to the crew by some
combination of auditory and visual alarm. In addition, each
out-of-limits condition is typically accompanied by a written fault
message to help the crew identify the source of the problem.

Limit-sensing software is computationally straightforward, making it
both robust and relatively easy to flight certify. Functionally,
limit-sensing software automates much of the information processing
associated with parameter monitoring and detection of off-nominal
readings. In a vehicle as complex as
![](media/image1.jpeg){width="1.6437007874015748in"
height="1.405511811023622in"}![](media/image2.pct){width="1.7665354330708662in"
height="1.3783464566929133in"}![](media/image3.pct){width="1.7665354330708662in"
height="6.181102362204725e-2in"}![](media/image1.jpeg){width="3.455511811023622in"
height="1.3in"}![](media/image4.pct){width="2.7456692913385825in"
height="1.3035433070866143in"}the shuttle, instrumented with over 2000
individual sensors, these activities would otherwise completely
overwhelm the crew. At the same time, however, the "single-sensor,
single-annunciation" interface with the software creates a serious human
factors problem. The original motivation behind the interface was to
maximize the crews' situation awareness by providing them with as much
information as possible about subsystem malfunctions and component
failures. However, because the software processes each data source
independently, the system has no ability to discriminate a legitimate
off-nominal reading from a spurious signal from either a failed sensor,
or a failure in a digital signal conditioner or other component of the
data processing system. More seriously, due to the complexity and
interconnectedness of shuttle subsystems, a failure in one component
frequently causes additional equipment and/or sensor failures, resulting
in a cascade of alarms and fault messages. In many cases, the
proliferation of alarms and messages makes it very difficult to identify
the "root cause" of the problem (McCandless, McCann, & Hilty, 2003).

A second drawback with the current C&W system is that, having generated
one or more fault messages, it provides no further assistance with the
fault management process. Once a crewmember has selected a fault
message, he or she must then navigate through a FDF to find the
appropriate checklist of fault management procedures. The checklist, a
series of specialized instructions, is written in the form of
specialized symbols, abbreviations, and spatial codes (i.e., line
indentations) that collectively require considerable training to decode
and understand. The instructions encompass activities such as reading
gauges, comparing values, and throwing cockpit switches to alter the
operating mode of the affected system. In some cases the procedures are
short and straightforward, as when they call for a switch from a failed
primary component, such as a pump or a fan, to a healthy backup. In
other cases, the instructions are considerably more complicated. For
example, sometimes the fault message is itself consistent with a variety
of "root cause" failures. The procedures associated with that message
may first designate switch throws to reconfigure the affected system
into a new operating mode that provides additional insight into the
underlying problem. Once the problem is more clearly understood,
additional switch throws may then be necessary to place the system into
a safe state, and return as much of the system as possible to its
nominal operational configuration. Importantly, these switch throws are
virtually all manual; the crew has to locate the appropriate switches
from the hundreds of switches in the cockpit, and then toggle them to
the commanded position. These activities are both time consuming and
attention-demanding, greatly disrupting crewmembers' nominal information
acquisition activities (Huemer, et al., 2005).

*Crew-Ground Interactions*. We noted earlier that only a small subset of
the sensed data could be displayed to the crew at any one time. In
addition, during dynamic flight phases, the crew must time-share systems
health management activities with numerous other critical
mission-management requirements, such as monitoring the vehicle's
attitude, velocity, and flight path. In many cases, the crew simply
cannot safely devote the time and resources necessary to perform the
health management activities, particularly when faced with multiple
alarms and fault messages, or when faced with a malfunction (or set of
malfunctions) not covered by existing procedures. To deal with these
circumstances, a much larger selection of sensor readings is included in
the near-real-time telemetry stream that flows from the vehicle to the
Launch and Mission Control Centers (MCC). Flight controllers and systems
subject-matter experts continuously monitor these data for anomalous
readings. Ground software flags out-of-limit parameters, sometimes using
tighter limits than those used onboard in order to detect faults more
quickly. Additionally, ground software can be upgraded more easily than
Shuttle software, and is thus closer to the state-of-the-art. The
software does, for example, provide trending information on some main
engine parameters in graphical form. Using these tools, ground
controllers will typically assist the crew in disambiguating the root
cause of a series of annunciations and fault messages, and also assist
with determining the root cause and appropriate procedures for novel
failure situations not covered in the flight data files. The crew must
still locate and flip the proper switches and verify that mitigation
strategies proceed as expected.

**Health Management on Next-Generation Vehicles**

NASA is designing a new Exploration Transportation System (ETS) to
replace the aging shuttles, return humans to the Moon, and enable human
exploration of more remote destinations. One of the most important
drivers for the design of ETS vehicles is that they will eventually have
to operate more autonomously than the shuttles. For example, once ETS
vehicles start to travel beyond low Earth orbit (LEO), speed-of-light
delays will combine with a lack of communications infrastructure to
eliminate real-time data links and voice communications with the ground.
Consequently, many critical health management operations (as well as
operations that rely on health management data) will have to be
performed without real-time ground assistance. Considering the large
role of ground-based resources in today's fault management activities,
this shift to more autonomous operations will require a considerable
augmentation in onboard health management capabilities.

Enhancing onboard health management capabilities of ETS vehicles is a
considerable challenge. As we have seen, managing the health of a
vehicle as complex as a human-rated spacecraft requires large amounts of
data-and-information processing resources. On the shuttles, neither the
human or machine resources are sufficient to meet these requirements.
With a maximum crew complement of four (at least for missions in the
foreseeable future), ETS vehicles will have no more "pairs of eyes"
onboard than the shuttles do. Furthermore, ETS vehicles are going to
resemble the capsules and modules of the Apollo era. If anything, the
visual real estate available for presenting systems health information
will be reduced compared to the shuttles.

*Current Machine Capabilities*. Fortunately, in the three decades since
the original shuttle design, the processing resources of computing
devices have increased dramatically. Software developers in industry,
government, and the applied artificial intelligence community have
exploited these increases to develop "end-to-end" health management and
process control systems with much more extensive capabilities than the
limit-sensing and fault annunciation software on the shuttles. Figure 3
depicts a representative example of today's "state of the art" in
end-to-end health management systems (Keller, Wiegand, Swearingen,
Reisig, Black, Gillis, & Vandernoot, 2001); note also that a
comprehensive (i.e., vehicle-level) health management architecture is
likely to consist of a network of such system-level managers, one for
each significant engineering system (and significant structures) on
![](media/image8.pct){width="2.7574803149606297in"
height="5.011023622047244in"}the vehicle (Atlas, Bloor, Brotherton,
Howard, Jaw, Kacprzynski, Karsai, Mackey, Mesick, Reuter, & Roemer,
1999). The architecture depicted in the figure assigns specific
computational functions and capabilities to distinct
data-and-information processing "layers." Starting from the bottom of
the figure, the Signal Processing Layer conducts various tests on
continuously varying sensor data to verify the validity of the data. The
Condition Monitoring Layer then applies data-monitoring and data-mining
algorithms in order to classify the time-varying data patterns (and, by
extension, the current functional mode of the system) as being
consistent with either nominal or off nominal operations. Then, in the
Health Assessment Layer, rule-based and/or model-based reasoners (MBR's)
make root-cause diagnoses of off-nominal patterns. Finally, the
Recovery/Safing Layer contains a reactive planner that 1) determines
what procedures are required in order to achieve the desired goal state
(typically, a return to the nominal operational mode), 2) determines the
correct sequence of actions to achieve that state, 3) physically
commands the actions, and 4) from sensor data, determines whether the
actions have been carried out and the desired goal-state achieved.

With ETS vehicles facing the same general limits on onboard human
processing resources as today's vehicles, any substantial enhancement in
onboard health management capabilities will clearly have to involve an
infusion of these advanced health management technologies. In the
following sections, we consider some of the design issues that will
accompany this infusion.

*Human-Machine Functional Allocation*. Modern "end-to-end" health
management systems have generally been designed to operate in
"stand-alone" (fully automated) environments, such as an unmanned
spacecraft. The health management software is wholly responsible for all
aspects of systems health management, including fault detection,
isolation, and recovery. On a crewed vehicle, with humans in the loop,
the question of what activities should be automated and what activities
should be left to the crew has to be addressed (Parasuraman, Sheridan, &
Wickens, 2000). We offer the following observations on this contentious
issue. First, a fully automated health management system is neither
appropriate nor optimal for a crewed spacecraft. Full automation is not
appropriate because health management software systems are not yet
robust, reliable, and capable enough to entrust the lives of a crew;
doing so would represent an unacceptable risk. Hardware and software
failures are more common in space than on the ground, in part because
space-based platforms are vulnerable to radiation-induced \"single event
upsets.\" Crewmembers are unlikely to trust software tools to the point
where they cede all control over emergency operations.

Aside from issues of trust and reliability, full automation is not
optimal for a crewed spacecraft because it does not fully utilize all
available onboard resources. Crewmembers are (and should continue to be)
trained in the architecture and functioning of vehicle systems to the
point where they become subject-matter experts in their own right.
Taking them out of the loop amounts to a decision to waste a valuable
onboard source of expertise. Equally important, humans and machines
bring different capabilities and different limitations to bear on the
health-management process. These capabilities are frequently
complementary, with the strengths of one compensating for weaknesses of
the other. For example, the leading technology for automated fault
diagnosis (Health Assessment Level in Figure 3) is MBR. To determine the
root cause of a set of symptoms (faults or anomalies), MBR utilizes
models of the systems that specify how components interact. Developing
high fidelity models of complex physical systems is very difficult, and
there are still issues to be resolved before MBR can offer a complete
diagnostic solution. Even if a model can be developed that reacts
correctly to all failure modes identified in advance, novel failures may
still occur as hardware ages and undergoes sustained exposure to the
launch and space environment. The model may not adequately deal with
these novel failures. Moreover, because MBR is dependent on sensor
placement and sensor coverage, an MBR system may not be able to
completely disambiguate a set of hypotheses that explain a given set of
symptoms. Humans excel (and surpass machines) in their ability to
retrieve situation-appropriate information from long-term memory, to
improvise and use procedures flexibly, and to exercise judgment based on
common-sense (nondeterministic) reasoning. A human may have deeper
insights and be able to apply knowledge from a somewhat similar
situation that happened under completely different circumstances to
arrive at a single root cause to a set of symptoms.

On the other hand, there are clear areas where machine capabilities
greatly surpass those of humans. Machines have much greater abilities to
perform highly complex operations virtually simultaneously (i.e., to
perform many different computations seemingly at once), and to monitor,
process, and recognize patterns in time-varying numeric data. In
conjunction with today's advanced electronic display devices, machines
can also retrieve, organize and display information much more
efficiently than humans can accomplish without machine assistance.

Given these considerations, we believe the key to health management on
next-generation space vehicles is to exploit the end-to-end capabilities
of today's health management systems, not to replace the crew, but to
evolve the traditional C&W system into a decision-and-action support
system (DASS) that *assists* the crew with all aspects of health
management operations (Scandura and Garcia-Galan, 2004). The outstanding
challenge with this approach is to develop a viable operational concept
that blends and coordinates the activities of the DASS with crewmembers.
We turn now to some of the issues of human-machine interface design that
must be addressed in order to meet this challenge. We then provide an
example of a candidate operational concept for real-time fault
management that illustrates these issues and offers candidate design
solutions.

**Guidelines For Human-DASS Teaming**

Fortunately, many of the issues that should be addressed in order to
achieve effective coordination of human and machine activities in a
spacecraft setting are already well documented (Malin, Schreckenghost,
Woods, Potter, Johannesen, Holloway, & Forbus, 1991). As applied to a
DASS system, these issues include, but are not limited to:

*Determining the optimal functional allocation between DASS and Crew*.
This determination must capitalize on the strengths and capabilities of
humans and machines, thereby optimizing the capabilities of the joint
human-machine system, and avoid the \"out-of-the loop unfamiliarity\"
(OOTLUF) problem (Endsley & Kiris, 1995) that accompanies systems with
too much automation. The functional allocation needs to strike a balance
between the reduction in workload that DASS makes possible with the
potential loss of situation awareness that can occur when DASS performs
operations without sufficient human oversight and involvement. There are
numerous examples from today's highly automated aircraft cockpits of
serious consequences where crewmembers have been insufficiently involved
in an automated operation, and are suddenly called upon to deal with the
consequences of that operation (Sarter, 2001).

Generic tools to determine an appropriate level of automation across a
wide variety of mission operations, including real-time fault
management, are currently under development. One such tool, based on
seminal work by Sheridan and his colleagues (Sheridan, 1992;
Parasuraman, et al., 2000) is the Function-specific Level of Autonomy
and Automation Tool (FLOAAT) under development at NASA Johnson Space
Center (Proud, Hart, & Mrozinski, 2003). FLOAAT begins by decomposing
task requirements to a functional level. These functions are then
classified into four types conceived by Col. John R. Boyd in the 1970s
as an air-to-air combat strategy for military fighter pilots. The four
types are Observe -- gather, monitor, and filter data; Orient -- derive
options through analysis, trend prediction, interpretation, and
integration; Decide -- rank the available options; and Act -- execute
the chosen option. Each type has eight corresponding levels of
automation with definitions ranging from fully manual (1) to fully
automated (8). For each function type, domain experts give a qualitative
rating (high to low) to specifically tailored questions whose aim is to
determine how much the experts trust the software to perform the
function correctly and what experienced developers believe is the
cost/benefit relationship of automating the function. The ratings are
tallied and a numeric score is generated for the level of automation
(LOA) that human operators will trust the system to be designed to, and
that balances the costs and benefits of automation performing the
function. These two limits are computed individually and the lower of
the two limits is suggested as the function's design level of automation
or autonomy.

*Ensuring crew visibility into DASS functioning*. Once a functional
allocation has been chosen, user interfaces must be designed to support
it. One of the fundamental requirements for these interfaces is that
they provide crew insight into DASS functioning. "DASS functioning" has
two distinct meanings. One meaning refers to any systems
reconfigurations performed by DASS (as opposed to the crew), and
resulting changes to system mode and system functioning. If a crewmember
is included in the control loop in any capacity, she (or he) must be
able to quickly synchronize her mental model with the actual state of
the system in order to address the detected failures and assist with
recovery actions. In some situations, automatic forms of compensatory
control can hide faults and failures until they are almost beyond the
operator's ability to recover. It is very important in such cases that
the compensatory actions are transparent to the operator, to allow for
earlier intervention or for planning a recovery strategy.

The second meaning of DASS functioning refers, not to the overt actions
of the system per se, but to the algorithms and computations that
underlie those actions. Following Billings (1997), automation is deemed
\"clumsy" if the underlying algorithms that support machine-based
decision-making and action taking are opaque to the human. User
interfaces are needed that allow the crew to make continuous
determinations such as, "Is DASS itself "healthy"? Is the system
performing in a manner consistent with what I know about its
functionality, the kinds of computations it should be performing, and
the outcomes of those computations?"

*Backup and Redundancy Requirements*. We have already noted the
particular vulnerability of spacecraft systems to subsystem failures and
malfunctions. At the hardware level, these vulnerabilities generate
stringent requirements for backup capabilities and functional
redundancies. The same considerations extend to a safety-critical
software system such as DASS, and place additional requirements on the
design of the crew-DASS system.

*Hardware/Software Redundancy*. On the shuttle, ascent and entry
guidance, navigation, and flight control functions are almost fully
automated. A failure in either the flight software itself, or in the
hardware on which the flight software is housed, would put the mission
in immediate jeopardy. Fully recognizing this vulnerability, the
original shuttle designers had separate contractors develop independent
flight software systems. One company's software was designated as
primary, the other as backup. The primary software system, housed on
four of the five onboard general-purpose computers, has nominal control
over the vehicle. If the primary system fails, due to some combination
of software or computer failure, the backup system, housed on the fifth
computer, can provide the most essential flight control functions.

There is an important user interface element illustrated by this
redundancy. The two software systems continuously and independently
compute critical flight parameters (e.g., vehicle attitude) that the
backup system requires in case the crew has to engage it. In addition,
at two minutes into ascent, additional guidance parameters (e.g., the
exact time of main engine shut down to achieve the targeted orbit
insertion point) are computed redundantly. These values, along with any
real-time discrepancies between the systems' computations of vehicle
attitude, are displayed continuously to the crew. By crosschecking these
values, the crew can assess the health of each software system and the
veracity of its navigation and guidance computations. As we move into a
next generation vehicle, where DASS is responsible for a much larger
proportion of health-related data and information processing activities
than today's C&W system, similar software and hardware redundancies may
need to be built into the DASS system to provide similar opportunities
to crosscheck DASS functioning.

*Crew as Backup*. Redundancy requirements do not stop with the hardware
and software. To protect against a general breakdown in the data
processing system (or critical component), the crew should retain as
much capability as possible to perform DASS functions manually (ideally,
the reverse would also be true; DASS should be able to act as backup in
the event of a crewmember "malfunction." We shall return to this rather
tricky issue when we discuss adaptive cockpits in the next section). Of
course, at the data processing level (Level 1 in Figure 3), this
requirement is unobtainable; three or four crewmembers simply do not
have the capacity to monitor and detect anomalies in thousands of data
sources, even if all the data could be displayed simultaneously. On the
other hand, crews today perform virtually all the physical actions,
predominantly switch throws, called for in the fault management
procedures. In principle, these purely physical actions could be
automated (see below). But this approach exposes a human factors
conundrum. In order to function effectively as a backup, a crewmember
must retain the skill set needed to perform nominally automated
functions. If a procedure or operation is always automated, crewmembers
will not get a chance to perform it, and will lack the skill set
necessary to serve as backup if the need arises. This issue could be
resolved through extensive training on scenarios that simulate DASS
failures and require a reversion to manual operations. Another option is
to design the crew-DASS functional allocation and supporting interfaces
in such a way that, although the automation actually performs the
operation, allowing the workload-reduction benefits of the automation to
be realized, the human is kept "in the loop" in a manner that
continuously reinforces the skill set necessary to perform the function
manually. In the next section, we describe one option that follows this
"continuous reinforcement" approach.

# Candidate DASS for Next-Generation Space Vehicles

The "state-of-the-art" health management system described in Figure 3
has the ability to automate virtually all of the fault management
activities currently performed by the crew and/or ground personnel
(Figure 1). Mindful of the human factors risks of over-automation,
McCann and McCandless (2003) argued for a functional allocation between
crew and health management system that would give the automation
responsibility for data monitoring, detecting off-nominal conditions,
making root-cause failure diagnoses, and retrieving and executing the
appropriate procedure(s). The crew would maintain overall control of the
process, however, by having to give the automation "permission" to
execute each individual fault management procedure in the (now
electronic) FDF. In this section, we describe a candidate crew-DASS
system based on this particular functional allocation. The candidate
provides a platform from which to describe user interfaces and
operational concepts that follow the functional allocation
determination, and begins to address some of the design requirements for
a Crew-DASS system identified earlier in the paper.

*Making DASS Activities Transparent to the Crewmember: System Summary
Displays.* A central design requirement is for user interfaces that
enable crewmembers to precisely track changes to the functional mode of
a system that accompany DASS actions. This would be quite difficult with
the current shuttle system summary displays, as they do not make the
current operating mode transparent to an observer. Fortunately, these
display formats were the target of a recently completed Cockpit Avionics
Upgrade (CAU) redesign by teams of NASA astronauts, mission operations,
and human factors personnel. The redesigned displays, described by
McCandless, Hilty, & McCann (2005), consolidate systems information onto
individual, task-oriented systems summary displays, eliminating the need
to navigate through several displays to acquire systems information.
More importantly, the displays incorporate many of the synoptic features
of systems displays in modern glass cockpit aircraft, organizing
graphical depictions of system components, such as valves, tanks, and
flow lines, around a spatial arrangement that emulates key aspects of
the underlying system architecture. In addition, CAU designers
incorporated luminance and color-coding schemes into the graphics so as
to provide "at-a-glance" information concerning current system
configuration and operating mode.

An example of these changes is the redesigned Main Propulsion System
(MPS) Summary Display, shown on the right side of Figure 2. The
zoomed-in section on the bottom right section of the figure shows the
graphical depiction of the helium supply system for the shuttle's three
main engines (essentially, this is the redesign of the section of
today's BFS GNC SYS SUM 1 display shown on the lower left). Our working
assumption is that system summary displays on next generation vehicles
will resemble the CAU redesigned displays much more than they will
resemble existing shuttle displays. For that reason, the CAU displays
will form the point of departure for our discussion of candidate
Crew-DASS interfaces.

The next major requirement for DASS displays is that they provide
insight into the health and functioning of the DASS even when all
systems are functioning nominally. Consider, first, how we might provide
insight into the functions associated with the Condition Monitoring
level in Figure 3. On CAU System Summary Displays, sensed values are
normally shown in white. If a value moves beyond its Caution and Warning
software limits, the out-of-limits value is signaled by changing the
color to either yellow or red, and presenting a yellow or red "up" or
"down" arrow (or other symbol) beside it. By contrast, the DASS
Condition Monitoring level would contain much more sophisticated data
processing algorithms such as time series analyses of the fluctuations
(variance) in sensor readings over time. These fluctuations are a normal
constituent of complex systems operations, and mathematical analyses of
the fluctuations can be a powerful tool for classifying nominal versus
off-nominal modes of system functioning (Huff, Tumer, & Moser, 2001).
During nominal systems operations, these algorithms will be operating
covertly, in the background as it were. It would be highly desirable to
provide a means of continuously informing the crew that these
classification algorithms are "alive and well" and actively performing
their function.

One design concept to meet this requirement would be to first
characterize the variability in sensor values that is consistent with
nominal systems operations (similar to the bandwidth used to assign
upper and lower limits in a limit-sensing system). During times of
systems operation, the current sensor readings on the system summary
displays would appear in grey, and variance in the readings over time
would be accompanied by physical changes in the intensity and/or color
of the depicted values. Variance consistent with nominal operations
would be accompanied by subtle changes in the brightness of the
parameter (a step up in brightness when the value increases, and a step
down when the value decreases). Since these changes would only affect
brightness along the grey scale, they would give the user "at-a-glance"
indications that DASS was classifying the current variability as
consistent with nominal systems operation, and also that the DASS
algorithms were "live", processing the data and performing the
nominal/off-nominal classification in real time. If, at some point, the
algorithms determined that the changes were consistent with the
beginning of a deviation from nominal functioning, all of the sensed
values contributing to that assessment would change to the warning
color, typically yellow. Yellow values would signal that an abnormal
trend has been detected; the values would change to red once the
abnormal condition was confirmed. In this scheme, color-and-intensity
coding would change from today's system, based only on limit sensing, to
a scheme that communicates information about computations that are
integrating information across both time and individual sensors.

The next algorithmic layer is Health Assessment, where the root-cause of
an off-nominal data pattern is determined. Again, we would like our user
interface to convey some indication of the reasoning behind root-cause
failure determinations. However, this issue is complicated by the
plethora of algorithms and computations employed by different reasoners.
Some are model-based, others rule-based, while still others take a
sensor-fusion approach that subtracts expected (nominal) values from the
current readings, and then matches the pattern of residuals against
known failure modes. Since the user interface will be highly dependent
on the specifics of these computations, we will not attempt to provide a
generic interface example here.

The final DASS level is Procedure Retrieval and Execution. From the
perspective of a "permissions-based" crew-DASS functional allocation,
this level involves the tightest interactions between crewmembers and
onboard automation, and places the greatest demands on user interface
design. The primary requirement for the interface is that it enables and
coordinates the "permissions-based" concept for procedure execution.
Arguably the most obvious candidate for this interface is a dedicated
fault management display patterned after the systems and fault
management displays developed for the Boeing 777 and Airbus "A" series
of glass-cockpit commercial aircraft. When a malfunction is detected on
an A320, for example, the appropriate checklist automatically appears in
written form in a dedicated (normally blank) section of the ECAM display
page. In addition, a "systems synoptic" appears on a separate Systems
Page. The synoptic depicts components of the system affected by the
malfunction in a spatial layout designed to match the crews' mental
model of system architecture and system functioning. The crew completes
each procedure manually, and keeps track of the status of each procedure
(completed versus not completed) via a checkmark that fills a box to the
right of each procedure, when completed. In addition, as the crew
performs each procedure, the synoptic changes to display the new system
configuration.

Figure 4 contains a prototype of a candidate fault management display
that incorporates elements of both the electronic checklist and the
"embedded-in-synoptic" formats, in a hypothetical DASS-equipped version
of the shuttle. The display depicts a rather complex failure situation
involving the helium supply system to one of the three (Center) engines
that make up the shuttle's Main Propulsion System. When an engine is
operating nominally, helium flows out of the tank at the top of the
figure and then splits into two redundant legs, each with a separate
pressure regulator, before rejoining and flowing to the engine in
question (where the helium continuously pressurizes a seal in the
engine's high pressure oxidizer turbopump). Following the design
criteria established by the CAU project, valves are depicted as circles
containing an embedded line. When the valve is open, the embedded line
is flush with the rest of the line, and the entire line is bright white,
indicating that helium is currently flowing. We can see in the figure
that the Isolation Valve for the right-hand (B) Leg is open and (as we
would expect), helium is flowing through Leg B.

However, the symbol for the Leg A Isolation Valve is colored red, the
interior line is perpendicular to the flow, and the Leg A line below the
valve is colored dark grey (signaling no flow). Together, these
indicators show that Isolation Valve A has failed to the "close"
position. Critically, however, the figure also indicates a second
failure in the system. The "dP/dT" value in the upper right-hand corner
is colored yellow, which indicates that helium is being depleted at a
higher than nominal rate from the tank, that is, the Valve A failure has
been compounded by a leak somewhere in the helium supply system. This
set of circumstances conforms to a "Non-Isolatable Helium Leak"
condition, indicated at the top of the display, which is associated with
two procedures. When the helium tank pressure (currently at 2253 PSI)
falls below 1150 PSI, a manifold connecting the engine's helium supply
system with a backup helium supply system must be opened, to keep helium
flowing to the engine for as long a period as possible. Then, to avoid
any chance of full depletion of the supply, the engine must be shut down
when the vehicle reaches an inertial velocity of 23,000 feet per second.
We assume that the DASS is capable of estimating the time remaining
before both conditions will be satisfied, and also that the tank
pressure will deplete to the target value (1 min 12 sec from the
present) before the vehicle reaches engine shut down velocity (almost 4
min from the present). Thus, the "Interconnect Open" procedure appears
at the top of a written procedures section (lower half of the display),
and a "countdown" timer is provided just to the right of the written
instruction. The timer currently shows 1 min 12 sec until this procedure
should be performed. All information is written in yellow, to signal
that the "Open Interconnect" procedure should not be performed
immediately, but 1 minute 12 seconds in the future.

The schematic in the upper half of Figure 4 depicts the same information
as the written procedure. The schematic is essentially a modified,
zoomed-in version of the helium supply section of the CAU Main
Propulsion System Summary Display, shown in the bottom right section of
Figure 2. The schematic always depicts the current operational
configuration, so the interconnect valve is shown in the closed position
with no flow through the interconnect manifold. The procedural
information is embedded in the schematic via size-and-color-coding: the
interconnect valve symbol is enlarged and yellow, and a yellow countdown
timer appears below the symbol. The color-coding signals that the
upcoming procedure will command a change from the current (closed)
position to the alternative (open) position.

As soon as the condition needed to carry out this activity is satisfied,
the fault management display changes to "command" mode, illustrated in
Figure 5. The countdown symbols disappear, and the interconnect valve
symbol turns magenta. In parallel, the written procedure also turns
magenta, and a virtual magenta "Accept" button appears to the right of
the procedure. Again, referring to the schematic, magenta signals a
recommended change from current (closed) to alternative (open) position.
After appropriate review, the crewmember signals his or her agreement to
proceed with the "Open Interconnect" procedure by touching the "Accept"
button. Once DASS has performed the action (not shown in the figure),
the "He Interconnect" procedure shifts down, and turns grey, and the
engine shutdown line moves to the top of the written procedures stack.
The schematic converts to a main engine synoptic showing the crucial
valves that must be closed in order to achieve Engine Shutdown.

The "permissions" mode of human-automation interaction, and this user
interface prototype, is designed to keep the crewmember in close
synchronization with DASS and closely track DASS actions. On the upper
(schematic) section, the mental exercise required to translate current
valve position into the commanded position will further enhance
situation awareness of the nature of the malfunction and the system
configuration that will result from the recommended action.

*Preserving Backup Capability with Head-Down Displays.* This interface
concept still contains a significant human factors drawback. Results of
a recent study of shuttle fault management behavior by relatively novice
operators (Huemer, Matessa, & McCann, 2005) provides strong evidence
that knowledge of the locations of the switch panels and their embedded
switches plays an important role in enabling crewmembers to work
malfunctions quickly and accurately. Because crewmembers are required to
locate and throw the switches themselves, mission training in
ground-based simulators continuously preserves and reinforces this
knowledge. By contrast, the proposed system does not require crewmembers
to physically locate the switches, potentially degrading their ability
to do so in the event of a DASS failure.

Could we redesign the user interface to retain the benefits of automatic
procedure execution and still preserve the crew's backup capability? One
possibility might be to overlay each of the cockpit switch panels with a
"head-down display", a (removable or retractable) layer of glass on
which arbitrary forms of information and symbology could be superimposed
on the panels. Since DASS performs the actual switch throws, the actual
hardware switches underneath the glass panels would always remain in
their "Computer-Controlled" position (usually, the middle position of a
standard up-down switch configuration). The head-down displays would
include "virtual" switch icons that depict the actual switch positions.
When a procedure calls for a switch throw, the change could be signaled
on the virtual switch panel via color-coding that makes the affected
switch "pop out" at the observer from the virtual display. And, by
having the "accept action" button appear right beside the commanded
switch, the crewmember would be forced to orient to the location of the
switch in the cockpit, constantly reinforcing his or her spatial
representation of switch locations. The crewmember would therefore
retain the knowledge to take over and throw the hardware switches
manually, if required.

The idea of overlaying cockpit switch panels with glass displays has
other potential benefits. Since all the hardware switches would be in
(and remain in) the "Computer-Controlled" position, the physical switch
panels would only provide information concerning actual (current) valve
positions (e.g., open, closed, or in transition) if talkback indicators
are present. The necessity to include a talkback indicator for every
switch greatly increases the real estate requirements of the switch
panels. If the switch panels were overlaid by head-down displays, the
virtual switch icons showing actual switch positions would perform the
function of the talkback indicators, greatly reducing the real-estate
requirements of the physical switch panels.

Finally, by distributing procedural information across two separate
cockpit locations (on top of the switch panels and on the dedicated
fault management display), we would have a chance to build in some
health management crosschecking similar to the crosschecking between the
Primary and Backup flight software systems. We have already noted that
reasoning systems come in many different forms. Suppose each
systems-level health manager incorporated two distinct reasoning systems
that worked redundantly and independently on root-cause determinations
and fault diagnosis. One system could be generating root-cause fault
determinations, and providing the appropriate procedures, on the fault
management display. The other system could be performing parallel
computations, also leading to the specification of the appropriate
procedures, but depicting them on the head-down displays. Crosschecking
the recommended procedures across the two display formats would enable
the crew to assess the veracity of the information being generated by
the fault management systems, and increase their trust in the
recommended actions. The crosschecking requirement would also
continuously reinforce crewmembers' spatial knowledge of actual switch
locations, again helping maintain the spatial knowledge needed to
execute procedures manually in the event of a complete DASS failure.

**Crew-DASS Interactions on Longer-Duration (Exploration) Missions**

Our candidate crew-DASS interfaces conform to current spacecraft design
conventions by conveying all DASS information to crewmembers via visual
displays, and restricting crew interactions with DASS to finger and hand
(manual) interfaces. This standard approach to spacecraft cockpit design
may simplify and streamline DASS development to the point where the
system can be incorporated into the initial (Block 1) version of the
Crew Exploration Vehicle (CEV; the replacement for the shuttle),
scheduled to begin operations as early as 2011 or 2012. However, these
conventional "input-output" channels will also force crewmembers to
process DASS (and other flight and mission) information in small serial
chunks (eye fixations) and, if more than one action is required, to
execute each action strictly sequentially. Extensive laboratory research
on human information processing limitations has also shown that more
central cognitive operations, such as, for example, deciding whether
DASS should be permitted to execute a procedure, are also carried out
strictly serially. Even with DASS assistance, the "end-to-end" serial
nature of fault management operations imposed by these interfaces will
place clear lower limits on the time needed for a crewmember to resolve
and complete the standard sequence of fault management activities. The
interfaces will also virtually eliminate crewmembers ability to
time-share malfunction handling operations on more than one malfunction
at any one time. While emergency situations involving multiple
overlapping malfunctions are rare, they are not unheard of. They pose a
particularly clear danger during dynamic flight phases, when cognitive
tunneling on fault management operations necessarily degrades situation
awareness of vehicle flight path, attitude, and other safety-critical
vehicle operations.

Fortunately, user interface technologies have advanced to the point
where systems information can now be presented via auditory and even
haptic displays, and machines can be operated and controlled by voice
command as well as the more traditional manual interfaces. Studies have
established the existence of a "preattentive" processing mode that
extracts and processes information from all modalities in parallel and
automatically directs focal attention to stimuli salient to the
operator's current task set (Folk, Remington, and Johnston, 1992; Woods,
1994). It is also well established that people have considerable ability
to plan and execute motor responses with their hands (automobile
steering) in conjunction with speaking (Wickens, Sandry, & Vidulich,
1983). These findings suggest that crew-DASS interfaces involving these
nontraditional modalities could increase the parallel processing
capabilities of crewmembers, in effect boosting the human information
processing resources available to work health management issues onboard
the vehicle. These enhancements may not be critical for Block 1 CEV
operations, when the vehicle will be confined to low Earth orbit and the
crew will be able to rely on near real-time assistance from the ground.
On later missions, when health management-related emergencies will have
to be handled without such assistance, such interfaces could well make
the difference between mission success and mission failure.

Therefore, in addition to considering traditional visual/manual
crew-DASS interfaces, we need to consider Crew-DASS interactions via
auditory and tactile channels. Malfunction warnings are an obvious
target. An auditory warning can be designed to encode the degree of
urgency of the problem, and 3-D audio technology can co-locate the
perceived location of an alarm with the affected subsystem (thereby
indicating the location of the threat) or with the visual fault
management page (thereby quickly attracting crewmembers' visual
attention to diagnostic information). Additionally, both auditory and
tactile channels have been shown to be useful for presenting other types
of data. Researchers have determined that humans can effectively extract
useful systems status information from sounds that encode information
relevant to systems operations, and that tactile feedback can be
effective for informing a user about system state (for example, thruster
operations can be conveyed through tactile elements \["tactors"\] whose
pressure on the skin maps to the current thrust level). Finally, natural
language interfaces may be an effective method for time-sharing
Crew-DASS communications with processing critical flight parameters on a
visual primary flight display (PFD). For example, during a vehicle
descent, DASS could communicate recommended procedures to a crewmember
via a synthetic speech generator, and the crewmember could provide
permission to perform the procedure via a speech recognizer, while the
crewmember monitors vehicle attitude on the PFD.

*Toward an Adaptive Cockpit*. Earlier in the paper, we noted that
automating fault management operations via DASS should be accompanied by
a requirement, where possible, for crewmembers to act as a backup for
DASS in case of hardware or software failure. We also made passing
reference to the reverse case, where the automation would take over and
accomplish functions that crewmembers normally perform. In our Crew-DASS
functional allocation concept, this requirement would involve adjusting
the automation level "upward" to the point where DASS would execute
fault management procedures automatically, either after a time period
elapses during which the crew can veto the action, or with no crew
involvement at all.

Again, this kind of adaptive capability may not be a priority for Block
1 CEV operations, as mission durations will be short, crewmembers are
likely to be functioning at a high level, and plenty of real-time ground
assistance will be available. As missions increase in duration, however,
requirements for adaptive capabilities are likely to grow. Crewmembers
will experience long-term exposure to various space-based environmental
stressors, such as circadian disruptions (fatigue), confinement,
microgravity, and possibly elevated doses of radiation. During the
quiescent (cruise) phase, minor performance decrements associated with
these stressors may not pose much of a mission risk, as there are few
situations where information processing and decision-making requirements
are high enough to stress human capabilities. However, these quiescent
periods are always followed by highly dynamic flight phases where, for a
short time, crewmembers are called upon to manage and participate in
activities that *do* place high demands on their information-processing
capabilities. In a relatively recent interview, Neil Armstrong
identified piloting and landing the lunar excursion module as the
highest workload phase of the Apollo 11 mission (despite the fact that
he was receiving near real-time assistance from the ground). To make
matters worse, possibly the greatest space stressor of them all, an
abrupt transition to a new gravitational environment, generally
accompanies the abrupt transition to these high-pressure operational
environments.

These stressors have considerable potential to reduce crewmembers'
ability to perform effectively in high-workload situations. Therefore,
depending on crew condition, it may be useful to flexibly adjust the
Crew-DASS functional allocation in order to decrease crew workload. The
crew could make this determination themselves, of course, based on their
own subjective determination of their current state and capabilities.
Alternatively, sophisticated monitoring tools are currently under
development that assess crew activities and performance patterns in real
time, and make automatic determinations of crew workload and current
cognitive function (Schmorrow & Kruse, 2004; Raley, Stripling, Kruse,
Schmorrow, & Patey, 2005). The threshold for what type of fault
management decisions requires crew input could be automatically raised
or lowered depending on the real-time assessments of crew state and
current workload. In addition, beyond their potential for enhancing
crewmembers' processing capabilities, multi-modal displays could play a
critical functional role in adapting Crew-DASS interfaces to accommodate
real-time assessments. The system could flexibly determine which
modality to use to notify the crew of a problem, for example. Research
is currently underway to develop tools that can select the appropriate
modality based on a model of the crew's real-time activities and a
stochastic determination of how detectable information presented in a
particular modality would be in light of current activities.

# Usability Evaluations

Bringing advanced automation into the flight deck of commercial and
military aircraft has generally enhanced operational safety margins.
Frequently, however, the automation has emergent properties, not
anticipated by designers, which negatively impact human performance to
the point of opening up new possibilities for human error. These
possibilities represent unacceptable risks for human spaceflight.
Therefore, when developing an operational concept for a system, such as
DASS, that automates many hitherto manual operations and carries
requirements for completely new classes of user interfaces, it is
crucial that a rigorous, thorough program of testing and evaluation be
incorporated into the design process.

Traditional usability programs rely heavily on metrics such as the
response time and accuracy with which test subjects perform overt manual
activities, such as switch throws. Evaluating the usability of
human-machine systems in which many of human activities become automated
is a more challenging problem. There is less overt activity to measure,
and a large proportion of the crewmembers' time is occupied by simply
monitoring the functioning of the automated system. Accordingly, human
factors evaluations of these systems have had to incorporate new, more
covert measures of crew-automation interaction.

For example, this approach is embodied in the Intelligent Spacecraft
Interfaces System (ISIS) laboratory at NASA Ames Research Center. The
ISIS lab is currently configured to simulate a single-seat version of
the Space Shuttle cockpit, but can be adapted to simulate a more general
CEV crew-station environment, as necessary. Human-in-the-loop
simulations have been completed of nominal and off-nominal shuttle
ascent scenarios that include realistic systems malfunctions that must
be identified and worked in real time. The lab records traditional overt
behaviors, such as keyboard key presses and switch throws. However, as
we just noted, these traditional measures of performance are not
adequate to make the kind of fine-grained determinations of information
acquisition and processing needed to evaluate crew-DASS interfaces.
Thus, the laboratory also incorporates an eye-tracking system that
records participants' gaze positions with high temporal and spatial
resolution.

To illustrate the usefulness of eye movement data for evaluating systems
such as DASS, consider the candidate fault management display concept
illustrated in Figures 4-5. The fault management page represents
DASS-recommended procedures in two distinct forms. The bottom half of
the display provides the procedures in the traditional written form
(checklist) common in today's glass cockpit aircraft. Following Malin,
et al., (2000), the top half embeds the commanded procedures into the
schematic of the affected system. The goal of this redundancy is to get
crewmembers to crosscheck the two recommendations for consistency and
accuracy of the automation, and enhance crew situation awareness of the
overall fault management process. Eye-movement recordings during
human-in-the-loop simulations will enable us to directly determine the
frequency of such crosschecking, and thereby make direct (i.e.,
empirical) determinations of whether the design is meeting the
functional goals of the designer, and of user preferences for one format
or the other. These data will then be used to iterate on the design
concept and optimize the features of the fault management display.

*Usability testing in analog environments*. Techniques that perform well
in a spacecraft simulator can be further evaluated in a spacecraft
analog environment. High performance aircraft, such as an F-18 military
jet, share many characteristics with a spacecraft. Because high
performance aircraft and spacecraft both are noisy, have a lot of
vibration, and operate under high-G loading, the supposition is that
there is a high correlation between usability of auditory and tactile
displays in a high performance aircraft and in a spacecraft. In many
situations, the workload of a fighter pilot is higher than that of an
astronaut. Hence, the effectiveness of visual displays can also be
evaluated in a high performance aircraft. Although systems in a high
performance aircraft are not as complex as those of a spacecraft,
arguments can even be made for testing the usability of proposed levels
of automation and autonomy in the analog environment of a high
performance aircraft. NASA ARC, NASA Dryden, and JPL are conducting an
initial investigation of this hypothesis with the evaluation of ISHM
systems on an F-18. Preliminary examination of this approach appears
promising.

**Summary and Conclusions**

Replacing the shuttle with a new generation of crewed space vehicles
gives designers a unique opportunity to enhance onboard health
management capabilities with state-of-the-art health management software
systems. However, there is no such thing as a free lunch. Making
fundamental changes to "tried and true" health management operational
concepts carries significant operational risks, cogently captured by the
Johnson Space Center slogan "Better is the Enemy of Good." The way to
ensure that "better" is a friend rather than an enemy is to adopt an
operational concept for onboard health management in which the health
management system becomes a team member, working fault management
activities in close collaboration with the crew. The development of a
successful "teaming" concept requires careful up-front analyses of the
relative strengths that crewmembers and machines bring to bear on fault
management operations, determination of a functional allocation,
development of the user interfaces to support that allocation, and
testing and evaluation of crew-system functioning in human-in-the-loop
simulation.

We close with the observation that, even though the development process
outlined in this paper is decidedly not trivial, there are broader
considerations that dictate that health management software infusion
should not proceed in a stovepipe fashion. There is some consensus that
individual sources of information about subsystem state and functioning
should be integrated into a more comprehensive "mission management"
system with capabilities to perform cross-system and mission-level
impact assessments of health-management-related decisions, such as
ascent aborts. Such a system would require a higher-level "mission
manager" that coordinates and integrates inputs from intelligent flight
software systems and health management systems. Once again, to make such
a complex knowledge engineering architecture functional, critical issues
of human-machine interaction will have to be addressed. Suppose
individual health managers (DASS systems) were all connected to a
central mission manager. Where would control reside for issuing crew
warnings and alerts? Should all off-nominal determinations from the
system-level health managers go through the mission manager, so that it
can act as a final "filter" to suppress false or nuisance alarms? In an
adaptive cockpit, where would authority reside for who should (or should
not) be alerted to an anomaly? The message from even these few questions
is clear. Developers of advanced software systems should always keep in
mind how their systems will interface with humans to accomplish mission
goals and mission operations in a collaborative fashion.

# 

# 

# 

# References

Atlas, L., Bloor, G., Brotherton, T., Howard, L., Jaw, L, Kacprzynski,
G., Karsai, G., Mackey, R., Mesick, J., Reuter, R., & Roemer, M. (1999).
An evolvable Tri-Reasoner IVHM system. *Boeing Company Publication*.

Billings, C. E. (1997). *Aviation automation: The search for a
human-centered approach*. Hillsdale, NJ: Erlbaum.

Endsley, M. R., & Kiris, E. O. (1995). The out-of-the-loop performance
problem and level of control in automation. *Human Factors*, *37*, pp.
381-394.

Folk, C. L., Remington, R. W., & Johnston, J. C. (1992). Involuntary
covert orienting is contingent on attentional control settings. *Journal
of Experimental Psychology: Human Perception and Performance*, 18, pp.
1030-1044.

Huemer, V., Matessa, M., & McCann, R. S. (2005). Fault Management during
Dynamic Space Flight: Effects of Cockpit Display Format and Workload.
*IEEE International Conference on Systems, Man, and Cybernetics CP* (in
press).

Huemer, V. A., et al. (2005). Characterizing Scan Patterns in a
Spacecraft Cockpit Simulator: Expert versus Novice Performance. *HFES
49th Annual Conference CP* (in press).

Huff, E. M., Tumer, I. Y., & Mosher, M. (2001). An Experimental
Comparison of Transmission Vibration Responses from OH58 and AH-1
Helicopters. *AHS 57th Annual Forum CP*.

Keller, K., Wiegand, D., Swearingen, K., Reisig, C., Black, S., Gillis,
A., Vandernoot, M. (2001). An architecture to implement integrated
vehicle health management systems. IEEE publication.

Malin, J. T., Schrechenghost, D. L., Woods, D. D., Potter, S. S.,
Johannesen, L., Holloway, M., & Forbus, K. D. (2991). Making Intelligent
Systems Team Players: Case Studies and Design Issues, Vol. 1:
Human-Computer Interaction Design. *NASA TM #104738*.

Malin, J., Kowing, J., Schreckenghost, D., Bonasso, P., Nieten, J.,
Graham, J. S., Fleming, L. D., MacMahon, M., & Thronesbery, C. (2000).
Multi\--agent diagnosis and control of an air revitalization system for
life support in space. *2000 IEEE Aerospace Conference CP*.

McCandless, J., Hilty, B., & McCann, R. S. (2005). Development of new
displays for the space shuttle cockpit, *Ergonomics in Design* (in
press).

McCandless, J., McCann, R. S., and Hilty, B. R. (2003). Upgrades to the
Caution and Warning System of the Space Shuttle, in *Human Factors and
Ergonomics Society 47^th^ Annual Meeting CP*, Denver, CO, pp. 16-20.

McCann, R. S., & McCandless, J. (2003). Human-Machine Teaming for
Dynamic Fault Management in Next-Generation Space Vehicles. *JANNAF 3rd
Modeling and Simulation Subcommittee CP.*

Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). "A Model of
Types and Levels of Human Interaction with Automation*," Proc. of IEEE
Transactions on Systems, Man, and Cybernetics -- Part A: Systems and
Humans*, Vol. 30, No. 3.

Proud, R. W., Hart, J. J., & Mrozinski, R. B. (2003). "Methods for
Determining the Level of Autonomy to Design into a Human Spaceflight
Vehicle: A Function Specific Approach," Proc. Performance Metrics for
Intelligent Systems (PerMIS '03), NIST Special Publication 1014,
September 2003.

Raley, C., Stripling, R., Kruse, A., Schmorrow, D., & Patey, J. (2005).
Augmented Cognition Overview: Improving Information Intake under stress.
*Annual Meeting of the Human Factors and Engineering Society CP*, New
Orleans, LA.

Sarter, N. B. (2001). "Multimodal Communication In Support of
Coordinative Functions in Human-Machine Teams," *Journal of Human
Performance in Extreme Environments*, *5*(*2*), 50-54.

Scandura, P. A., & Garcia-Galan, C. A. (2004). A Unified System to
Provide Crew Alerting, Electronic Checklists and Maintenance Using IVHM.
*IEEE DASC Conference*, CP312, 2004.

Sheridan, T. B. (1992). Telerobotics, Automation, and Human Supervisory
Control. M.I.T. Press, Comabirdge, MA.

Schmorrow, D., & Kruse, A. (2004). Augmented Cognition. *Berkshire
Encylopedia of Human-Computer Interaction.*

Wickens, C. D., Sandry, D. L., & Vidulich, M. (1983). Compatibility and
Resource Competition Between Modalities of Input, Central Processing,
and Output. *Human Factors*, *25*, 227-248.

Woods, D. (1995). The alarm problem and directed attention in dynamic
fault management. *Ergonomics*, *18*, 2371-2393.