The Human Factors of Sensor Fusion 

by Bruce P. Hunn 

ARL-TR-4458 May 2008 

Approved for public release; distribution is unlimited.
NOTICES 

Disclaimers 

The findings in this report are not to be construed as an official Department of the Army position 

unless so designated by other authorized documents. 

Citation of manufacturer’s or trade names does not constitute an official endorsement or 

approval of the use thereof. 

DESTRUCTION NOTICE⎯Destroy this report when it is no longer needed. Do not return it to 

the originator. 
Army Research Laboratory 

Aberdeen Proving Ground, MD 21005-5425 

ARL-TR-4458 May 2008 

The Human Factors of Sensor Fusion 

Bruce P. Hunn 

Human Research and Engineering Directorate, ARL 

Approved for public release; distribution is unlimited. 
ii 

REPORT DOCUMENTATION PAGE Form Approved 

OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining 

the data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for 

reducing the burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, 

VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not 

display a currently valid OMB control number. 

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 

1. REPORT DATE (DD-MM-YYYY)

 May 2008 

2. REPORT TYPE 

Final 

3. DATES COVERED (From - To) 

 January to December 2007 

5a. CONTRACT NUMBER 

5b. GRANT NUMBER 

4. TITLE AND SUBTITLE 

 

 The Human Factors of Sensor Fusion 

5c. PROGRAM ELEMENT NUMBER 

5d. PROJECT NUMBER 

 622716.AH70 

5e. TASK NUMBER 

6. AUTHOR(S) 

 Bruce P. Hunn (ARL) 

5f. WORK UNIT NUMBER 

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

 U.S. Army Research Laboratory 

 Human Research and Engineering Directorate 

 Aberdeen Proving Ground, MD 21005-5425

8. PERFORMING ORGANIZATION 

 REPORT NUMBER 

ARL-TR-4458 

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR”S ACRONYM(S) 

11. SPONSOR/MONITOR”S REPORT NUMBER(S 

12. DISTRIBUTION/AVAILABILITY STATEMENT 

 Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES 

14. ABSTRACT 

This report discusses select, cognitively based principles associated with the sensor fusion process. A review is made of the 

standard definitions and descriptions of sensor fusion from an information processing perspective, and that review is tied to 

basic principles of human cognitive processes which are involved with processing information. Each step of the definition 

of sensor fusion provided by the Joint Directors of Laboratories is used as a basis of comparison in this report, and elements 

of human cognition associated with those steps are described in theory and practice. Comparisons of machine cognition are 

contrasted and compared with human cognition, and recommendations are made about allocation of functions to human and 

machine in sensor-fused information systems. Implications of sensor fusion as applied to military operations are also 

discussed.

15. SUBJECT TERMS

 automation; human factors; information processing; sensor fusion 

16. SECURITY CLASSIFICATION OF:

19a. NAME OF RESPONSIBLE PERSON

 Bruce P. Hunn 

a. REPORT 

Unclassified 

b. ABSTRACT 

Unclassified 

c. THIS PAGE 

Unclassified 

17. LIMITATION 

OF ABSTRACT 

SAR 

18. NUMBER 

OF PAGES 

35 19b. TELEPHONE NUMBER (Include area code) 

 520-538-4701 

Standard Form 298 (Rev. 8/98) 

Prescribed by ANSI Std. Z39.18
iii 

Contents 

List of Figures iv

1. Introduction 1

1.1 Background .....................................................................................................................1 

1.1.1 Basic Concepts ....................................................................................................1 

1.1.2 Sensor Fusion as a Process..................................................................................2 

1.1.3 Sensor Fusion Defined ........................................................................................2 

2. Purpose of Report 4

3. Method 4

3.1 Human Engineering Principles Applied to Sensor Fusion..............................................4 

3.1.1 Human Versus Machine Process Cycles Comparison ........................................5 

3.1.2 Human Processing of Information ......................................................................6 

3.1.3 Human Logic in the Processing of Information ..................................................7 

3.1.4 Subjective Factors That Affect Human Processing of Cognitive Information, 

 Data Risk, Data Value, and Cognitive Bias...............................................................9 

4. Results 12

4.1 The Five Levels of the JDL Model of Sensor Fusion ...................................................12 

4.1.1 JDL Level 0, Pre-processing: Signal or Feature States ....................................12 

4.1.2 JDL Level 1, Object Refinement: Entity Parametric and Attributive States....13 

4.1.3 JDL Level 2, Situation Refinement...................................................................17 

4.1.4 Level 3, Fusion: Threat Refinement, Estimation, Prediction, and Utility........20 

4.1.5 Level 4, Fusion: Process Refinement and Human Learning............................20 

4.1.6 Level 5, Fusion: User Refinement....................................................................20 

5. Summary Discussion 21

6. Future Directions 21

7. References 23

Acronyms 26

Distribution List 27 
iv 

List of Figures 

Figure 1. The JDL model, 1987-91 with 1997 revisions (Steinberg, Bowman, & White, 1999).. 3 

Figure 2. Revised JDL user model, 2003-2006 (Blasch & Plano, 2003)....................................... 4 

Figure 3. The Wickens' model of human attention (Wickens & Hollands, 2000)......................... 7 

Figure 4. Two types of human bias (illustrations drawn from public domain world-wide web 

images).................................................................................................................................. 11 

Figure 5. Gestalt perception (Image from public domain world-wide web). .............................. 14 

Figure 6. Stereotypes and iconic symbols (image from the Federation of American Scientists, 

2006). .................................................................................................................................... 15 

Figure 7. Signal detection example (Heeger, 1998). ................................................................... 16 

Figure 8. Mackworth (1942) vigilance curve (human performance decrement over time)........ 17 
1 

1. Introduction 

This report discusses how human factors engineering (HFE) is a critical element in the process 

of sensor fusion. For this report, the definition of sensor fusion is the one created by the Joint 

Directors of Laboratories (JDL)1. With the JDL definition and models, we review each sensor 

fusion process step from an HFE perspective, we discuss how human factors variables play a 

significant role for each element of that process, and we discuss how enhancing the role of HFE 

in the sensor fusion process will result in a more effective human-machine system. Examples to 

illustrate these issues often refer to the military domain, military aviation, military intelligence, 

automated systems, or computer operations. 

1.1 Background 

1.1.1 Basic Concepts 

Sensor fusion is the process of collecting data, combining those data through a variety of methods, 

with a variety of sensing technologies, and presenting those data as an integrated product to a 

machine or a human. The basic advantage to fusing data is that (a) it should be faster and more 

effective in presentation than if the data were not fused and (b) combining data results in a richer, 

more intelligible output that is functionally greater than the sum of its parts. The following definition emphasizes the concept of fusion from a cognitive perspective, regardless of the type of sensor 

system involved: 

Fusion pools multiple bodies of evidence to produce a single body of evidence 

that emphasizes points of agreement and de-emphasizes points of disagreement 

(SRI2, 2006).

This definition of fusion emphasizes that the process of fusing data is essentially like a logical 

argument, in which principles, facts, and rules all combine to form and then persuade a conclusion. 

It is analogous to the testing of a hypothesis in the scientific process and is called an evidential 

operation (SRI, 2006); however, fusion is also selective in its processing of information and may 

not always follow a strict a priori logical approach. This is evident in the consideration of humanbased fusion of information where heuristics (rules of thumb) or recognition-primed decisions may 

sort information in ways that machines are not currently capable of doing. In principle, fusion may 

be accomplished by human or machine systems. 

Most of the discussion in this report relies on the fact that machines have electro-mechanical 

sensors and computer-based cognition, whereas humans have the five senses plus human cognition. Humans combine the input of their senses into a cohesive product through cognition, which 

 1Army, Navy, and Air Force science and technology directors 

2not an acronym 
2 

then can result in actions. As a discussion point, it is posited that humans operate in ways similar 

to machines, from the standpoint of combining input data and information into an output action. 

This system’s view recognizes differences as well as similarities between machines and humans in 

the acquisition and processing of data, as well as the final output products achieved by both. Some 

of these similarities and differences are discussed in this report. 

Traditional sensor fusion often involved detailed mathematical algorithms that controlled the 

processing of information to reach a complex goal such as detecting, classifying, and identifying 

a target (Ceruti, 2004). However, for the purposes of this discussion, the interaction of human 

cognitive processes in creating and using information is the focus of the discussion. For example, 

in the aviation environment, some sensors provide system status information, others provide 

mission information, while still others provide communications linkages, weapons status, or 

current enemy threat status. A similar process is followed in the electronic warfare environment 

where “multiple levels of abstraction, multiple sensors and various evidential operations all 

combine to predict changes in observables” (SRI, 2006). The original goal for electronic warfare 

and aerospace system’s sensor fusion was to combine all these sources of data into a cohesive 

whole from which a machine or human operator could draw in order to make more effective 

operational decisions (Bowman & Murphy, 1980). This approach for aerospace was in marked 

contrast to earlier aerospace systems where routines were fully automated (programmed or “hard 

wired” as in an early ballistic missile), and it was also in contrast to systems where federated or 

isolated sensors, acting as single sources and providing information for a single purpose, were 

usually monitored by a human operator (as in an early aircraft cockpit or an intelligence analyst 

monitoring one radio channel). Now, on advanced aircraft such as the F-22, the pilot manipulates 

the throttles to provide a thrust request from the central processing computer on the aircraft, and 

the computer in turn determines if that request can be complied with, based on fused input from 

sensors that monitor aircraft and external parameters. Those variables are in turn fused by a mainframe computer and the thrust request is completed or mitigated, which is very different from 

traditional manual control of that process. Likewise, modern electronic warfare systems may 

include sensor feeds from multiple sources that are combined on a single display format into a new 

image reflecting the qualities of all the sensors but not resembling any one sensor. 

1.1.2 Sensor Fusion as a Process 

In terms of human-computer interaction, current sensor fusion systems provide a cohesive 

picture, often using graphic and auditory technologies. Display screens, headsets, and auditory 

warnings are all examples of the final delivery means from which raw sensor data are fused into 

information and then channeled to the system operator in ways they can comprehend accurately 

and quickly. 

1.1.3 Sensor Fusion Defined 

The JDL’s (White, 1988) Sub-panel on Data Fusion has defined data fusion as
3 

...a process dealing with the association, correlation, and combination of data and 

information from single and multiple sources to achieve refined position and identity 

estimates, and complete and timely assessments of situations and threats, and their 

significance. The process is characterized by continuous refinements of its estimates and 

assessments, and the evaluation of the need for additional sources, or modification of the 

process itself, to achieve improved results... 

This definition provides guidance that describes a closed loop learning process that iteratively 

refines an assessment of a situation, much like a sensor-fused guidance system supplying 

navigation input to a missile. Note that the JDL definition provides a process statement that uses 

human information processing techniques of association, correlation, and a combination of data 

and information from numerous sources (sensor fusion) to identify the goals refine position and 

identity estimates and to quickly and iteratively make assessments of situations and threats while 

using a closed loop approach to achieve improved results. What is missing from this definition is 

the focus on how a human can perform this process, particularly in an environment where 

automated systems are providing the bulk of the information and the human is under the dynamic 

conditions of multiple tasking. 

A review of the data fusion model as proposed by the JDL describes the data fusion domain as one 

of parallel processing with sources such as “national, distributed, and local” providing intelligence, 

electronic warfare, sonar, radar, or other types of information sources as outside the central block 

that describes the five levels of processing (see figures 1 and 2). In addition, the human-computer 

interface is seen as outside the central data fusion domain (it is shown to the right side of the 

diagram). The levels are described in figure 1 and are shown in a model format in figure 2. The 

JDL model shown in figure 2 also includes another step which is user refinement (an element of a 

closed loop process). The two figures also show the changes that have occurred in the model over 

time. 

The JDL Model (1987-91)

and the draft revised model (1997)

Level 0 — Sub-Object Data Association and Estimation: Pixel/signal level data association and 

characterization 

Level 1 — Object Refinement: Observation-to-track association, continuous state 

estimation (e.g. kinematics) and discrete state estimation (e.g. target type and 

ID) and prediction

Level 2 — Situation Refinement: Object clustering and relational analysis, to include 

force structure and cross force relations, communications, physical context, etc. 

Level 3 — Significance Estimation [Threat Refinement]: Threat intent estimation, 

event prediction], consequence prediction, susceptibility and vulnerability 

assessment 

Level 4 — Process Refinement: Adaptive search and processing (an element of resource 

management) 

Figure 1. The JDL model, 1987-91 with 1997 revisions (Steinberg, Bowman, & White, 1999). 
4 

Figure 2. Revised JDL user model, 2003-2006 (Blasch & Plano, 2003). 

2. Purpose of Report 

The purpose of this report is to provide a general review of the sensor fusion process as defined 

by the JDL from the perspective of selected human engineering principles. 

3. Method 

3.1 Human Engineering Principles Applied to Sensor Fusion 

The discussion of how humans process information includes a discussion of each of four basic 

principles that apply to human cognition and subsequent action. These principles are listed next 

and form the structure for the discussion that follows regarding the specific elements of the JDL 

model: 

 Human Versus Machine Process Cycles Comparison 

 Human Processing of Information 

 Human Logic in the Processing of Information 

 Subjective Factors That Affect Human Processing of Cognitive Information, Data Risk, 

Data Value, and Cognitive Bias 
5 

3.1.1 Human Versus Machine Process Cycles Comparison 

Although both machines and humans work through an input-throughput-output cycle, the processes 

within that cycle are very different. Machines, as discussed in this report, represent complex computer-integrated systems that have the ability to collect data from sources or sensors and perform 

complex activities during complex circumstances, and such machines typically operate in an objective, digital environment where the flow of work is controlled tightly by the processes inherent to 

that machine’s design and programming. For example, a computer uses electricity generated by fuel 

combustion to create digital data through hardware and software. In contrast, a human metabolizes 

food to generate electricity to control neurons that process information collected and coded by the 

senses; this information is subsequently organized by the brain (a cognitive process). Both systems 

(machine and human) can yield a similar result in many cases, but understanding each of their 

processes allows a better understanding of the suitability of each of those systems in producing a 

defined overall system output and is also critical in allocating functions to each. In the case of many 

examples described in this report, a system usually consists of human and machine contributions 

toward a complex goal that is ultimately defined by the human. 

The human, as opposed to the machine, primarily functions in a subjective world where all information and action are controlled by numerous, subjective processes. This huge difference in basic 

operating system features provides for the strengths and weaknesses of each system (human versus 

machine) and the need remains for both to work together in an integrated and complementary 

fashion to achieve goals that neither could accomplish alone. 

The process styles mentioned are biologically or electrically based, but in processing machine 

data, sensor fusion systems use set filtering methods (e.g., Kalman filters) that “keep updating the 

system’s view of data changes in a very efficient way (i.e., they use the last updated result/data 

point and only use new information to update it, and then that new data point is later used as the 

reference point for the next round of new information) overall, which helps reduce the processing 

requirement on the system. However, it is critical that the algorithms are “human factored” so that 

they do not get progressively off track” (de Pontbriand, 2006; Kalman, 1960; Kalman & Bucy, 

1961). 

Other types of information filters include Markov, stochastic, or Bayesian approaches or even 

neural nets where software agents selectively seek specific information and update their models 

based on that information (these machines not only selectively use information sources but allow 

the system to tune itself through an understanding of measurement noise and the use of recursive 

processes where the system can learn by its mistakes) (Abdi, 2003; Pinheiro & Lima, 1999). 

However, many of these algorithms are based on fixed sets of logic, whether it is the consideration 

of Gaussian noise in the collection of data (Kalman filtering) or updating information based on 

fixed algorithms of a set distribution (other Stochastic methods) (Pinheiro & Lima, 1999). 

In the case of humans, the context for these cognitive processes is often based on not only recent 

or easily available data nor on probabilities but on other factors such as described by Klein as 
6 

external knowledge sources unavailable to machine systems. The recognition-primed decision 

model (RPDM) as postulated by Ross, Klein, Thunholm, Schmitt, and Baxter (2004) was a 

refinement of a model originally established in 1989 by Klein, Calderwood, and Clinton-Cirocco 

and describes the theory behind several human cognitive processes. The RPDM postulates that 

pattern matching, diagnosing a situation, and evaluating a course of action are all higher order 

processes that humans use in processing information and solving problems. Within that threeelement structure, other factors such as perception of a situation as typical or atypical then leads 

to a recognition involving expectancies: relevant cues, plausible goals, and typical actions. This 

approach is quite removed from machine logic where a more mechanistic, deductive process is 

often implemented. 

3.1.2 Human Processing of Information 

One of the processes illustrated in the JDL diagrams is parallel processing of information at several 

levels (Steinberg, Bowman, & White, 1999; Hall, 2005). While it is true that humans can process 

information in parallel, when that task is accomplished, real-time, overall task performance often 

suffers dramatically (Wickens, 1987). An analogy is the task of driving while one is talking on a 

cell phone, an action that can and is accomplished with regularity but with significant performance 

decrements in other tasks. The converse of this dual tasking problem is that people can either 

drive or talk on a cell phone with very good performance but not in parallel unless they wish to 

reduce their performance of one or both activities dramatically, even though both tasks may be 

heavily practiced (Cohen, Dunbar, & McClelland, 1990). This area of primary and secondary 

tasking is also complicated by the human sensory channels being used for that processing 

(Wickens, 1987). However, it is also possible for humans to process information in parallel with 

greater facility as that activity becomes more automated via repetition and practice (Wickens, 

1987; Cohen, McClelland, & Dunbar, 1990). Humans with a lot of experience in a certain area of 

activity are called experts, and after an activity is learned to a high level of expertise, it becomes 

more automated in nature. However, no matter what the level of human of expertise that an 

individual has in performing a task, that task must compete for limited time and mental resources 

in order to continue effectively (Sanders & McCormick, 1993). Humans can operate to some 

degree using parallel processing but not nearly to the degree that automated, mechanical systems 

can parallel process information. A specific model of how humans process information is 

discussed in the following sections. The Wickens’ model of attention (Wickens & Hollands, 

2000) is shown in figure 3. 

This model posits that our attentional resources (and by analogy, all our resources) are finite and 

through various mechanisms associated with human physiology and cognition, address problems 

through a closed loop process involving sensing data (short-term sensory store), perceiving data (as 

a sensor would for a machine), and then applying attention resources (energy, motivation, 

cognition, etc.) through an iterative process to reach an outcome (response). In turn, the closed 

loop nature of this action is the feedback process called learning. This model of human 
7 

performance is similar to that of the JDL process model but has a wide variety of differences that 

are discussed in this report. 

Figure 3. The Wickens' model of human attention (Wickens & Hollands, 2000). 

Human cognition or the process of thinking involves actions and outcomes that are currently 

orders of magnitude more complex than those of any machine. With a computer as an example 

of a typical machine, it contains an ON and OFF control and then an ever-broadening array of 

energy flow-switching and storage mechanisms that allow digital processes to occur at the speed 

of light via a binary, deductive, and digital logic system. However, with all that complexity, 

even the most powerful computers would be no match for an infant in terms of discovery and 

cognitive growth. Without the ability of locomotion (inferring continual and dramatically 

varying sensor input), self-sustainment of operating energy, and a diverse cognition logic basis, 

a machine cannot progress or learn at the same rate as the human in a variety of domains. 

3.1.3 Human Logic in the Processing of Information 

How do these human-machine examples relate to the process of sensor fusion in the formation of 

conclusions and then actions? 

If humans processed information purely in parallel, they would have to consider all input at the 

same time (which they cannot do since they are perceptually limited to only a few channels of 

information processing [visual, auditory, and proprioceptive]), and those channels are bandwidth 

limited. If they were processing the same items in series, they would have to consider each item 

as it contributes to the solution as a sum (a very time-consuming process). Neither approach is 

regularly used to reach human-derived decisions. 

Quite often, humans do not use the deductive logic process where premises follow directly to 

conclusions such as the method of formal proof that is used in mathematics or that often drives 
8 

common machine logic. Instead, humans tend to use a combination of deductive and inductive 

logic, as well as intuition and emotion to reach general conclusions; however, if formal logic is 

used, humans often use more of the inductive process (called the method of discovery) or infer 

from the specific to the general (Searles, 1956). 

It is also critical to note that humans process all information subjectively, even when it is objective 

in nature. If posed with the question, “Will the sun rise tomorrow?,” virtually everyone would 

indicate (and truly believe) that the sun will rise tomorrow, but that conclusion is based strictly on 

subjective information as reinforced by experience. Perhaps one person in a million could fully 

comprehend celestial mechanics and mathematics to the degree of producing an objective 

mathematical proof to describe that conclusion, but even that individual would have to rely on 

“objective data” derived by subjective human experience and intellect. In contrast, machines are 

strictly objective in terms of basic processes such as the binary effects of current being on or 

current being off as the basis for action. 

The method of deductive logic is more often used in machine processes where conclusions are 

made from the general to the specific. With a military example of machine logic, the following 

premises might be made by an automated system: (a) a tank is an object that moves; (b) a moving 

metal object can be detected by a radar sensor; (c) deductively, an item “tank” can be associated 

with hypotheses about what constitutes “tank”; (d) a “tank” object can also be tracked or observed 

by a sensor; (e) a machine can be programmed to confirm that the object is a “tank” if pre-existing, 

deductive criteria are met (speed, size, heat output, turning radius, etc.). Given this logic stream, 

that same machine could then transfer a sensor-driven pixel representation of a “tank” object to a 

screen (in the form of moving pixels). Then that movement of pixels can (with a screen display) 

be observed by a human who might then conclude that it was “a tank”. This same machine system 

might use elements of the deductive logic stream to assess the “tank” further and provide “conclusions” based on that sensor stream and logic to conclude that the object was a T-73 tank or an M60 tank. 

In comparison, if a human were observing that same “tank” image, s/he might analyze it for 

familiar patterns (pattern recognition, pattern matching) and then form tentative hypotheses and 

create a theory about the “object” tank. Carrying this example further, a human might predict an 

action performed by that tank that cannot currently be accomplished by a machine. 

S/he would do so using inductive logic processes, but s/he can also inductively extrapolate the 

concept and effects of a “tank” to a much greater degree than any machine. For example, a person 

may see the “tank object” and conclude that an attack could result against a friendly force at a 

distance of 3 kilometers across a small stream in 20 minutes—a conclusion not backed by 

deductive logic but that may be correct and was subjectively determined based on experience, 

stored knowledge, and very dynamic inductive processes. Humans may also mix and match 

logical methods and in doing so, do not follow a formal logical pathway of pure deduction or 

induction. For example, if the object is a T-73 tank, it could actually elicit a fear reaction from the 
9 

observer because it is likely being driven by an enemy crew with hostile inten-tions—a conclusion 

that could be programmed into a machine but only resulting in a warning or caution and not as the 

actual emotion felt by the observer. The element of fear induced by the “tank object” would feed 

into the information processing by that human observer and could even subjectively influence his 

or her decision-making process. 

The use of a strict deductive process is not possible for a human to accomplish because of the 

vast number of possible moving objects that could be a tank, so humans typically would not 

deductively identify a moving object as a tank versus a cow versus a cloud versus a tuning fork, 

but they would use an inductive process to pattern recognize and thus quickly determine what 

that object was based on their past experience (Duda, Hart, & Storck, 2001). This type of 

reaction to information is also mirrored in the expectancies category of Klein’s RPDM 

(discussed in section 3.1.1). 

In contrast to human thinking, machine pattern matching is deterministic and mathematical in 

nature and is bound by various set algorithms (Haskel, 2007). It typically is not capable of 

learning and prioritizing in real time to compensate for information loss, ambiguity, or deception. 

As a tactical illustration of this idea, a tank camouflaged by white sheets and big red balloons 

could conceivably halt or interfere with the process of automated detection by a machine because 

that particular camouflage approach simply would not have been chosen by a rational human when 

trying to deceive another human (based on “common sense”) and would be a poor camouflage 

choice for a human observer. However, a human would quickly recognize that deception because 

of his or her ability to inductively prioritize and weigh new or unusual information in real time. 

As humans, we take this process for granted, but machine logic cannot, and thus our abilities as 

humans currently vastly exceed machines in that regard. Our ability to prioritize and eliminate 

information is as valuable as our ability to collect and process that information. 

3.1.4 Subjective Factors That Affect Human Processing of Cognitive Information, Data 

Risk, Data Value, and Cognitive Bias 

The previous paragraphs have mentioned basic logical process differences between humans and 

machines, but human information processing contains considerably more complexity in how 

humans process information. A review of some of those processes follows. 

3.1.4.1 Data Risk 

A human using a machine-fused sensor system is completely dependent on sources of information 

that are removed from his or her own experience. If all data were created equally, this would not 

present such a problematic situation, but each bit of data available to a sensor fusion process has an 

attendant risk level for humans, with some data being quite a bit more risky than other bits of data. 

When data are collected first hand by the human, data risk can be assessed based on that human’s 

experience, but in the remoteness of a situation where information is far removed from the source, 

the credibility of those data is usually not included with the presentation of those data. This is all 
10 

too common an issue when one is processing printed, verbal, or visual information. Each type of 

information has various levels of face validity acceptance, as established by the user, and that level 

of validity drives the acceptance or rejection of the data (Meister, 1985). A pilot tapping a finger 

on a mechanical altitude display or a computer operator logging out and then back in are examples 

of how humans test information and assess its risk level. Objectively, a machine has no consequence of risk, i.e., it cannot die from a mistake. 

3.1.4.2 Data Value 

In terms of information value, simply declaring something to be “the way it is” is low value and 

high risk, rather than a good logical basis for action. That is, unsupported information is lower 

value than if that information were presented with the use of some type of objective, externally 

supported references, citations, or the concurrence of subject matter experts (SMEs). 

Humans weigh the value of any data using very complex rules, some of which are not consciously 

apparent to the human making the decision. Processes of verification of information involve not 

only logic but circumstance and even an understanding of the motivation someone may have in 

providing those data (Salvendy, 1987). This may also be thought of in terms of signal detection 

theory, where the criterion point being set is based on the possible magnitude of con-sequences of 

a particular type of occurrence. This weighing of data is important for any subjective decision, and 

one way in which this type of data could be viewed is from the perspective of bias. 

3.1.4.3 Cognitive Data Biasing 

The term “cognitive bias” implies a conscious or unconscious attempt to weigh an outcome 

according to some type of pre-existing rule or criterion. Bias may or may not cloud a person’s 

judgment in regard to making decisions. The issue is that cognitive bias cannot be easily 

dismissed and can influence human judgments independently of facts. 

Cognitive bias is one of the factors that control human actions to some degree, and in this regard, 

we are very different from a machine that does not operate under that type of subjective condition. 

Our sensory systems are all susceptible to unconscious and uncontrollable biases, and we call 

certain of these perceptual biases “illusions”. Most everyone is familiar with optical illusions in 

which objects do not appear to represent reality, depending on perspective (figure 4). Simply 

knowing that such biases exist does not make them go away; they are present even in the face 

of our logical knowledge that tells us what information we think should be presented. As an 

example, when we view an object such as a set of railroad tracks moving away to the horizon, our 

visual sense limitations would lead us to believe that the rails converge at a distance, but logically, 

we know that they do not. As such, we must also recognize that humans are biased not just in the 

perceptual sense but in the cognitive processes that we employ to collect and process data. The 

work of Kahneman and Treisman (1984) covers a number of cognitive biases as well as ways that 

humans process information through rules of thumb, technically known as heuristics. A heuristic 
11 

is a pattern of thinking which we apply cognitively, in much the same way as Klein and his RPD 

model would describe as an expectancy or plausible goal. 

Zoellner Optical illusion bias (the vertical lines are parallel) 

Cognitive bias, are the tires squealing on these cars? 

Figure 4. Two types of human bias (illustrations drawn from public 

domain world-wide web images). 

Conversely, a bias can also be a result of learned behavior, which empowers us to make a good 

decision with a moderate or even low level of information being present. For example, the sound 

of squealing tires immediately attracts attention because of our prior experience and a bias toward 

associating that precursor sound with an impending vehicle accident (figure 4). That potential 

accident may or may not occur, but our previous experience and a bias to trust our experience 

allows a quick, often life-saving decision to be made in the absence of additional information. 

Like the startle response to a loud noise, our action is involuntary but is evolutionarily adaptive, 

and with the RPDM and the previous example, the sound of squealing tires is not only a relevant 

cue to action but an expectancy of things to come. 

The human user of sensor systems data often can conclude outcomes based not on vast amounts of 

stored information like a computer but on critical, linked memories that indicate an overall trend. 

This is a critical function of human cognition in that we quite often make judgments based only on 

fragments of information, and sometimes those judgments are correct. 
12 

4. Results 

4.1 The Five Levels of the JDL Model of Sensor Fusion 

Each of the sensor fusion levels has characteristics, and each of those, in turn, has human factors 

implications. The following sections discuss the human factors of human interaction with sensor 

processing. The JDL Fusion Model levels are 

• Level 0 – Pre-processing 

• Level 1 – Object Refinement 

• Level 2 – Situation Refinement 

• Level 3 – Threat Refinement 

• Level 4 – Process Refinement 

• Level 5 – User Refinement 

4.1.1 JDL Level 0, Pre-processing: Signal or Feature States 

This function is largely out of the realm of human engineering and is one of machine design at the 

hardware and software level. At this stage, the organization and identification of data are based 

more on machine capabilities than on human limitations or capabilities. The ability to detect a 

target within a forest or the range of a sensor is driven more by physics than by human psychophysics. However, the mechanical features of any sensor or data collection system should consider 

the capabilities and limitations of human users and should address those inherent values in their 

design specifications and data products so as not to overload the human who receives that information stream. Sensor systems should output their information in ways to maximize human 

recognition and processing. 

The collection and use of data of any type should consider the ultimate user of those data, and the 

product of that process should match the abilities of its user. For example, if an electro-optical 

machine can scan 500 square miles of territory per minute, it must store that information and 

provide it to an image analyst in a format s/he can understand and comprehend, and most likely, 

this would not be at the rate that it was originally collected. However, if the purpose of that scan 

was to identify any large troop movements within that area, as defined by a pre-set moving target 

identification (MTI) algorithm, then that type of troop movement could be automatically isolated 

from the less valuable mass of data and transmitted to the analyst in seconds, thereby allowing him 

or her to make a critical determination of intent without ever having to view all the collected data. 

In this sense, pre-processing and sensor fusion have refined the data, converted them to useful 

human information, and communicated that information to an analyst in a form that s/he can 
13 

understand and interpret in an order of magnitude less time than if s/he were forced to observe the 

entire stream of information. 

One of the biggest interface mismatches at Level O processing is between what a machine can 

generate and what a human can absorb. Time, as well as the quantity and quality of information, 

must be matched so that information is fed to the human at a rate that the human can actually 

use. 

4.1.2 JDL Level 1, Object Refinement: Entity Parametric and Attributive States 

This section is the core step of where the JDL model directly interacts with the human information 

refinement processes. In order to refine, define, or organize objects, human perception is the first 

step of that process. Human perception steps as described by Foley and Moray (1987) are discussed in terms of the perception of information brought to us by our senses and include information about sight, hearing, rotation, falling and rectilinear movement, taste, smell, touch, vibration, 

pressure, temperature, cutaneous pain, position and movement (kinesthesia). Unfortunately, very 

few of these sensory faculties are available to help us make decisions in a typical sensor fusion 

environment, which probably provides only visual imagery and perhaps sound information. For 

example, an unmanned aircraft system (UAS) operator may only see a video screen image transmitted by his or her UAS, with no aircraft sound, no movement, and no other sensory feedback 

from that system other than the visual channel. In comparison to actual experience, the information contained in that control environment is very limited and superficial. 

In contrast, if that same operator were able to be in that UAS, s/he would see what it sees, hear its 

operation noise, smell the UAS exhaust, the engine vibration, observe the firing of a missile and 

its effects, and would thus build an integrated and richer mental model of the concept “UAS”. 

However, in the typical sensor fusion operations environment, nearly all those sources of information about the UAS are not available. The object UAS becomes abstracted when presented by 

filtering and a single channel presentation (video screen). This principle of abstractness has often 

led to a disconnection with the object being observed remotely and is seen as a problem with work 

stations in UAS or other robotic systems (Cooke, Pringle, Pedersen, & Connor, 2006). 

All sensor systems, whether they are machine or human, create multiple levels of abstraction 

since they represent reality but are not the reality (Ceruti, 2004). Most information systems, 

including sensor fusion systems, would be more effective if they were more immersive in nature 

and provided complementary sources of information that mirrored reality. In effect, the results 

would be less abstract and less symbolic. If a UAS operator could actually be on board a UAS, 

his or her awareness of that system would be enhanced, but that connectedness to the UAS is by 

definition impossible to achieve. Therefore, the only way to approximate that state is for the 

operator to have an interface that duplicates the more important system performance features that 

might be found within and about that UAS. The following object refinement human processes 

are discussed in relation to level 1 JDL fusion and are believed to be critical to proper human 
14 

understanding of a sensor-fused or other mechanical system, as well as being critical to the 

cognitive processing of any information. 

4.1.2.1 The Speed-Accuracy Trade-off 

The speed-accuracy trade-off is easily summarized since actions can be fast or accurate but not 

extreme in both directions at the same time (Sanders & McCormick, 1993). Skilled performance 

of an expert can be fast and accurate in comparison to that of a novice, but even within that expert, 

there is always a trade-off between these competing factors in a conflicting and competing 

resource allocation process (Proctor & VanZandt, 1994). 

4.1.2.2 Top-Down Processing 

Top-down processing is a series of human cognitive processes that involve the processing and 

refinement of information (Klapuri, 2007; Parasuraman, 2006). These processes include the 

Gestalt process, perceptual stereotypes, signal detection, and attention (Wickens, 1987). A 

discussion of Gestalt processes, stereotypes, and signal detection follows. 

4.1.2.3 Gestalt Processes 

Gestalt processes refer to the human ability to see patterns in data or imagery that intuitively seem 

to fit, without detailed analysis or thought (Koffka, 1935). These are a form of pattern recognition 

that is quick and fairly effortless. In principle, Gestalt perception is often associated with visual 

processes, but it is not just confined to the visual sense. 

Gestalt principles are a very powerful tool to quickly discern patterns in information. For example, 

in the scene in figure 5, the Gestalt perspective quickly identifies objects that do not “fit” in the 

picture, so both the unusual conically shaped tree and a concealed tank are visible at a glance 

because of pattern recognition and matching. The “Gestalt” of this picture is that of two objects 

that do not “fit” into the background and thus are quickly identified as something other than 

background. 

Figure 5. Gestalt perception (Image from public domain world-wide web). 
15 

Stereotypes are learned stimuli that can allow an observer to define objects quicker than if the 

information associated with those objects were not coded in this manner. This is because they 

have a certain compatibility, agreed to by a large portion of a population (Bullinger, Kern, 

Muntzinger, in Salvendy, 1987). 

Common stereotypes include color coding (green = good, red = hot, red force = enemy, blue force = 

allies). This type of perceptual coding system (Proctor & VanZandt, 1994) is used widely within the 

Army and is typical in the coding of force types (cavalry, artillery, tanks, platoon, battalion, etc.). 

Stereotypes often have no inherent information but rely on memorization in order to achieve their 

meaning; however, a positive trend in symbolic stereotyping is to use objects (icons) that resemble 

the objects that they portray, so for example, a tank icon would be displayed with a track, a basic 

tank chassis outline, and a gun barrel, each bit of information contributing to the concept “tank,” 

and to a limit, this provides faster and more accurate detection and discrimination by the observer. 

Stereotypes also have limitations in the amount of information they can convey and they primarily 

focus on quick, not necessarily accurate, discriminations. As figure 6 shows, many military symbols 

do not have realistic (iconic) qualities; thus, they must be memorized for content. However, some of 

the symbols do have iconic qualities (e.g., helicopter blades or the outline of a tank [armor]). 

EXAMPLES OF UNITS IN DIVISIONS

Figure 6. Stereotypes and iconic symbols (image from the Federation of 

American Scientists, 2006). 
16 

4.1.2.4 Signal Detection Theory 

The term “signal detection theory” was first used in electronic signal communication in the 1950s 

and subsequently was adopted and expanded greatly by psychologists to describe the perception of 

a signal in the presence of noise (Shannon, 1948). For psychologists, the term “signal” has been 

broadened to tasking, goals, or perception, while the term “noise” indicates any factor, environmental or other, that interferes with that process. 

This concept of signal detection is typically diagrammed with the use of two normal curves. One 

curve represents the signal (goal, task, perceived object) and the other curve represents noise or 

anything that interferes with that process of perception; the intersection of the two curves represents the signal-to-noise relationship for a circumstance. If the curves are widely separated (lower 

part of figure 7), there is little interference with the task or goal, but as the noise curve further 

intrudes on the task or goal, the greater the level of masking and thus, the greater difficulty of 

performing the task or meeting the goal (illustrated by the upper part of figure 7). When the two 

curves overlap completely, the goal or task is not possible. This concept is a powerful analog for 

human performance in terms of human workload. 

Figure 7. Signal detection example (Heeger, 2007). 

4.1.2.5 Attention and Vigilance 

Humans, unlike machines, do not operate on a set or binary type of scale (that is, on or off). As a 

system, they operate through a variety of operating continua, one of which is called attention. For 

any task, a person’s involvement with that task is driven by his or her level of attention. Generally, 

poor attention equals poor performance, proper attention equals proper performance, and overattention can lead to a cognitive myopia or attentional tunneling (excessive focus on too small an 

area). 

The presentation of any data involves time, and one area that humans are not noted for is their 

ability to sustain vigilance for extended periods of time. For example, during World War II, sonar 

operators were shown by Mackworth to have greatly diminished vigilance after only 30 minutes of 

operation, often missing vital signals because of boredom and a vigilance decrement (Parasuraman, 

2007). UAS operators often are confronted by long periods of inactivity followed by frantic action, 
17 

followed again by inactivity, and this work environment is not conducive to maintaining vigilance. 

Performance decrements have been modeled and reductions in effectiveness predicted (Cooke et al., 

2006), and there is a large body of research that concerns the loss of performance with sustained 

vigilance. A sample of research showing a short-term vigilance decrement is shown in figure 8 and 

is called the Mackworth (1948) vigilance curve. This curve involved the results of detecting changes 

in the ticking of a clock over time. Essentially, it showed that for boring, repetitious, or infrequent 

tasks, performance suffers greatly. 

Contrary to vigilance decrement, sensor fusion system users must be presented with information 

for a time adequately long enough to process that information (minimal response time limits). 

Continuity of information flow at the right pace is critical in matching machine feeds of information and tasking to the abilities of the user. 

Mackworth Vigilance Data

0

5

10

15

20

25

30

35

0 20 40 60 80 100 120

Minutes

Percent of Targets 

Missed

Figure 8. Mackworth (1948) vigilance curve (human performance 

decrement over time). 

4.1.3 JDL Level 2, Situation Refinement 

After the perceptual elements have been addressed by the analyst at Level 1 processing, situation 

refinement begins to become more dominant at the JDL Level 2 (situation refinement level). At 

this stage, the human processes of cognition (rather than just perception) become more involved. 

It is also important to note that there is no clear defining line between human perception and 

cognition, and perceptual biases can easily cross over into cognitive processes and conclusions. 

Raw data from external sources are pre-processed by mechanical systems in Level 0, perceptually 

arranged and manipulated in Level 1, and are then evaluated in Level 2. As an example from a 

military intelligence analyst environment, the six functions associated with this level are (Blasch & 

Plano, 2003) 

1. Aggregation of information into entities or force structures, 

2. Determination of relationships and working rules, 

3. Interpretation of the actions of entities of forces, 
18 

4. Determination of the overall purpose of large and small actions, 

5. Hypotheses of current and future activities and testing those hypotheses, 

6. Resolution of anomalous factors and the consideration of deception. 

Each of the six processes within Level 2 builds an awareness of a situation and does so iteratively 

as information is brought together, tested, and evaluated. There is never actually an end product 

for situation awareness (SA); it is an ongoing, dynamic process that continues even after the resolution of a particular situation. 

Before discussing the six individual processes of Level 2, it would make sense to define SA since 

it is an awareness of information in the first place that then proceeds to all further processes. 

Dr. Mica Endsley’s (1988) definition of SA is the commonly accepted origin of that term: 

The perception of the elements in the environment within a volume of time and space, the 

comprehension of their meaning, and the projection of their status in the near future. 

The user of a sensor fusion system must perceive information that is dependent on that system’s 

perceptual abilities and limitations. S/he must be able to comprehend that information (thus, much 

of the discussion here about the processes of comprehension and cognition), and then s/he must be 

able to project or predict a possible outcome, based on that perception and comprehension. Both 

SA and sensor fusion systems have a similar closed loop process and use an input-throughputoutput cycle that is familiar to industrial engineers and human factors practitioners. Considering 

this overall perspective of SA, the following steps of the JDL level 2 model are discussed in terms 

of human factors principles. 

4.1.3.1 JDL Level 2, Step 1: Aggregation of Human-Related Data 

Of the six processes, the process of aggregation is readily suitable for integration with current 

automated systems. The ability to collect data and determine correlations or separate data into 

categories is most easily done by automated means as directed by the human. For example, given 

a large collection of enemy combatants and the goal of finding a key person(s) from that database, 

a database sorting program can immediately sort those players into categories as defined by a 

military intelligence analyst. Factors associated with geography, age, birth place, combatant 

service affiliation, notable activities that the person has been involved with, places where s/he has 

lived, personnel s/he has been in contact with, education or schooling affiliations, friends, family 

lineage, military specialty, and a wide variety of known variables can all be sorted and organized 

by an automated routine. Aggregation could also apply to combination of sensor data from a 

variety of sources, the prioritization of those data, and the use of those data to build a model or 

reach a conclusion. 

4.1.3.2 JDL Level 2, Step 2: Determination: Human Relationships and Attributes

Determination of relationships, attributes, and working rules is an area where automation could 

provide help to the human in the loop. With the military intelligence domain as an example, 
19 

enemy personnel relationships can be provided by the correlation of information (such as which 

enemy personnel might be related to whom), which set of personnel might be associated with 

another set of personnel in a chain of command, or a general understanding of the psychological 

or sociological working rules for determining such enemy activities. Nevertheless, machine-based 

associative algorithms for enemy personnel matching could provide clues to human analysts of 

correlations not obvious from the mere collection of human intelligence (HUMINT) in the field. 

4.1.3.3 JDL Level 2, Step 3: Interpretation 

This sub-process seeks to answer the question why was an action taken, and it often involves 

human subjective data. It seeks to answer questions such as “What role did this action have?” or 

“What effect would an action like this have on some defined process?” It is attempting to derive 

meaning from actions. It is almost entirely a cognitive process that is aided little by current 

automated systems. One way in which an automated system might be able to contribute to 

answering these types of questions could be in the graphic presentation of magnitude of effects 

or associations with the use of pie charts or graphs that show trends. By compiling numeric data 

and reporting them in a graphic way, an analyst can improve his or her overall understanding. 

4.1.3.4 JDL Level 2, Step 4: Determination: Large and Small Actions and Their Effects 

This process is like construction of a jigsaw puzzle with the analyst placing piece after piece of the 

puzzle together to assess its impact on the overall picture. Not all information is created equally; 

some provides higher information content than others. Human beings have a fairly good ability to 

generalize from small pieces of information and to determine a sense of proportion of the value of 

information. This intuition process is based on past knowledge and often can manifest itself in 

conclusions with only a small amount of information being required; however, it can also lead to 

quick conclusions not supported by facts. 

4.1.3.5 JDL Level 2, Step 5: Hypotheses: Testing of Human-Related Information 

The formation of hypotheses by the sensor fusion information user is perhaps one of the most 

important cognitive tasks that s/he must perform, and that testing is often performed with 

subjective data. Often, a checklist or matrix approach can be used by the analyst to evaluate 

(but not prove) a hypothesis. In many cases, intuition alone is the basis for decision, whereas 

some more formal and structured method of testing hypotheses should be encouraged. 

4.1.3.6 JDL Level 2, Step 6: Resolution: Situational Human Factors 

This process often involves unusual cases where ambiguity is very high. If a situation does not 

appear to make sense or fit an established pattern, it very well may be some act of deception 

accomplished deliberately to raise doubt, promote confusion, or preempt the use of resources in 

its investigation. This aspect of sensor fusion has no machine analog since machines have not yet 

been created to lie, subjectively obscure, or in other ways purposefully distort information. The 

creation of decoys, setting up false troop movements, creation of meaningless activity, propaganda 

releases, hacking, and other forms of deception are often challenges to military sensor fusion 
20 

systems users because they impugn, distort, or destroy the quality of information that the sensor 

fusion system is transmitting. 

4.1.4 Level 3, Fusion: Threat Refinement, Estimation, Prediction, and Utility 

This stage of the sensor fusion process begins to isolate causes, to predict outcomes, and to assess 

the utility of those actions. At this stage of sensor fusion, the human and the machine are taxed 

with interpolating data, creating predictive outcomes, and then determining if those predictions 

have utility in comparison to the real-world events they are modeling. The third step of SA, 

“prediction,” is critical in this stage. 

4.1.5 Level 4, Fusion: Process Refinement and Human Learning 

Reviewing the sensor fusion process from start to finish is also a learning process and involves 

training. Since training is also a cognitive process, psychologists or human factors experts should 

be consulted when a training plan or system is being created. Improvement and optimization of 

any process is the domain of industrial engineering, but the psychological elements of learning are 

taught as basics to human factors practitioners. The learning process should always be closed 

loop, that is, iterative and repetitive in order to continually improve a product or a process. 

4.1.6 Level 5, Fusion: User Refinement 

This process should involve far more than visualization, but it is currently associated with 

visualizing enhancements of the other four levels. This stage also reflects the ability of the 

user to customize or tailor the process used to achieve the sensor fusion system goals. It must 

be remembered that human beings have five sensory channels, each of which provides information to support cognitive activities. If a system interface is limited only to visual processes, it is 

inherently disconnected with how humans naturally process information. It is understandable that 

much of our information is acquired visually as we sit in front of a computer screen and observe 

data products; however, our ability to learn in the real world is greatly enhanced by our using the 

rest of our sensory channels to process information. The following elements could be added to 

tools for user enhancement beyond the most common visual channel and primarily reflect future 

technological or human engineering approaches to fusing data from complex sources such as 

HUMINT. 

4.1.6.1 Auditory Feedback and Learning Human-Machine Comparisons 

The human auditory channel is a rich, high-density communication medium. In the analysis of 

speech alone, many linguistic principles can be used in providing feedback and learning through 

sensor fusion. However, linguistics alone is a very complex study and not currently very successful 

in terms of machine processing, other than simple word-for-word or phrase translator systems. In 

contrast, human observers using logic and experience can extract meaning from text or spoken 

language which is veiled by semantic features such as sarcasm or formalism that contain a very 

great amount of information but would not be detected by machine translation systems. Psycholo-
21 

gists understand that inflection, the choice of words, and the manner in which they are delivered all 

contain huge amounts of information that are lost when those words are merely written. In this 

respect, human perception of auditory feedback, when understood in the context of linguistic and 

semantic rules, can provide a human with far better information conclusions than a machine can. 

4.1.6.2 Sociological and Biodynamic Factors 

As humans who live in a sociological setting, posture, gesture, physical movement, and internal 

bio-physiological factors can tell us a lot about human behavior. Body language is a colloquial 

term for an externally driven, sociologically influenced communication process. It is entirely 

possible that a sensor-fused system could even provide cues from posturing, gestures, or movements that could indicate critical details such as deception and denial. In terms of human perception and cognition, simply observing an enemy’s spoken language over time may contribute to a 

sociological perspective that helps an analyst better in fusing complex information. In the past, 

lie detectors were used to assess, from a physiological perspective, the presence of deceptive 

techniques with the use of only one type of electro-physiological measurement (electromyography). In the future, powerful, automated routines could be developed to assess deception and 

denial by combinations of bio-analytical routines involving eye movement, posture, limb movement, or physiological measures such as electro-cardiogram, electro-encephalography, or electromyography. 

5. Summary Discussion 

This report has briefly covered the essential elements of the JDL definition of sensor fusion and 

discussed how, for each of those elements, human factors can contribute valuable human-centered 

information to those processes and products of sensor fusion. The information conveyed in this 

report was presented to demonstrate that human engineering principles, the principles of human 

psychology, cognition, learning, and an understanding of human behavior can all help create a 

better method of fusing data into useful information. 

6. Future Directions 

The tasking demands required for many military operations are bound to become more, rather than 

less, complex over time. Asymmetric, small-scale war operations in a network-centric environment provide personnel with the risk of being lost in a sea of data, while having to anticipate an 

enemy’s action that is not traceable by simple observation of battalions of tanks or thousands of 

troops en masse. In addition, being removed from direct, experiential contact with the enemy and 
22 

the ability to employ all of one’s senses in assessing information can result in superficial judgments based on information that is brittle in context, rather than rich in content. In addition, the 

current warfare environment still has huge risks associated from enemy combatants that do not 

follow many of the historic rules of warfare, nor do they provide traditional trails of information. 

In order to combat this trend, technology and sensor-fused systems have been implemented to 

compensate for this change in direction. 

Planning for conventional military operations against conventional military forces may not be the 

common battle environment for this generation, but that process must adapt to this new environment in order to maintain the upper hand in maintaining capable military strength. The use of 

sensor-fused systems has only now begun to emerge from a beginning in rudimentary tracking 

algorithms to the concept that sensor fusion may provide a very powerful tool in synthesizing and 

simplifying information. This fusion process may have more inherent value than matching large 

numbers of conventional military forces against each other in a well-defined tactical environment. 

If knowledge is power, sensor fusion may be a powerful tool in future military operations. 
23 

7. References 

Abdi, H. Neural Networks. In M. Lewis-Beck, A. Bryman, T. Futing (Eds), Encyclopedia for 

Research Methods for the Social Sciences, pp. 792-795; Sage: Thousand Oaks, CA, 2003. 

Blasch, E.; Plano, S. Level 5: User Refinement to Aid in the Fusion Process; In B. Dasarathy 

(Ed.), Multisensor, Multisource Information Fusion: Architectures, Algorithms, and 

Applications, 2003. 

Bowman, C. L.; Murphy, M. S. Description of the VERAC N-Source Tracker/Correlator; Rep. 

R-010-80; Naval Research Laboratory: Washington, DC, 1980. 

Bullinger, H. J.; Kern, P.; Muntzinger,W. F. Design of Controls. In Salvendy, G., Handbook of 

Human Factors. John Wiley and Sons: New York, 1987. 

Ceruti, M. G. Ontology for Level-One Sensor Fusion and Knowledge Discovery Space; Naval 

Warfare Systems Center, Code 24121, San Diego, CA, 2004. 

Cohen, J.D.; Dunbar, K.; McClellen, J.L. On the Control of Automatic Processes: A Parallel 

Distributed Processing Account of the Stroop Effect. Psychological Review 1990, 97(3), 

332-361. 

Cooke, N. J.; Pringle, H. L.; Pedersen, H. K.; Connor, O. Human Factors of Remotely Operated 

Vehicles. Advances in Human Performance and Cognitive Engineering Research, 7, Ed 

Eduardo Salas, Elsevier, Amsterdam, 2006. 

dePontbriand, R. Personal communication; U.S. Army Research Laboratory: Aberdeen Proving 

Ground, MD, 2006. 

Duda, R. O.; Hart, P. E.; Storck, D. G. Pattern Classification (2nd edition), ISBN 0-471-05669-

3 Wiley: New York, 2001. 

Endsley, M.R. Design and Evaluation for Situation Awareness Enhancement. In Proceedings of 

the Human Factors Society, 32nd annual meeting, pp. 97-101, 1988. 

Federation of American Scientists. Stereotypes and Iconic Symbols; retrieved from www.fas.org., 

2006. 

Foley, P.; Moray, N. Sensation, Perception and Systems Design. In G. Salvendy (Ed.), The 

Handbook of Human Factors. John Wiley and Sons: New York, 1987. 

Hall, D. L. Challenges in Data Fusion: Dirty Secrets, Current State of Technology and a Research 

Roadmap. Retrieved from www.ics.psu.edu/springnotes/DataFusion2005.pdf, March 12, 2005. 

Haskell. Web site: http://www.haskell.org/tutorial/patterns.html, 2007. 
24 

Heeger, D. Retrieved from http://images.google.com/ imgres?imgurl=http://www-psych. 

stanford.edu/~lera/psych115s/notes/signal/ir2.gif&imgrefurl=http://www-psych.stanford. 

edu/~lera/psych115s/notes/signal/&h=270&w=262&sz=8&hl=en&start=2&tbnid=zOpLJhf8

00dAdM:&tbnh=113&tbnw=110&prev=/images%3Fq%3Dsignal%2Bdetection%2Bcurves

%26gbv%3D2%26ndsp%3D20%26svnum%3D10%26hl%3Den%26safe%3Dactive%26sa%

3DN, 2007. 

Kahneman, D.; Treisman, A. Changing Views of Attention and Automaticity. In R. 

Parasuraman, D.R. Davies, and S.J. Beatty (Eds), Varieties of Attention, pp 29-61, Academic 

Press: New York, 1984. 

Kalman, R. E. A New Approach to Linear Filtering and Prediction Problems. Transactions of 

the ASME - Journal of Basic Engineering 1960, 82, 35-45. 

Kalman, R. E.; Bucy R. S. New Results in Linear Filtering and Prediction Theory. Transactions 

of the ASME - Journal of Basic Engineering 1961, 83, 95-107. 

Klapauri. Web site. http://www.cs.tut.fi/~klap/iiro/topdown.html, 2007. 

Klein, G.A. Recognition-Primed Decisions in Advances in Man-Machine Systems Research, 

W.B. Rouse, ed.; pp 47-92, JAI Press, Inc.: Greenwich, CT, 1989. 

Klein, G.A.; Calderwood, R.; Clinton-Cirocco, A. Rapid Decision Making on the Fire Ground. 

Proceedings of the Human Factors and Ergonomics Society, Vol 1, pp 576-80, 30th annual 

meeting, Dayton OH, 1986. 

Koffka, K. Principles of Gestalt Psychology; Harcourt & Brace: New York, 1935. 

Mackworth, N.H. The Breakdown of Vigilance During Prolonged Visual Search. Quarterly 

Journal of Experimental Psychology 1948, 1, 6-21. 

Meister, D. Behavioral Analysis and Measurement Methods; John Wiley and Sons: New York, 

1985. 

Parasuraman, R. Designing Automation for Effective Use by Human Operators. Cognitive 

Science Lab, Catholic University of America: Washington, DC, 2006. 

Parasuraman, R. Cognitive Engineering Psyc 530, Vigilance. Catholic University of America: 

Washington, DC, 2007. 

Pinheiro, Pedro; Lima, Pedro. Bayesian Sensor Fusion for Cooperative Object Localization and 

World Modeling. Instituto de Sistemas e Robótica, Instituto Superior Técnico Av Rovisco 

Pais 1, 1049-001 Lisboa, Portugal {pmpin@mega.ist.utl.pt, pal@isr.ist.utl.pt}, 1999. 

Proctor, R. W.; VanZandt, T. Human Factors in Simple and Complex Systems. Allyn and 

Bacon: Boston, 1994. 
25 

Ross, K.G.; Klein, G.A.; Thunholm, P.; Schmitt, J.F.; Baxter, H.C. The Recognition-Primed 

Decision Model, Military Review; U.S. Army Combined Arms Center: Fort Leavenworth, 

KS, July-August 2004. 

Salvendy, G. Handbook of Human Factors, Wiley inter-science publication. John Wiley & 

Sons: New York, 1987. 

Sanders, M.S.; McCormick, E.J. Human Factors in Engineering and Design. McGraw-Hill: 

1993. 

Searles, H.L. Logic and Scientific Methods, 3rd edition. The Roward Press Co.: New York, 

1956. 

Shannon, C. E. A Mathematical Theory of Communication, Bell System Technical Journal, pp 

379-423 and 623-656, 27 July and October, 1948. 

SRI International. Evidential Reasoning Operations. Menlo Park, CA: Retrieved from 

http://www.ai.sri.com/~gister/er-ops.html#fusion, August 30, 2006. 

Steinberg, A. N.; Bowman, C. L.; White, F. E. Revisions of the JDL Data Fusion Model. SPIE 

Proceedings: Sensor Fusion: Architectures, Algorithms and Applications III 1999, 3719, 

430-441. 

White, F. E. A Model for Data Fusion. Proceedings of the First National Symposium on Sensor 

Fusion, 2, 1988. 

Wickens, C. D. Information Processing, Decision-Making and Cognition, in Salvendy, 

Handbook of Human Factors; John Wiley and Sons: New York, 1987. 

Wickens, C. D.; Hollands, J. Engineering Psychology and Human Performance (3rd ed.); 

Prentice-Hall: Upper Saddle River, NJ, 2000. 
26 

Acronyms 

F-22 Fighter, Model 22, U.S. Air Force 

HFE human factors engineering 

HUMINT human intelligence 

JDL Joint Directors of Laboratories 

MTI moving target indication 

RPDM recognition-primed decision model 

SA situation awareness 

SME subject matter expert 

SRI not an acronym 

UAS unmanned aircraft system (formerly called an unmanned aerial vehicle [UAV]) 
27 

NO. OF 

COPIES ORGANIZATION

 1 DEFENSE TECHNICAL 

 (PDF INFORMATION CTR 

 ONLY) DTIC OCA 

 8725 JOHN J KINGMAN RD 

 STE 0944 

 FORT BELVOIR VA 22060-6218 

 1 US ARMY RSRCH DEV & ENGRG CMD 

 SYSTEMS OF SYSTEMS INTEGRATION 

 AMSRD SS T 

 6000 6TH ST STE 100 

 FORT BELVOIR VA 22060-5608 

 1 DIRECTOR 

 US ARMY RESEARCH LAB 

 IMNE ALC IMS 

 2800 POWDER MILL RD 

 ADELPHI MD 20783-1197 

 1 DIRECTOR 

 US ARMY RESEARCH LAB 

 AMSRD ARL CI OK TL 

 2800 POWDER MILL RD 

 ADELPHI MD 20783-1197 

 1 DIRECTOR 

 US ARMY RESEARCH LAB 

 AMSRD ARL CS OK T 

 2800 POWDER MILL RD 

 ADELPHI MD 20783-1197 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR ML J MARTIN 

 MYER CENTER RM 2D311 

 FT MONMOUTH NJ 07703-5601 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MC A D DAVISON 

 320 MANSCEN LOOP STE 115 

 FT LEONARD WOOD MO 65473 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MD T COOK 

 BLDG 5400 RM C242 

 REDSTONE ARSENAL AL 35898-7290 

 1 COMMANDANT USAADASCH 

 ATTN AMSRD ARL HR ME J HAWLEY 

 5800 CARTER RD 

 FT BLISS TX 79916-3802 

NO. OF 

COPIES ORGANIZATION

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MM DR V J RICE 

 BLDG 4011 RM 217 

 1750 GREELEY RD 

 FT SAM HOUSTON TX 78234-5002 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MG R SPINE 

 BUILDING 333 

 PICATINNY ARSENAL NJ 07806-5000 

 1 ARL HRED ARMC FLD ELMT 

 ATTN AMSRD ARL HR MH C BURNS 

 BLDG 1467B ROOM 336 

 THIRD AVENUE 

 FT KNOX KY 40121 

 1 ARMY RSCH LABORATORY - HRED 

 AWC FIELD ELEMENT 

 ATTN AMSRD ARL HR MJ D DURBIN 

 BLDG 4506 (DCD) RM 107 

 FT RUCKER AL 36362-5000 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MK MR J REINHART 

 10125 KINGMAN RD 

 FT BELVOIR VA 22060-5828 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MV HQ USAOTC 

 S MIDDLEBROOKS 

 91012 STATION AVE ROOM 348 

 FT HOOD TX 76544-5073 

 5 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MY B HUNN 

 2520 HEALY AVE STE 1172 BLDG 51005 

 FT HUACHUCA AZ 85613-7069 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MP D UNGVARSKY 

 BATTLE CMD BATTLE LAB 

 415 SHERMAN AVE UNIT 3 

 FT LEAVENWORTH KS 66027-2326 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MJF J HANSBERGER 

 JFCOM JOINT EXPERIMENTATION J9 

 JOINT FUTURES LAB 

 115 LAKEVIEW PARKWAY SUITE B 

 SUFFOLK VA 23435 
28 

NO. OF 

COPIES ORGANIZATION

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MQ M R FLETCHER 

 US ARMY SBCCOM NATICK SOLDIER CTR 

 AMSRD NSC WS E BLDG 3 RM 343 

 NATICK MA 01760-5020 

 1 ARMY RSCH LABORATORY-HRED 

 ATTN AMSRD ARL HR MT J CHEN 

 12423 RESEARCH PARKWAY 

 ORLANDO FL 32826 

 1 ARMY RSCH LABORATORY-HRED 

 ATTN AMSRD ARL HR MT C KORTENHAUS 

 12350 RESEARCH PARKWAY 

 ORLANDO FL 32826 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MS C MANASCO 

 SIGNAL TOWERS BLDG 29808A RM 303 

 FORT GORDON GA 30905-5233 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MU M SINGAPORE 

 6501 E 11 MILE RD MAIL STOP 284 

 BLDG 200A 2ND FL RM 2104 

 WARREN MI 48397-5000 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MF C HERNANDEZ 

 2421 NW AUSTIN RD STE 220 

 FORT SILL OK 73503-9042 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MW E REDDEN 

 BLDG 4 ROOM 332 

 FT BENNING GA 31905-5400 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MN R SPENCER 

 DCSFDI HF 

 HQ USASOC BLDG E2929 

 FORT BRAGG NC 28310-5000 

 1 ARMY G1 

 ATTN DAPE MR B KNAPP 

 300 ARMY PENTAGON ROOM 2C489 

 WASHINGTON DC 20310-0300 

 5 MARINE CORPS SYSTEMS CMD 

 ATTN A SCHWEIZER PM-ICE 

 2201A WILLIS STREET 

 QUANTICO VA 22134-6050 

NO. OF 

COPIES ORGANIZATION

 ABERDEEN PROVING GROUND

 1 DIRECTOR 

 US ARMY RSCH LABORATORY 

 ATTN AMSRD ARL CI OK (TECH LIB) 

 BLDG 4600 

 1 DIRECTOR 

 US ARMY RSCH LABORATORY 

 ATTN AMSRD ARL CI OK TP S FOPPIANO 

 BLDG 459 

 1 DIRECTOR 

 US ARMY RSCH LABORATORY 

 ATTN AMSRD ARL HR MR 

 F PARAGALLO 

 BLDG 459