# Decision Aiding to Overcome Biases in Object Identification

Authors: Mary Fendley; S. Narayanan
Journal: Advances in Human-Computer Interaction
 (2012)

---

## Abstract

Human decision makers typically use heuristics under time-pressured situations. These heuristics can potentially degrade task performance through the impact of their associated biases. Using object identification in image analysis as the context, this paper identifies cognitive biases that play a role in decision making. We propose a decision support system to help overcome these biases in this context. Results show that the decision support system improved human decision making in object identification, including metrics such as time taken to identify targets in an image set, accuracy of target identification, accuracy of target classification, and quantity of false positive identification.


---

## Body

## 1. Introduction

As the growth of sensor technology outpaces the analyst’s ability to process captured images, object identification within the military image analysis task has become an increasingly time-critical human problem-solving task [1]. Intuitively, in this information-rich domain, the pressure associated with time-critical decision making can lead human operators to deploy a variety of techniques to alleviate the time pressure. When this time pressure persists, the decision maker often changes their cognitive processing methods, leading to the use of cognitive heuristics and their resulting biases.Cognitive heuristics are rules-of-thumb employed during decision making that can lead to biases that degrade the quality of decisions. Huey and Wickens [2] identify how heuristics and biases impact decision making through the distortion of hypothesis formulation and situation awareness. They also conclude that this distortion, which can degrade decision making, can occur during information processing.Pioneering work by Tversky and Kahneman [3] and others in the judgmental decision making field [4–8] identifies several heuristics and biases that commonly appear during decision-making tasks. Although much research has been done on the effects of biases in judgmental decision making tasks, there has been little work done that specifically identifies cognitive biases within a time-critical task such as object identification. Thus there is a need to understand potential biases and develop support systems to mitigate their negative impacts, thereby aiding the analyst [9]. While decision support tools such as algorithms are currently being developed, they are presently not employed extensively by image analysts (IAs) in field settings [10]. Clearly, the dearth of tools indicates further work is needed to develop effective decision support methods to relieve the cognitive demands of the IAs task. This paper presents a study to identify the impact of cognitive biases that occur during object identification and describes how a decision support framework was designed, implemented, and empirically evaluated, which aids the human analyst in information processing.


## 2. Biases in Object Identification

A recent work by Arnott [11] contributes an exhaustive taxonomy of cognitive biases identified by decision theory researchers. This taxonomy of biases is divided into six broad categories. They are described as follows.(i)
Memory. Biases involving the storage and recall of information.(ii)
Statistical.Biases referring to the decision maker going against normative principles of probability theory during information processing.(iii)
Confidence. Biases serving to increase the decision maker’s confidence in their ability to make good decisions.(iv)
Presentation. Biases skewing the way decision makers perceive and process information.(v)
Situation. Biases concerning the manner in which people respond to the overall decision making environment.(vi)
Adjustment. Biases affecting the way decision makers make adjustments from a given position.To have the knowledge necessary to understand which biases potentially occur during the decision-making task, where in the decision making process they occur, and the types of errors produced, a combination of a pilot study, similar to the study described in this paper, and interviews with subject matter experts was conducted. This information was integrated to get a better picture of the participants’ cognitive processes. These results were used to determine which cognitive biases potentially affect the analyst during the decision-making task. These biases, shown in Table1, fit into four of the categories listed in the work of Arnott [11].Table 1
Potential biases in object identification.
Bias categoryCognitive biasMemory biasesImaginabilityAn event that is easily imagined is judged to be more probableRecallAn event may seem more probable if an instance is easily recalledSearchAn effective search strategy may make an event seem more frequentStatistical biasesCorrelationProbability of the co-occurrenceof events may be overestimated due to previous cooccurrenceConfidence biasesConfirmationConfirming, rather than disconfirming, evidence is soughtRedundancyRedundant data may cause undue confidence in its accuracy and importanceSelectivityExpectation of the nature of an event influences what information is thought to be relevantPresentation biasesOrderUndue importance may be placed on the first or last data pointThe participants were asked to identify their rationale for choosing an object as a target. The choices given to the participants were (1) I saw a similar target in the same area in previous images, (2) it made sense that the target was in this location because of its type, (3) there are similar targets in the image about which I was confident (easily detectible), (4) this target was located near another target in a previous image, and (5) I am unsure of the type of target, but do not remember seeing any other type in this area in previous images. These responses, along with data gathered through concurrent protocol and a tracer to record mouse clicks and keystrokes as they interacted with the computerized system, were integrated to determine which biases were potentially occurring and at what time.Decision points were chosen to determine the presence of a particular bias. An example of one of these decision points is the influence of the order bias. In one of the sequences, errors were made in identifying a target in the tenth image. Figure1 shows the first (shown on left) and the tenth image (shown on right). The plane can easily be seen in the lower right of the image (on the tarmac) in the image on the left. In the image on the right, only a white mark is there. Seventeen percent of the participants incorrectly identified this as a plane. The identification of the bias was supported by the rationale the participants chose, stating that they saw the “targets” in the first two images.Figure 1
Sample images showing bias.Object identification, is at its core, an information-processing task. The extension of this relationship between object identification and information processing suggests that the biases present in one task related to information processing have the potential to exist in any task where information processing is central to its execution. This assertion is bolstered by the command and control research of Duvall [12], which mapped the Observe, Orient, Decide, and Act model to Huey and Wickens’ information-processing model and denoted the presence of several biases. Some of these biases were also present in this object identification task.The research literature suggests that the very nature of the image analyst’s object identification task makes it highly likely that biases will be present. Biederman’s theory on human recognition of objects in two-dimensional images suggests that humans completing such a task are easily susceptible to cognitive biases, and he proposes their presence, as the final identification of the object is done by matching the human’s perception of the object with what is held in their memory [13].


## 3. Decision Support System Development

Decision support systems are designed to improve decision making by enhancing cognitive decision-making capabilities and should be integrated with the decision process of the operator. The comments regarding the negative perceived impact of time constraints verify the existing body of literature [14–16] which suggests that by having to search faster, the human changes their cognitive strategies to the heuristics that lead to biases. To design an effective decision support system, it is imperative to accurately match the expressed bias with the well-established debiasing strategy, cognitive forcing, visualization, feedback, memory reliance, and task complexity, which has the best chance to mitigate the expressed bias. This will need to be considered in the development of the decision support to address each of the cognitive biases.The decision support system (DSS) includes three separate artifacts that, together, were intended to enhance overall performance through both debiasing and enabling the productive use of heuristics. These three artifacts include a repository of sample snapshots of the targets taken from different angles and under different conditions, a message board relaying potentially useful information regarding the area where the sets of images were taken, and a marking aid used to draw attention to specific areas/entities in the images. The image repository was designed to mitigate specific biases through visual cues and aiding the decision making without relying solely on their memory. The message board intended to help mitigate specific biases through providing feedback and cognitive forcing, aiding in selecting a strategy to optimize decision making [17], and aiding with task complexity. The marking aid serves to mitigate the specific biases shown by attempting to reduce task complexity, cognitive forcing, and aiding with memory reliance. The following sections describe each of these in greater detail.


### 3.1. Image Repository

An image repository serves as the first component of the DSS. The image repository serves as a debiasing method by decreasing the decision maker’s reliance on memory, decreasing the level of task complexity, and increasing the ability to visualize. These sample snapshots of the targets were taken from images similar to those presented to the participants in these trials. The repository was designed to show the analyst instances of each target without having to scroll through a menu. Therefore, each of the targets was presented under five different conditions (one image for each condition for each target) within one viewing window. These images showed the target from the front or top view, side view, black and white, partially occluded, and mostly occluded or in busy surroundings. A menu of the available targets is located above the view windows for the analyst to select the target they wish to see. Figure2 shows an example of what the participants would see when pulling up the snapshots of a target.Figure 2
Display of example smages of SCUD.The images in the repository were chosen to make the decision maker aware of the different possible appearances of the targets and help them make faster, more accurate decisions by having these constantly available while performing the task. This represents a small sample of what could be expanded to become a very useful tool for the image analyst under real working conditions.


### 3.2. Message Board

A message board is the second component of the DSS. The message board functions as a debiasing method by serving as a cognitive forcing strategy and by providing reliable feedback to the decision maker. The message board provided information regarding the area where the trial images were taken. Five distinct pieces of information were provided for each of the image sets. This information was presented as the likelihood of a given type of target being present and the percentage of targets previously found in specific areas of the terrain (e.g., on roads, near hangar, around wooded areas). Figure3 shows a screenshot of how the message board appeared to the participants.Figure 3
Message board.The messages were available to the participants throughout the trial set and were used to help them look beyond the obvious areas of interest in the image and to give them an idea of whether their use of heuristics was going to lead them in the right direction. As image analysts look at multitudes of images taken in the same area, this concept could be expanded to develop an automatic calculation of these figures for real time use.


### 3.3. Marking Aid

The marking aid is the third component of the DSS. The marking on the images from the algorithm serve as debiasing support as a cognitive forcing strategy that also decreases reliance on memory. Markings were done in MATLAB according to two location-based criteria discussed below.(i)
Results from the condition without decision support showed that targets were frequently and consistently missed around the perimeter of an image, and when there was a primary and secondary area in the image. An example of this is where there is a main road with several targets and a smaller road, which may or may not have targets (Figure4(a)).(ii)
Targets were also frequently missed when they were smaller (due to location or type) than other targets in the image, when they blended with an adjacent, man-made object, or there was more than one target in a similar area (Figure4(b)).Marking aid processes.
(a)(b)The images sets using the algorithm were presented in the same format as those without the aid’s assistance. Figure5 shows a sample image with the markings.Figure 5
Image with markings.While performing the task with decision support, the participant could have an image with markings like the one seen above, and at the same time was shown the message board and had access to the image repository.


## 3.1. Image Repository

An image repository serves as the first component of the DSS. The image repository serves as a debiasing method by decreasing the decision maker’s reliance on memory, decreasing the level of task complexity, and increasing the ability to visualize. These sample snapshots of the targets were taken from images similar to those presented to the participants in these trials. The repository was designed to show the analyst instances of each target without having to scroll through a menu. Therefore, each of the targets was presented under five different conditions (one image for each condition for each target) within one viewing window. These images showed the target from the front or top view, side view, black and white, partially occluded, and mostly occluded or in busy surroundings. A menu of the available targets is located above the view windows for the analyst to select the target they wish to see. Figure2 shows an example of what the participants would see when pulling up the snapshots of a target.Figure 2
Display of example smages of SCUD.The images in the repository were chosen to make the decision maker aware of the different possible appearances of the targets and help them make faster, more accurate decisions by having these constantly available while performing the task. This represents a small sample of what could be expanded to become a very useful tool for the image analyst under real working conditions.


## 3.2. Message Board

A message board is the second component of the DSS. The message board functions as a debiasing method by serving as a cognitive forcing strategy and by providing reliable feedback to the decision maker. The message board provided information regarding the area where the trial images were taken. Five distinct pieces of information were provided for each of the image sets. This information was presented as the likelihood of a given type of target being present and the percentage of targets previously found in specific areas of the terrain (e.g., on roads, near hangar, around wooded areas). Figure3 shows a screenshot of how the message board appeared to the participants.Figure 3
Message board.The messages were available to the participants throughout the trial set and were used to help them look beyond the obvious areas of interest in the image and to give them an idea of whether their use of heuristics was going to lead them in the right direction. As image analysts look at multitudes of images taken in the same area, this concept could be expanded to develop an automatic calculation of these figures for real time use.


## 3.3. Marking Aid

The marking aid is the third component of the DSS. The marking on the images from the algorithm serve as debiasing support as a cognitive forcing strategy that also decreases reliance on memory. Markings were done in MATLAB according to two location-based criteria discussed below.(i)
Results from the condition without decision support showed that targets were frequently and consistently missed around the perimeter of an image, and when there was a primary and secondary area in the image. An example of this is where there is a main road with several targets and a smaller road, which may or may not have targets (Figure4(a)).(ii)
Targets were also frequently missed when they were smaller (due to location or type) than other targets in the image, when they blended with an adjacent, man-made object, or there was more than one target in a similar area (Figure4(b)).Marking aid processes.
(a)(b)The images sets using the algorithm were presented in the same format as those without the aid’s assistance. Figure5 shows a sample image with the markings.Figure 5
Image with markings.While performing the task with decision support, the participant could have an image with markings like the one seen above, and at the same time was shown the message board and had access to the image repository.


## 4. Empirical Evaluation

### 4.1. Hypotheses

The first hypothesis is that it will take less time to analyze the image sets with the decision support tool. The second hypothesis is that target identifications will be more accurate with the decision support tool. The third hypothesis is that target classifications will be more accurate with the decision support tool. The fourth hypothesis is that there will be fewer false positives while identifying targets with the decision support tool. The next section describes the study conducted to determine the role of the decision support system in terms of how biases impacted the identification and classification task.


### 4.2. Design of Experiment

A between-subjects experiment was designed and conducted to test the impact of the design support tools on the cognitive biases impacting object identification.


#### 4.2.1. Participants

Forty-six participants were recruited from the university community and consisted mainly of graduate students with some classroom or field experience working with images. All participants were asked if they were color blind, as not being color blind is a requirement for military image analysts.


#### 4.2.2. Apparatus and Stimuli

The experiment was conducted on an office environment on a 17-in and display monitor, and the input device was a mouse. The participants viewed images through an interface that allowed them to move from image to image and zoom in on the image. The relevant information displayed on the left-hand side includes the coordinates of the cursor over the image and a top drop-down menu showing the file name of the image being viewed. The remaining buttons are used for listing, viewing, and saving the images. The right-hand side lists the file names of all the images in the system. The number listed to the left of the filename reflects the number of detected targets. The file names will appear in descending priority order once the images have been viewed. The images used for the experiment were series of frames taken from infrared movies, similar to what an analyst might view from sensor data. A sample of the interface with an image displayed can be seen in Figure1.


#### 4.2.3. Procedure

Five sequences of ten images were shown to the participants in random order. The images were modified to extract the biases already shown to potentially be present in the decision-making task. Once the participant selects an image to view, a new window appears showing the image with a box that allows them to zoom in on a particular area of the image. Figure3 shows a screenshot of a sample image being viewed.The participant uses the mouse to draw a box around areas they determine to be a target. They are then asked to classify the type of target and choose their confidence in selecting this object as a target. The participants were then tasked with determining target location and classification by type. They were also instructed to rate their confidence level in their classification.The result of this exercise is a set of images marked with the location of the targets. During the experiment they were asked to explain their decision-making processes out loud. This was followed up by a questionnaire designed to extract additional information on the participant’s cognitive processes during the completion of the task. Data were collected through the use of concurrent protocol and a tracer within the system, collecting information regarding where in the image the participant spent time examining, how many times they returned to view a specific area, and so forth.


## 4.1. Hypotheses

The first hypothesis is that it will take less time to analyze the image sets with the decision support tool. The second hypothesis is that target identifications will be more accurate with the decision support tool. The third hypothesis is that target classifications will be more accurate with the decision support tool. The fourth hypothesis is that there will be fewer false positives while identifying targets with the decision support tool. The next section describes the study conducted to determine the role of the decision support system in terms of how biases impacted the identification and classification task.


## 4.2. Design of Experiment

A between-subjects experiment was designed and conducted to test the impact of the design support tools on the cognitive biases impacting object identification.


### 4.2.1. Participants

Forty-six participants were recruited from the university community and consisted mainly of graduate students with some classroom or field experience working with images. All participants were asked if they were color blind, as not being color blind is a requirement for military image analysts.


### 4.2.2. Apparatus and Stimuli

The experiment was conducted on an office environment on a 17-in and display monitor, and the input device was a mouse. The participants viewed images through an interface that allowed them to move from image to image and zoom in on the image. The relevant information displayed on the left-hand side includes the coordinates of the cursor over the image and a top drop-down menu showing the file name of the image being viewed. The remaining buttons are used for listing, viewing, and saving the images. The right-hand side lists the file names of all the images in the system. The number listed to the left of the filename reflects the number of detected targets. The file names will appear in descending priority order once the images have been viewed. The images used for the experiment were series of frames taken from infrared movies, similar to what an analyst might view from sensor data. A sample of the interface with an image displayed can be seen in Figure1.


### 4.2.3. Procedure

Five sequences of ten images were shown to the participants in random order. The images were modified to extract the biases already shown to potentially be present in the decision-making task. Once the participant selects an image to view, a new window appears showing the image with a box that allows them to zoom in on a particular area of the image. Figure3 shows a screenshot of a sample image being viewed.The participant uses the mouse to draw a box around areas they determine to be a target. They are then asked to classify the type of target and choose their confidence in selecting this object as a target. The participants were then tasked with determining target location and classification by type. They were also instructed to rate their confidence level in their classification.The result of this exercise is a set of images marked with the location of the targets. During the experiment they were asked to explain their decision-making processes out loud. This was followed up by a questionnaire designed to extract additional information on the participant’s cognitive processes during the completion of the task. Data were collected through the use of concurrent protocol and a tracer within the system, collecting information regarding where in the image the participant spent time examining, how many times they returned to view a specific area, and so forth.


## 4.2.1. Participants

Forty-six participants were recruited from the university community and consisted mainly of graduate students with some classroom or field experience working with images. All participants were asked if they were color blind, as not being color blind is a requirement for military image analysts.


## 4.2.2. Apparatus and Stimuli

The experiment was conducted on an office environment on a 17-in and display monitor, and the input device was a mouse. The participants viewed images through an interface that allowed them to move from image to image and zoom in on the image. The relevant information displayed on the left-hand side includes the coordinates of the cursor over the image and a top drop-down menu showing the file name of the image being viewed. The remaining buttons are used for listing, viewing, and saving the images. The right-hand side lists the file names of all the images in the system. The number listed to the left of the filename reflects the number of detected targets. The file names will appear in descending priority order once the images have been viewed. The images used for the experiment were series of frames taken from infrared movies, similar to what an analyst might view from sensor data. A sample of the interface with an image displayed can be seen in Figure1.


## 4.2.3. Procedure

Five sequences of ten images were shown to the participants in random order. The images were modified to extract the biases already shown to potentially be present in the decision-making task. Once the participant selects an image to view, a new window appears showing the image with a box that allows them to zoom in on a particular area of the image. Figure3 shows a screenshot of a sample image being viewed.The participant uses the mouse to draw a box around areas they determine to be a target. They are then asked to classify the type of target and choose their confidence in selecting this object as a target. The participants were then tasked with determining target location and classification by type. They were also instructed to rate their confidence level in their classification.The result of this exercise is a set of images marked with the location of the targets. During the experiment they were asked to explain their decision-making processes out loud. This was followed up by a questionnaire designed to extract additional information on the participant’s cognitive processes during the completion of the task. Data were collected through the use of concurrent protocol and a tracer within the system, collecting information regarding where in the image the participant spent time examining, how many times they returned to view a specific area, and so forth.


## 5. Results

Information gathered from the concurrent protocol and the tracer was used to empirically evaluate the decision support. For the quantitative variable, time, at-test was used to compare the time taken to identify the targets. For the variables accuracy of target identifications, accuracy of target classifications, and number of false positives identified, a Pearson’s chi-squared test was used to compare the analysis process with and without the decision support. P values less than 0.05 were considered significant. The decision support system was evaluated on four quantitative aspects measured by the tracer (time taken to identify targets in an image set, accuracy of target identification, accuracy of target classification, and quantity of false positive identifications). The quantitative results are shown in Figures 6, 7, 8, and 9.Figure 6
Time for identifications.Figure 7
Targets identified.Figure 8
Accuracy of classifications.Figure 9
Number of false positives.


### 5.1. Time to Identify Targets

The times taken to identify targets without decision support and with decision support were analyzed. A significant difference (t(45)=-2.983, P<0.0046) was shown for time to identify targets between the two trials, indicating the decision aid was able to reduce the time taken to identify targets overall. We are 95% confident that the average difference in time to identify targets lies between −3.69 and −0.72.


### 5.2. Accuracy of Target Identification

The accuracy of identifying targets without decision support and with decision support was analyzed. The percentage of correctly identified targets differed with the use of decision support,χ2(1,N=14283)=673.08,  P<0.0001, indicating the decision aid was able to increase the accuracy of target identification.


### 5.3. Accuracy of Target Classification

The accuracy of correctly classifying targets without decision support and with decision support was analyzed. The percentage of correctly classified targets differed with use of decision support,χ2(1,N=14283)=554.06,  P<0.0001, indicating the decision aid was able to increase the accuracy of target classification.


### 5.4. Identifications of False Positives

The number of false positives identified without decision support and with decision support was analyzed. The percentage of identified false positives differed with use of decision support,χ2(1,N=535)=268.49, P<0.0001, indicating the decision aid was able to reduce the number of false positives identified.


## 5.1. Time to Identify Targets

The times taken to identify targets without decision support and with decision support were analyzed. A significant difference (t(45)=-2.983, P<0.0046) was shown for time to identify targets between the two trials, indicating the decision aid was able to reduce the time taken to identify targets overall. We are 95% confident that the average difference in time to identify targets lies between −3.69 and −0.72.


## 5.2. Accuracy of Target Identification

The accuracy of identifying targets without decision support and with decision support was analyzed. The percentage of correctly identified targets differed with the use of decision support,χ2(1,N=14283)=673.08,  P<0.0001, indicating the decision aid was able to increase the accuracy of target identification.


## 5.3. Accuracy of Target Classification

The accuracy of correctly classifying targets without decision support and with decision support was analyzed. The percentage of correctly classified targets differed with use of decision support,χ2(1,N=14283)=554.06,  P<0.0001, indicating the decision aid was able to increase the accuracy of target classification.


## 5.4. Identifications of False Positives

The number of false positives identified without decision support and with decision support was analyzed. The percentage of identified false positives differed with use of decision support,χ2(1,N=535)=268.49, P<0.0001, indicating the decision aid was able to reduce the number of false positives identified.


## 6. Summary and Conclusions

Employment of the decision support system produced a statistically significant improvement in the participant’s ability to both accurately identify targets and accurately classify targets by type. The statistically significant improvement was present in each similar image set pair, as well as the aggregated experimental comparison. Employment of the decision support produced a statistically significant reduction in the number of false positives identified by the subjects.Clearly these results indicated that the combination of artifacts selected as part of the decision support was good choice. Interestingly, the concurrent protocol found that the participants felt that the image repository was the most helpful decision support artifact. This was followed closely by the message board artifact. The least helpful artifact was the marking aid, which ironically would be the most likely artifact to be automated.Further analysis was done to determine the types of biases the decision support was successful in mitigating. Examining Arnott’s [11] broad bias categorizations (memory, statistical, confidence, presentation), the decision support showed improvement across each one. Looking at the individual biases, each shows improvement, most notably in the redundancy bias, which was completely eliminated. Imaginability, correlation, and order were also nearly eliminated. This was likely due to the fundamental nature of the object identification task.These results indicate that the artifacts used in this decision support system work together to mitigate several of the biases very nicely, but that there is still room for significant improvement. It would seem most likely that the decision support could be further refined to lower the presence of biases that were not mitigated well by the current version, but at some point tradeoffs will have to be made as not all biases will be mitigated 100% of the time, and attempts to mitigate some biases may have the opposite effect on others by causing an increase in their influence.The work done with this research can be applicable across many different domains. As previously discussed, there is a lack of research in the area of aiding image analysts. This work provides a foundation for developing systems based on sound cognitive engineering principles to aid the image analyst. The next step for improving the performance of the decision support system is to automate the information provided by the decision support so that real-time feedback can be reported and to examine the interaction of the automated system and the human image analyst to ensure accurate cognitive coupling for improved performance.



---

*Source: 790304-2012-05-07.xml*