# The Personal Equation of Complex Individual Cognition during Visual Interface Interaction

Tera Marie Green, Brian Fisher

► To cite this version:

Tera Marie Green, Brian Fisher. The Personal Equation of Complex Individual Cognition during Visual Interface Interaction. 2nd Human-Computer Interaction and Visualization (HCIV) (INTER-ACT), Aug 2009, Uppsala, Sweden. pp.38-57, 10.1007/978-3-642-19641-6_4. hal-01572651

HAL Id: hal-01572651

https://inria.hal.science/hal-01572651

Submitted on 8 Aug 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# The Personal Equation of Complex Individual Cognition during Visual Interface Interaction

Tera Marie Green and Brian Fisher

School of Interactive Arts + Technology
Simon Fraser University
terag@sfu.ca

**Abstract.** This chapter considers the need for a better understanding of complex human cognition in the design of interactive visual interfaces by surveying the availability of pertinent cognitive models and applicable research in the behavioral sciences, and finds that there are no operational models or useful precedent to effectively guide the design of visually enabled interfaces. Further, this chapter explores the impact of individual differences, and in particular, inherent differences such as personality factors, on complex cognition. Lastly, it outlines how knowledge of human individuality, coupled with what is known about complex cognition, is being used to develop predictive measures for interface interaction design and evaluation, a research program known as the Personal Equation of Interaction. ...

## 1 Introduction

Generally speaking, interactive visualizations are considered to have a number of advantages over more conventional visual interfaces for learning, analysis, and knowledge creation. Much of the support for these claims comes from a variety of sources, such as user evaluations, comparative studies of error rates, time to completion, etc., as well as and designer/developer intuition. One common claim concerns the development of insight. From early on, visualization has been proposed as a preferable interface approach for generating insight (e.g. [3], [39], [43]). As a concept, however, insight is a construct that is often either loosely defined as some variety of meaningful knowledge or is left undefined (e.g. [40], [44]). More recently there have been efforts to define insight, although not in ways that might enable it to be quantified. For example, North described insight as a broad construct, which is complex, deep, qualitative, unexpected, and/or relevant [27], without characterizing the cognitive processes that give rise to it, or the outcomes that are generated by it. Chang et al. defined insight as comprising two categories: knowledge-building insight, which is a form of learning and/or knowledge generation, and spontaneous insight, which is method of problem-solving for previously intractable problems, commonly described as an a-ha! moment. This dual definition has advantages over a unitary definition in that it supports focused analysis of the component aspects of the overall construct. Spontaneous insight, however, has been an elusive notion for researchers in several disciplines;
neuroscientists and psychologists have studied the phenomenon, but as yet do
not know how insight is triggered [4].

By any definition, there is little empirical evidence that supports claims of visualization superiority in insight generation, though user evaluations are often conducted to demonstrate visualization efficacy over other types of interface. Plaisant et al. [30] identified four current themes in the evaluative literature: controlled experiments comparing design elements, usability evaluation, controlled experiments comparing two or more tools, and in situ case studies. In all four groups, evaluations and comparative studies have largely focused on perception, motor learning, focal attention, target recognition and/or target acquisition. For example, musing behaviors were used as a predictor of user focus in a geospatial visualization [1]. Jeong et al. compared two visualization tools to determine in which interface users were more efficient in finding outliers and identifying highly correlated items in a matrix [22]. Nodes were the subject of an evaluation of target identification in large tree tools [29]. And Wang et al. evaluated whether users could focus on the count of visualized objects (in this case, paper proposals) over a period of time [52]. In these evaluations as well as in cognition as a whole, perceptual, cognitive, and motor processes are important to the overall interaction. However, each of these identified cognitive systems are feeder processes. That is to say, they support and inform the more complex processes, such as reasoning, problem-solving, and knowledge generation, which form the backbone of systematic analysis or task solution. These complex processes are impacted demonstrably, as we will see, by the individuality of the interface environment, the knowledge domain, and the inherent differences within the user, over which visualization design has no control. To date, visualization evaluation has insufficiently considered the complexity of human cognition. This, in turn, has hampered the design of intuitive interfaces capable of mixed-initiative collaboration.

In this chapter, we will explore a variety of challenges to the consideration of
cognitive complexity in visual analytics design, from the current lack of opera-
tional models and applicable research to a consideration of individual differences.
We will then explore how an consideration of how these complex processes impact
the understanding of common visual analytics tasks, and discuss a continuing
exploration of how human individuality can be measured and charted, leading
to a differentiating set of predictive measures that can not only predict interface
performance, but guide visualization design. We call this the Personal Equation
of Interaction.

## 2 The Challenge of Complex Cognition

Very little research examines the use of what is commonly known as higher cog-
nition during interaction, which includes processes such as reasoning, problem-
solving, and decision-making. Frequently, when a visualization design or evalu-
ation argues that a specific technique or tool improves insight (which is learn-
ing and/or problem-solving) or analysis (which involves every major cognitive
process), the evidence is actually task completion times for single-step tasks, improved target identification, or other simple outcomes. One reason for this, perhaps, is the practice of inferring the success of complex behaviors from measurements of simpler ones. A common example is the generalization of findings from simple, semantically-unrelated target acquisition tasks to human problem-solving as a whole, without a discussion of which of the many problem-solving theories or heuristics the finding might speak to (e.g.[8], [35]). This practice over-simplifies the complexity of cognition, but is understandable, given that our best complex cognitive models are black box or descriptive. We will now consider the best known of these descriptive models, the sensemaking model.

## 2.1 The Sensemaking Loop

The most familiar approach to descriptively outline task-oriented processes is Pirolli and Cards sensemaking loop ([31], [38]). See Figure 1. Russell et al. defined sensemaking as *the process of searching for a representation and encoding data in that representation to answer task-specific questions* [38]. In the rest of this section, we will summarily explore the complexity of analytical cognition through a brief discussion of the sensemaking loop in the broader context of human reasoning. This seems necessary, for, as valuable as the sensemaking loop is to describing certain analytical tasks, its use tends to be overgeneralized in the visualization literature. Indeed, *sensemaking* is often the term given to most or all of the cognitive processes analysts employ during visual analytics tasks (e.g. [47], [31], [20]).

Sensemaking, as defined in the previous paragraph, creates a mental or physical representation (i.e. a “mental model” or “story”). This singular representation may well be necessary for problem solving, but may not be in itself sufficient for generating valid implications. Analyses may create multiple alternative mental representations of a situation in order to compare them in a variety of ways, using a variety of evaluative heuristics in order to draw their conclusions. At this larger scale of problem solving strategy, analytical cognition exhibits a great deal of variability, and is informed by both human and task individuality. For example, the sensemaking loop makes an effort to delineate top-down and bottom-up task descriptions. However, as some of Pirolli and Cards participants indicated, the cognition involved in the early tasks of the loop can rarely be so cleanly categorized. Further, though seemingly simplistic, even the first steps of the sensemaking loop (the “lower-effort” tasks of searching and filter) require complex cognition in the form of various reasoning heuristics to categorize, evaluate, and assemble pertinent information. These heuristics could be elimination heuristics like elimination by aspects [51], or satisfying [24] or they could be more complicated, such as the comparing possible shoebox members (concepts the analyst has gathered during sensemaking and think may be related to each other) to an ideal before addition. According to the Loop, pertinent shoebox members become part of and Evidence File which is used as part of the formal structure of the Schema, which is a structured narrative of how the evidence collected thus far fits together [31]. Thus, while sensemaking effectively describes
<figure><img src="image_2.png" /><figcaption>Fig. 1. The Sensemaking Loop. From [38].</figcaption></figure>

one key component of the complex cognitive processes involved in the generation of insight, it does not claim to capture higher levels of abstraction, i.e. generation of problem-solving strategy, nor does it attempt to capture lower-level pattern recognition processes that support its operation. The latter set of processes are arguably the most critical for visualization to support, since they are precisely those aspects that are most closely tied to the visual systems own information processing capabilities (i.e. Irving Rocks logic of perception [36]).

Similarly, Cherubinis “models to rules mechanization”[6] suggests that the formation of schemata and hypothesis generation are not necessarily higher effort tasks. According to Cherubini, after human reasoning uses the generated model (which does require some cognitive effort to create in novel instantiations), the human infers a rule from the knowledge structure in the mental model. Or in his own words: *After grasping the common structure of the problems, most people should be able to devise a simple rule to solve all problems with the same structure (i.e., a domain-specific rule), disregarding the number of possibilities underlying them.*[6] This rule may be created after only one or two uses of a
newly created model. Thus, depending on the information under consideration, hypothesis generation could actually require less cognitive bandwidth than the initial information search.

At a higher level, the sensemaking loop articulately describes one subset of reasoning, that of reasoning types which generate inferred hypotheses or generalized rules, or abduction and induction. Abduction (or abductive reasoning) is process of approaching seemingly unrelated information with the assumption that the data points or concepts are indeed interconnected; abduction creates and infers relations between two previously unrelated data points; the end product of abductions series of inferences is the creation of a hypothesis which explains these relational inferences [41], and usually about a specific item or concept. This fits well with the structure of the sensemaking loop, which situates the search and compartmentalization of small, unassociated details in the early stages of the loop and builds from there to an identifiable hypothesis or story.

The sensemaking loop also illustrates induction. Inductive reasoning, as is described in the behavioural literature, is generally referred to the process of making acceptable generalizations from the similarities of facts or properties (see for example [34], [50]). The validity or strength of the generalization depends in large degree upon the strength of these similarities. One of the characteristics of induction which makes induction different from abduction is that relationships are not as important to the successful generalization. Once a fact(s) or similarity has been accepted as valid, it or they become the basis of generalization; additional information may or may not be considered. Induction is a powerful form of reasoning that allows us to quickly categorize and infer rules; it is often referred to as a form of bottom-up reasoning, but can utilize top-down cognition as is needed. To some degree, the sensemaking loop describes induction, as this type of reasoning tends to build from smaller facts to a broader concept. However, unlike with hypothesis generation and analysis, induced generalizations do not necessarily require vetting, e.g. careful consideration of available evidence for the generalization. Induced hypotheses can jump past multiple steps, such as the shoebox or schema creation, straight to a generality.

While abduction or induction may accurately describe behavior during discovery or during exploration of novel problems for which the analyst does not already have a script or mental model to guide her as to what to do next, most problem-solving or decision-making tasks are driven by an articulated goal, theory or hypothesis from the start. These reasoning heuristics are neither abductive, which ends with a hypothesis, nor inductive, ending in an inferred rule, but rather deductive.

One of the more actively studied reasoning types, deductive reasoning falls outside of the sensemaking loop. By deduction (or, for that matter, induction), we are referring to reasoning as the subject of decades of empirically conducted research, which is usually evaluated through use of normative tasks like those that we will briefly itemize in the next section. A discussion of the philosophy of deduction, such as the traditional deduction of Aristotle, propositional (or first order) deduction, or natural deduction ([21], [10]), and the similarities or
differences between these definitions, is neither intended nor implied.

There are several theories of deduction in the behavioral literature, but we will focus on two of the broader categories of deduction, which assume that human deductive reasoning is either rule based or model based. This dichotomy of rules vs. models raises interesting issues for visual analytics interfaces. Rule-based theories assume that humans use a script or a formal sequential logic while working through a deductive task (e.g. [53]). From this perspective, the content or information manipulated during the task doesn't demonstrably influence the reasoning, because the inferences drawn during induction are part of this formal process[54]. From the rules perspective, for the development of more intuitive deductive visual analytics tools, it would only be important to uncover the pertinent rules the analyst would use; theoretically, these deductive rules would generalize to all similar visual analytics tasks.

Model-based theories, however, are quite different. Models can be either concrete or abstract, complete or incomplete, pictorial or conceptual (e.g. [53],[54]). Models are also flexible; they can change as pertinent information changes, and can quantify degree (such as few or often), as well as causal conditions. Models depend heavily on semantic relationships, and so can be heavily influenced by the content of the information at hand. This, too, influences visualization design, for what data is presented when, and in what context, can influence the development of the model, as well as its completeness and validity. From the model-based perspective, discovering quickly inferred rules is not nearly as helpful as assuring that the human has all pertinent information readily at hand. With pertinent information, the human can generate a mental model, which can be manipulated as needed to reasoning through the task to a valid conclusion. Shneidermans Mantra [42]: Overview first, zoom and filter, details-on-demand assumes deductive reasoning. The big picture, or hypothesis, drives the interactive behavior. It is not surprising, then, as powerful as models would seem to be in the successful use of visual analytics interfaces, that they have a place in the visual analytics literature, even if the theory and implications of models are rarely discussed.

This has been only a brief, general discussion of the sensemaking loop and how it fits into the broader context of common reasoning types. Human reasoning is about reaching a usable or verifiable conclusion, but the ways in which we reach these conclusions, as we have seen, can vary widely. For this reason, it is easy to see why analytical reasoning processes have yet to be operationalized in a manner that meaningfully informs research and design. For while descriptive models like the sensemaking loop do much to frame the big picture, intuitive interfaces will require a more detailed working-order understanding of what lies inside the frame.

## 2.2 The Absence of Precedent

As we saw in the last section, there is, as yet, no unifying theory of reasoning (if such a thing is even possible). What does exist is a complication of decades of research into specific laboratory tasks, usually characterized by small-scale
problems, which are intended to uncover reasoning heuristics and biases. These
are of limited use for real-world applications, and in particular map poorly onto
visually enabled human reasoning (e.g. interactive visualization for cognitive
tasks). Further, the theories that motivate these studies are often bound to
a particular task and environment. Thus the field of behavioural research as a
whole is characterized by contradictory, often esoteric theories that fail to explain
the narrative of reasoning from beginning of task to end.

For example, deductive reasoning is almost entirely studied in a laboratory
trials. Both rule based and model-based deduction has traditionally studied by
presenting participants with syllogisms and evaluating the conclusions that are
drawn. Phillip Johnson-Laird often uses syllogisms to study aspects of reasoning,
which can take forms such as this inference about object properties:

Only one of the following statements is true:

At least some of the plastic beads are not red, or

None of the plastic beads is red.

Is it possible that none of the red beads is plastic? ([23], pg. 150).

Other common uses of syllogisms involve mental reasoning and inferences about
spatial relationships, such as:

The cup is on the right of the plate.

The spoon is on the left of the plate.

The knife is in front of the spoon.

The saucer is in front of the cup.

What is the relation between the knife and the saucer? ([23], pg. 130)

Cherubini and Johnson-Laird [5] studied qualified inferences in iterative reason-
ing through word problems like the following:

Everybody loves anyone who loves someone.

Anne loves Beth.

Does it follow that everyone loves Anne?

...

Does it follow that Carol loves Diane?

Cherubini and Mazzocco also evaluated the mental models to rules mechaniza-
tion through use of a computer program loaded with a series of virtual card
problems [6] as illustrated in Figure 2. The participant was asked whether, based
on the presented cards, a proposed sentence was *certainly true*.

Gigerenzer, in his evaluation of “fast and frugal” reasoning heuristics, used
what he considered to be common knowledge about cities in questions about
which he asked participants to make quick reasoning decisions. The questions
were simple, such as *Is this [city name] the capital of the country?* [11]. Gigeren-
zer postulated that humans could make quick decisions based on very simple
elimination heuristics which depended on accumulated general knowledge. These
decisions were found to be more accurate than more sophisticated human and
computer reasoning simulations.
<figure><img src="image_3.png" /><figcaption>Fig. 2. The reasoning task from Cherubini [6].</figcaption></figure>

The behavioral literature contains decades of research similar to the research we have discussed, with each study having its own novel, usually non-real world, problem formulation. Study problems are often designed to study some small subcategory of reasoning (iterative inferred, probabilistic, etc.) and very few or no studies are published which are designed to explain how humans solve a complex problem from start to finish.

Perhaps it is not surprising then that, with all of this research, there is still a lack of precedent on how to conduct research into visually enabled reasoning. It is not at all clear how one might evaluate interfaces with respect to their ability to scaffold higher-order cognitive tasks. Further, unlike many of the simpler cognitive tasks, higher cognition is almost never binary, sequential, or singly threaded. It is, in practice, dynamic, combinatorial, and capable (at least to some degree) of parallel processing. Which heuristics are used during complex cognition and when will depend on the task, the environmental framing, and, as we will now discuss, differences in how an individual assimilates and manipulates new information.

## 3 Individual Differences

Complex cognition, for all of its variety, is also influenced by human individuality. There is no standardized unit of human cognition. It is influenced, sometimes profoundly, by the users distinctive abilities and bottlenecks, beliefs about the world, preferred methods of categorizing and prioritizing information, and other individual differences. This is one reason that the modeling of reasoning has traditionally been difficult. Human behavioral research has demonstrated the
impact of individual differences on learning and analysis in traditional environ-
ments.

There is also a plethora of examples in the behavioral literature of how in-
dividual differences impact cognition; for the sake of brevity, we will focus on
the impact of personality factors, which also have a broad literature of their
own. For example, personality factors predicted preferences and visual percep-
tion of landscapes [26]. Visual impairment in children is heavily influenced by
individual personality differences [7]. Individual differences also affect how hu-
mans categorize, including the categorizing of stereotyping and prejudice [19].
Palmer found that interactive behaviors in information search can be categorized
by personality factors [28]. Another study found that problem-solving behaviors
could be predicted by responses to the Thematic Apperception Test [37]. In rea-
soning behaviors, individual differences impact rationality and reasoning as well
([45],[46]). These are just a handful of studies in a deep literature of individu-
ality and the impact of these differences on every major cognitive process, as
well as behavioural outcomes, such as academic or organizational performance.
The question is not whether individual differences impact cognition, but how we
can use individual differences to improve our understanding of visually enabled
analysis. In addition, users in a particular domain can share personality charac-
teristics and learning preferences, both inherent and institutional, which implies
that some common traits can be aggregated into specific user profiles which can
inform superior design requirements and aid in evaluation protocols. These dif-
ferences will be discussed as part of the Personal Equation of Interaction in a
following self-titled section.

4 The Human Cognition Model

In earlier work ([12], [13], [14]) we outlined an operational framework, the Hu-
man Cognition Model (HCM), whose objective was to inform customization of
human-computer cognitive collaboration in mixed-initiative interactive systems.
Todays information visualization applications tend to be passive; primary inter-
face processes sit and wait for user initiation. This is not a problem if the user
knows exactly where to go and what to do. But for the large semantically-rich
datasets which visualizations are increasingly called upon to capture, and the
complex analytical reasoning the visualization must scaffold and support, a truly
intuitive interface must be capable of initiating a variety of processes on its own.
The HCM identifies many of these tasks and the varieties of cognition the tasks.

The central process identified by the HCM is Knowledge Discovery. (See
Figure 3.) This was envisioned as a human and computer paired process: the
interface presents information and the human user indicates interest in a spe-
cific area or point, which the computer in turn presents in a related context. If
Knowledge Discovery is goal oriented, the human will, among other processes,
use temporally moderated perception, semantic categorization, and elimination
reasoning heuristics to search and filter through the information space. If the
discovery is not goal-oriented, the user may browse, stopping the explore data
items that stand out or that are associated to items of interest.
Other processes in the HCM include information search by pattern and ex-

<figure><img src="image_4.png" /><figcaption>Fig. 3. The Human Cognition Model[13].</figcaption></figure>

ample. Search is an interesting cognitive task, as it is deceptively simple. Some
types of search are simply target identification, utilizing perceptual logic, ma-
nipulating the interface and information space through a procedural script, and
using the simplest of elimination heuristics (a binary filter that asks a question:
Is this the specific item Im looking for?). Other types of search can be much more
complex. When the task is to find items similar to an exemplar, for example, all
the cognitive processes from the simpler search tasks serve to feed more complex
processes, such as inferential and deductive reasoning, which utilize more com-
plicated models or rules for comparison and contrast. Thus, even in the more
routine of interface tasks, complex cognition cannot be ignored.

The HCM also outlines the creation and analysis of hypotheses. As discussed
in previous sections, hypotheses can be created in a variety of ways, from the
loose associations in abduction to the use of more demanding heuristics. Some-
times hypothesis are brought to the task by the user, and drive the interaction
from the start. Wherever hypotheses are generated, they serve an important
function. They drive discovery and search behaviors, they determine how information is viewed and filtered, and they promote some data-derived conclusions over others. For these reasons, interfaces which promote the generation of valid hypotheses, either through framing, adaptive search, or more intuitive interaction, might be considered more valuable than others.

Other pertinent processes discussed in the HCM literature (e.g. [13]) include an interface being aware of and predicting user intent in order to keep pertinent information visible, supporting human working memory, caching data subsets of interest to introduce them in a sequence and timing that will support the flow of reasoning and ongoing discovery, and conducting analyses and providing their findings in a contextual framework, which supports a variety of hypotheses generation. In short, the HCM sketches out an interface that collaborates on cognitive processes per se, informed by a growing understanding of human preferences, abilities and limitations.

# 5 The Personal Equation of Interaction (PEI)

Humans are cognitive individuals. As we have seen, humans individuality influences cognitive performance. These differences, as discussed in Section 3, shape the way we approach and perform cognitive tasks. We have discussed personality and self-beliefs in this chapter for sake of brevity, but we are also aware that humans also exhibit differences in psychophysical characteristics, such as perceptual categorization, focused attention, and haptic preferences. These individual variations interact with each other and the task to which they are applied in manners not yet understood.

Further, as we have seen, there is great variety and complexity to analytical tasks, and so it makes sense that not all cognitive tasks would be equally impacted by the same inherent differences. For example, in our research, we have found that persons who tend to believe in that good things that happen to them are due to "luck" (an external locus of control) are predictably slower in target identification. [15]. But the same cannot be said for more cognitively complex problems, such as comparing and contrasting multi-dimensional glyphs [16]; for these tasks, believing in luck seems to give users a decided advantage. (See the section on Research in Personal Equation of Interaction.) It makes sense, then, that not all cognitive tasks would be equally impacted by the same inherent differences; some reasoning tasks may be better predicted by one collection of inherent traits over others.

## 5.1 The Personal Equation of Interaction Defined

Our goal of parameterizing a general model in order to predict performance of a particular individual builds upon foundational work in human perception conducted in the early 19th century by Friedrich Bessel [2]. Bessel recognized that the variability of reports of astronomical observations could be separated into the
differences between the average ratings of observations made by each individual
observer (in statistical terms, the between-subject variance around the global
mean) and variation within observations made by a given observer (the within-
subject variance around that individual's mean for a given task and situation).
While within-subject variability could not be easily factored, the deviation from
the overall mean judgment for a given individual was relatively consistent over a
range of similar situations. The error for a given individual could be measured to
generate a "personal equation" for that individual. This could be used to factor
out their characteristic error to bring their data into agreement, more or less,
with the global mean. This meant that data from fewer observers were needed
in order to achieve the same level of precision. In addition, one could predict any
given observer's raw measurement to a reasonable degree of accuracy given the
global mean and their personal equation.

Much of the research [32], [9], [16] in our laboratory has been devoted to
defining a modern version of the personal equation, the "personal equation of
interaction". This is defined as the quantifiable interaction of human perceptual,
motor, and cognitive limitations with tasks and perceptual stimuli that are gen-
erated by modern visual information systems. In Po et. al., we demonstrated that
system variables such as cursor visibility and display lag interact with individual
differences to produce characteristic patterns of behavior for subpopulations of
individuals [32]. These effects are unlikely to be observed by perceptual testing
that does not focus on the particular characteristics of active visual information
displays and how they differ from those we experience in the physical world that
have informed both human evolution and visual experience of individuals.

There are three goals in this effort: first, to to predict how a given individ-
ual will perform on a given task and information display; second, to build a
framework for interface design that to support customization of a visual infor-
mation system along dimensions that are psychologically valid (i.e. that would
track aspects of individual differences in such a way that their accurate fit to an
individual's capabilities would measurably improve their performance with the
system); and lastly, to build increasingly accurate and comprehensive estimates
of personal equations and methods for assessing them. This includes both persis-
tent differences between individuals (e.g. color blindness) and short-term shifts in
capabilities (e.g. their performance under stress). The latter could potentially be
generated "on the fly" by an attentive system and applied as conditions changed
to maintain optimal performance over changes in the capabilities of the human
operator.

Our approach builds on existing psychometrics methods and materials. How-
ever the goal of this particular line of inquiry is to build towards a natural science
of human-information interaction. By focusing on the specific kinds of changes
in perceptual and interactive experience that are generated by modern visual
information systems, we can address how changes in the properties and statis-
tical regularities of information displays interact with the human visual system
in general, and that of an individual observer in particular. For example, many
studies in perception (e.g. [25]) show how our ability to parse complex visual
scenes given limited perceptual information (the so-called "poverty of the stimulus") is supported by our internal expectations, which in turn are built through a lifetime of sampling the statistical regularities of our physical environment. Phenomena such as change blindness [18], [33] demonstrate the adaptation of human vision to an environment where abrupt changes are rare. Our visual system is not optimized to detect those changes, but pays a small price for this in the physical world. Active updating of visual displays increase the frequency of abrupt display events, and the probability increases that one will coincide with the observer's saccadic eye movement and so escape detection. Thus we find that the study of human performance with information systems cannot simply rely on applying perceptual and cognitive science research taken from the literature. It is necessary to actively seek answers to questions that arise from the use of information systems in cognitive task performance, and to do so using methods from the natural sciences.

It is an open question to what extent aspects of the PEI are inherent and what aspects are acquired through experience with complex visualizations. Some factors such as low vision, color and stereo blindness etc. clearly fall into the former category. Estimation of these factors might require psychophysical testing of a given individual, but may also be estimated by covariates of the given conditions, whether or not we can establish a causal link between them. To the extent that these covariates are predictive, they can be used to support the design and customization of information systems now, as well as contributing to ongoing research about human differences.

The first clustering factor for individual differences is through cognitive experience, e.g. in a given institution or profession. Members of a professional or skilled cohort tend to share jargon and conceptual understanding. Additionally, they are often practiced in methodologies and task heuristics that are specific to the group. These methodologies become a part of the way in which the user searches for and uses information, and can impact the conclusions drawn. They can also introduce group-specific biases that might not be found in the general user population. For these reasons among others, understanding an institutional user profile is important to the design of an expert system interface. A second factor might be perceptual and perceptuomotor experience through interaction with some specific environment.

We define the Personal Equation of Interaction as a compilation of predictive measures based upon inherent individual differences, including but not limited to personality; each measure will be validated to predict performance for one type of cognitive task integral to analysis. The PEI has three current and future end goals: the prediction of analytical performance based on inherent differences [15], [16], the ability to inform real-time interface individuation, and the creation of fuller-bodied user profiles, broken down by the reasoning tasks performed [17].

The first goal – that of performance prediction – is being undertaken through a series of human research studies. Participants are asked to complete a series of tasks similar to common interface tasks, such as interface learnability, target identification, categorization, etc. The measured outcomes vary by task, and in-
clude completion times, errors, and free response feedback. In addition to the
performance and qualitative feedback, participants are asked to complete hun-
dreds of items from a battery of psychometric measures we have chosen for their
inter-relatedness and their relationships to learning outcomes in the behavioral
literature. Post-study analysis includes evaluating the trending of the psychome-
tric items with measured outcomes. In addition, followup testing such as factor
analysis is done to isolate highly predictive psychometric items or groups of
items, such as in [15] for a particular type of reasoning task. This allows us to
delineate our findings by cognitive process and compare multiple interfaces or
problem-solving environments in hopefully a more even fashion. Currently, we
can predict performance on simple outcomes like completion times in a variety
or common interface tasks; these findings are being replicated and expanded.
Not surprisingly, this research is currently quite fluid, and continues to inform
the second goal of what matrices will be needed to support real-time interface
adaptation. In addition, having hundred of participants complete these studies
has allowed us to sketch out initial user profiles, or describe inherent character-
istics of a user based how the user performs on an analytical task [17].

The goal of the PEI is not, at least in the short term, to tell designers and
developers specifically which visualization techniques to use and which to avoid
generically, but rather to give interface creators a robust understanding of what
the individual analyst needs in order to optimally support the analytical tasks
that must be performed. The Personal Equation of Interaction does not replace
interface design; it augments design by making designers aware of user strengths
and weaknesses. It cannot replace user studies for a particular interface, but it
provides new metrics with which to evaluate study outcomes. And as a research
program, it amplifies visual analytics as the study of analytical reasoning sup-
ported by interactive visual interfaces by adding to the body of understanding
on analytical reasoning and analytical reasoners.

These are aggregated to build an ever more comprehensive and accurate
personal equation of interaction that could be used by an application to para-
metrically modify its display of information in such a way as to optimize the
cognitive performance of an individual decision-maker. As research progresses
we hope to find a way to integrate system models with models of human inter-
action to better predict the course of fluent human-information interaction.

## 5.2 Research in the Personal Equation of Interaction

Our research has demonstrated that inherent differences can and do influence
learning and analytical performance during interface interaction. In combination
with the environmental and institutional variations, there is evidence that the
impact of inherent differences could theoretically be used to derive a personal
equation of interaction.

Our recent research has demonstrated that personality factors can predict
efficiency during varying task types [15]. We designed a series of tasks we asked
participants to complete in two visual interfaces using the same dataset: menu-
driven web application, and an information visualization using direct interaction
on hierarchical graphs. These tasks were designed to test two very different types
of learning: procedural and inferential. Procedural learning, as defined for this
study, was the ability to use the interface to find target information. This type
of learning tends to be inductive: a rule is inferred which generalizes to other
similar target identification tasks in the interface. Other the other hand, the
inferential learning tasks were highly deductive. Participants were asked to eval-
uate a multi-dimensional exemplar and find another conceptual object in the
hierarchy that was similar to (or different from) the exemplar for the specified
dimensions. This type of reasoning involves the creation of a mental model, which
is then used to evaluate complex concepts to reach a valid conclusion. For each
task, we tracked errors, completions, as well as qualitative feedback.

In addition, we administered several longstanding and well-documented psy-
chometric measures to participants [15]. These measures were created to measure
personality traits such as neuroticism (a tendency toward emotional instability),
extraversion (a tendency toward sociability or seeking the company of others),
and trait anxiety, which is a tendency to be more anxious generally, regardless of
the environment. Trait anxiety differs from state anxiety, which is the tendency
to be anxious when in a situation that triggers anxiety.

Another personality trait that proved to have a demonstrable impact was
locus of control, which is a measure of how in control a person feels he or she is
over the events in life. Persons with an external locus tend to believe strongly
that they are not in control, and attribute events to factors outside themselves,
such as luck, other people, or circumstances outside of their control. On the other
hand, persons with an internal locus tend to believe that they are responsible for
both positive and negative events in their lives. They are more likely to attribute
events to some behavior or attitude of their own than to outside influences, and
tend to give very little credence to luck.

Other measures designed to test other personality factors, such as a discomfort with problem-solving situations where important factors are unknown (an intolerance of ambiguity) or self-regulation, which is the ability to hold it together emotionally when the problem or situation becomes difficult, were also evaluated but were not found to be particularly predictive in performance of the tasks under study.

**Results of Study 1.** Results demonstrated [15] that whole score locus of control predicted inferential task efficiency in the data visualization; participants with a more external locus were able to complete the deductive inferential tasks more quickly.

For the inductive procedural tasks, no singular full measure could predict
behavior, which was not unexpected, given that none of these psychometrics
were designed to evaluate these traits in this interface environments. But factor
analysis uncovered 9 items, largely from the trait anxiety measure, that pre-
dicted target identification efficiency across both interfaces. Participants who
were more trait anxious found target items more quickly, even when the target was buried several layers down in the hierarchy.

Results indicated that no single personality factor measure could predict errors in either interface.

<table><caption>Fig. 4. Items in the 9 item short measure. Adapted from [15].</caption><thead><tr><th>Item<br>(Originating Meaure)</th><th>How scored</th></tr></thead><tbody><tr><td>Unable to Relax<br>(Beck's Anxiety Inventory)</td><td>0(never) to<br>3(severely)</td></tr><tr><td>Fear the Worst<br>(Beck's Anxiety Inventory)</td><td>0(never) to<br>3(severely)</td></tr><tr><td>Heart Pounding<br>(Beck's Anxiety Inventory)</td><td>0(never) to<br>3(severely)</td></tr><tr><td>What we are used to is<br>always preferable to<br>what is unfamiliar.<br>(Tolerance of Ambiguity)</td><td>1(strongly disagree<br>to 7(strongly agree))</td></tr><tr><td>Talk to a lot of people.<br>(Extraversion)</td><td>1(low) to 5(high)</td></tr><tr><td>Hands Trembling<br>(Beck's Anxiety Scale)</td><td>0(never) to<br>3(severely)</td></tr><tr><td>Numbness<br>(Beck's Anxiety Inventory)</td><td>0(never) to<br>3(severely)</td></tr><tr><td>Don't talk a lot.<br>(Extraversion)</td><td>1(low) to 5(high)</td></tr><tr><td>Am easily disturbed.<br>(Neuroticism)</td><td>1(low) to 5(high)</td></tr></tbody></table>

**Results of Study 2.** Results of a similar study using procedural tasks [16], currently under submission) expanded these findings somewhat. This study used the same interfaces and similar procedural tasks. The scores and outcomes of participants in both studies were combined for greater statistical power (N = 105). Results demonstrated that both neuroticism and extraversion predicted efficiency; the more neurotic/extraverted participants found items more quickly. Additionally, analysis of the combined set found that locus of control predicted procedural performance, in directly the opposite way to that of the inferential
tasks. Participants with an internal locus (a belief that they were in control of life events) found targets more quickly than those with an external locus. This evidence alone demonstrates that not only that personality factors affect interface interaction performance, but that different tasks are impacted differently by inherent individual differences. See Figure 5.

<table><thead><tr><th></th><th>Completion times</th><th>Errors</th><th>Insights</th></tr></thead><tbody><tr><th>Interface</th><td>faster times in MapViewer</td><td>fewer errors in MapViewer</td><td>more insights in Gvis</td></tr><tr><th>Locus of Control</th><td>internal locus faster times</td><td>none</td><td>external locus more insights</td></tr><tr><th>Extraversion</th><td>more extraverted faster times</td><td>none</td><td>less extraverted more insights</td></tr><tr><th>Neuroticism</th><td>more neurotic faster times</td><td>none</td><td>less neurotic more insights</td></tr></tbody></table>

5.jpg

<caption>Fig. 5. Summary of findings from Study 2. From [16].</caption>

**Discussion of Results** The existence of significant trending between personality factors and interface interaction outcomes is interesting for a variety of reasons. First, it demonstrates that even complex cognition can, at least to some degree, be predicted. Secondly, it demonstrates that inherent individual differences, over which we as designers have no control, could inform design if we knew the psychometric makeup of our target user group. This holds potential for experts systems, which are designed for users whose differences are likely to trend in similar ways. Thirdly, these studies open a promising doorway; if these few personality factors can predict performance, what else about complex cognition might we be able to predict if we knew more about our users, as well as about the expert cohorts for whom we design visually enabled interfaces?
# 6 Conclusion

The reasoning used during task analysis is complex. In this chapter, we have discussed this complexity by highlighting a handful of reasoning heuristics. We have underscored this complexity with a discussion of Pirolli and Cards sensemaking loop. And we have explored how this complexity complicates the current state of design and evaluation thanks to the absence of applicable reasoning research and pertinent precedent in the behavioural literature.

We have also broadly discussed the impact of human individuality on every primary cognitive process, and surveyed our current research in pursuit of the generation of new system development models that optimize the cognitive performance of human decision-makers. Optimization in this context must include complex criteria such as insight, innovation, creativity and awareness in uncommon, unique and novel problems and situations. Research has shown that inherent individual differences between users impacts the task and learning performance in visually embedded interfaces. Our previous work in the development of the Human Cognition Model continues to inform our research direction. Our ongoing research in the Personal Equation has highlighted the need to study not only inherent differences in personality factors, but also other user differences, including those in which affect other inherent individualities as well as differences in institutional cohort and environment. These individual differences in human capabilities are great enough that any unitary system will be at best a compromise between the needs of the various sub-populations of users. Our ongoing research seeks to take advantage of human individuality, rather than to ignore it. While still in the early stages of this research, we have already highlighted several inherent differences which predict performance, depending on reasoning task. We intend to explore further, with the expectation of isolating and understanding influential individual differences and how they impact interface interaction, which could benefit visual analytics interface designers by informing design requirements and opening up new areas for innovation.

# References

1. Aoidh, E.M., Bertolotto, M. Wilson, D.C.: Understanding geospatial interests by visualizing map interaction behavior Information Visualization, 7: 27586. (2008)

2. Brooks, G., Brooks, R.: The Improbable Progenitor Journal of the Royal Astronomical Society of Canada, 73(1), 9-23. (1979)

3. Card, S., Mackinlay, J.D., Shneiderman, B.: Readings in Information Visualization Using Visualization to Think. San Francisco: Morgan Kaufmann, (1999)

4. Chang, R., Ziemkiewicz, C., Green, T.M., Ribarsky, W.: Defining insight for visual analytics. Computer Graphics and Applications 29(2), 14-17. (2009)

5. Cherubini, P., Johnson-Laird, P.N.: Does everyone love everyone? The psychology of iterative reasoning. Thinking and reasoning 10(1). 31-53. (2004)

6. Cherubini, P., Mazzocco, A.: From models to rules: Mechanization of reasoning as a way to cope with cognitive overloading in combinatorial problems. Acta Psychologica. 116. 223243. (2004)
7. Corn, A.L.: Visual function: A theoretical model for individuals with Low Vision. Journal of Visual Impairment and Blindness, 77(8). 373. (1983)

8. Dou, W., Jeong, D.H., Stukes, F., Ribarsky, W., Lipford, H.R., Chang, R.: Recovering reasoning processes from user interactions. IEEE Computer Graphics and Applications 29(3):5261. (2009)

9. Fisher, B. Science and Smart Graphics. Information Technology 3,142-148. (2009).

10. Gentzen, G.: Investigations into Logical Deduction. Translation printed in M. Szabo The Collected Papers of Gerhard Gentzen (Amsterdam: North-Holland) 1969, pp. 68-131. (1934/5)

11. Gigerenzer G., Goldstein D.G.: Reasoning the fast and frugal way: Models of bounded rationality. Psychological Review, 103(4). (1996)

12. Green, T.M., Ribarsky, W., Fisher, B.: Visual analytics for complex concepts using a human cognition model. Proceedings of IEEE Visual Analytics Science and Technology, October 2008, Columbus, OH, USA. (2008)

13. Green, T.M., Ribarsky, W.: Using a human cognition model in the creation of collaborative knowledge visualizations. Proceedings of Hot Topics in Visual Analytics, SPIE Defense + Security, Orlando, Florida, USA. (2008)

14. Green, T.M., Ribarsky, W., Fisher, B.: Building and applying a human cognition model for visual analytics. Information Visualization 8(1), 1-13. (2009)

15. Green, T.M., Jeong, D.H., Fisher, B.: Using personality factors to predict interface learning performance. Proceedings of Hawai'i International Conference on System Sciences 43, January 2010, Koloa, Hawai'i, USA. (2010)

16. Green, T.M., Fisher, B., Jeong, D.H.: Towards the personal equation of interaction: The impact of personality factors on visual analytics interface interaction. Proceedings of IEEE Visual Analytics Science and Technology, Salt Lake City, UT, USA. (2010)

17. Green, T.M., Fisher, B., Jeong, D.H.: The impact of personality factors on interface interaction and more robust user profiles: Next steps in the Personal Equation of Interaction. Journal of Management Information Systems, Special issue on Hawaii International Conference on System Sciences 43. (2010, invited, under review)

18. Grimes, J. On the failure to detect changes in scenes across saccades in Akins, K., Perception (Vancouver Studies in Cognitive Science), 2, New York: Oxford University Press, pp. 89110. (1996).

19. Heaven, P.C.L. Quintin, D. S.: Personality factors predict racial prejudice. Personality and Individual Differences, 34. 625-634. (2002)

20. Heer, J., Agrawala, M.: Design considerations for collaborative visual analytics. IEEE Symposium on Visual Analytics Science and Technology. Sacramento, CA. IEEE Computer Society. (2007)

21. Jaskowski, S. : On the Rules of Suppositions in Formal Logic. Studia Logica v.1. Reprinted in S. McCall (1967) Polish Logic 1920-1939 (Oxford: Oxford Univ. Press) pp. 232-258. (1934)

22. Jeong, D.H., Green, T.M., Ribarsky, W., Chang, R: Comparative Evaluation on Two Interface Tools in Performing Visual Analytics Tasks Proceedings of BELIV Workshop, CHI 2010, Atlanta, GA, USA. (2010)

23. Johnson-Laird, P. : How We Reason. New York. Oxford University Press. (2008)

24. Kozielecki, J..: Elements of a psychological decision theory. Studia Psychologica, 13(1), 53-60. (1971)

25. Marr, D. Vision MIT Press, Cambridge MA. (1982)

26. Macia, A.: Visual perception of landscapes: Sex and personality differences. Our national landscape: A conference on applied techniques for analysis and management of the visual resource, General Technical Report PSW-35: 279-85. (1979)
27. North, C.: Toward measuring visualization insight. IEEE Computer Graphics and Applications, 26(3):69. (2006)

28. Palmer, J.: Scientists and information: II. Personal factors in information behaviour. Journal of Documentation, 3, 254-275. (1991)

29. Plaisant, C., Grosjean, J., Bederson, B. B.: SpaceTree: Supporting exploration in large node-link tree: design evolution and empirical evaluation. In Proceedings of IEEE Symposium on Information Visualization. 57-64. (2002)

30. Plaisant, C: The challenge of information visualization evaluation. Proceedings of the Working Conference on Advanced Visual Interfaces, Gallipoli, Italy, (2004)

31. Pirolli, P., Card, C.: The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. Proceedings of International Conference on Intelligence Analysis. McLean, VA, USA. (2005)

32. Po, B., Fisher, B., Booth K. S. Pointing and Visual Feedback for Spatial Interaction in Large-Screen Display Environments. Springer Lecture Notes in Computer Science Smart Graphics 2003 Heidelberg, Germany. Butz, A., Krger, A., Olivier, P., Lexicle Ltd, York, UK (Eds.) (2003).

33. Rensink R.A., O'Regan J.K., and Clark J.J. To see or not to see: The need for attention to perceive changes in scenes. Psychological Science, 8:368-373. (1997)

34. Rips, L. J.: Inductive judgments about natural categories. Journal of Verbal Learning and Verbal Behavior, 14, 665-681. (1975)

35. Robinson, A.: Collaborative synthesis of visual analytic results. In IEEE Visual Analytics Science and Technology, Columbus, OH USA. 6774, Oct. 2008. (2008)

36. Rock, I.: The logic of perception. MIT Press/Bradford Books, Cambridge, MA. (1985)

37. Ronan, G.F., Senn, J., Date, A., Maurer, L., House, K., Carroll, J., Vanhorn, R.: Personal problem solving scoring of TAT responses: Known groups validation. Journal of Personality Assessment, 67. (1996)

38. Russel, D. Card, S.: The Cost of Sensemaking. In Proceedings of Interact 1993. (1993).

39. Saraiya P., North C., Duca K.: Evaluation of microarray visualization tools for biological insight. IEEE Symposium on Information Visualization. (Austin, TX, USA); 2004; 18. (2004)

40. Saraiya, P. North, C. Duca, K.: An Insight-Based Methodology for Evaluating Bioinformatics Visualizations. IEEE Transactions on Visualization and Computer Graphics. 11. 443-456. (2005)

41. Sebeok, T.: You Know My Method. In Sebeok, T. The Play of Musement. Bloomington, IA: Indiana. (1981)

42. Shneiderman, B.: The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations. In Proceedings of the IEEE Symposium on Visual Languages, Washington, DC. IEEE Computer Society Press, 336-343. (1996)

43. Spence,R.: InformationVisualization, Addison-Wesley. (2001)

44. Springmeyer, R.R., Blattner, M.M., Marx, N.L.: A Characterization of the Scientific Data Analysis Process. Proceedings of IEEE Visualization 1235-1242. (1992)

45. Stanovich K.E., West, R.F.: Individual differences in reasoning: Implications for the rationality debate? Behavioral and Brain Sciences, 23. 645 726. (2000)

46. Stanovich, K.E.: Who is rational? Studies of individual differences in reasoning. Erlbaum. Mahwah, NJ. (1999)

47. Stasko, J., Gorg, C., Liu, Z., Singhal, K.: Jigsaw: Supporting Investigative Analysis through Interactive Visualization. In: J. Dill and W. Ribarsky (eds.) Proceedings of 2007 IEEE Symposium on Visual Analytics Science and Technology; Sacramento, CA: IEEE CS Press, 131138. (2007)
48. Sternberg, R. J.: Intelligence, Information Processing and Analogical Reasoning. Hillsdale, NJ: Erlbaum. (1977)

49. Sternberg, R.J., Gardner, M.K.: Unities in inductive reasoning. Journal of Experimental Psychology: General 112(1). 80-116. (1983)

50. Tversky, A.: Elimination by aspects: A theory of choice. Psychological Review, 79, 281-299. (1972)

51. Wang, T.D. Plaisant, C. Quinn, A.J. Stanchak, R. Murphy, S. Shneiderman, B.: Aligning temporal data by sentinel events: discovering patterns in electronic health records. Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy. (2008)

52. Beth, E.W., Piaget, J.: Mathematical Epistemology and Psychology. D. Reidel, Dordrecht, The Netherlands. (1966).

53. Johnson-Laird, P.N.: Deductive Reasoning. Annual Review of Psychology. 50. 109-135. (1999).

54. Johnson-Laird, P., Byrne, R.M.J.: Deduction. Psychology Press (1991)