**Neo-symbiosis: A Conceptual Tool for System Design**

Doug Griffith

*Cognitive Solutions Laboratory*

*General Dynamics Advanced Information Systems*

*douglas.griffith@gd-ais.com*

[Abstract]{.underline}

In 1960 Licklider advanced the notion of symbiosis between humans and
computers that would think as no human brain has ever thought. This
paper updates Licklider's vision of symbiosis and recasts it into
current theories of cognition. This updated version of Licklider's
vision is termed neo-symbiosis. Kahneman's notion of two processing
systems provides a useful theoretical framework for capturing both
expertise and information processing biases. System 1, termed Intuition,
is fast and effortless. It is simultaneously the source of much human
expertise, while also being the locus of cognitive and perceptual
illusions. System 2, termed Reasoning, is the locus of rational thought.
It also has the task of monitoring System 1 output. Computers need to
support System 2 processing. A neo-symbiotic design philosophy is
presented followed by an illustrative example. Evaluative metrics are
discussed. It is concluded that even absent evaluation metrics,
neo-symbiosis provides a desirable design goal.

[Introduction]{.underline}

Licklider's vision of human/computer symbiosis provided a grand goal for
human/sys­tem design. Licklider wrote, "The hope is that in not too many
years, human brains and computing machines will be coupled toge­ther very
tightly, and that the resulting partnership will think as no human brain
has ever thought and process data in a way not approached by the
information-handling machines we know today" \[19\]. Unfortunately, it
appears that this vision has, for the most part, been lost. The focus of
human factors has been on usability. Although usability is a necessary
condition for symbiosis, it is not a sufficient condition \[7\]. It is
wonderful to have the opportunity to participate in this symposium. Let
us hope that Licklider's vision has returned and that this symposium
will foster and promote that vision.

Neo-symbiosis is an effort to update Licklider's vision. A number of
objections have been raised to Licklider's notion of symbiosis. One is
that the term "man-machine" symbiosis is politically incorrect.
Neo-symbiosis substitutes the more acceptable term "human-machine"
symbiosis. A second objection is that symbiosis implies symmetry between
humans and machines. Under the neo-symbiosis concept, the human always
maintains the super-ordinate position.

Kahneman's Nobel Prize Lecture, *Maps of Bounded Rationality: a
Perspective on Intuitive Judgment and Choice* \[12\] and subsequent work
\[13\], \[14\] have what some might find to have surprising relevance to
symbiosis and to expertise. This paper elucidates that relevance. A
basic conceptual framework is presented for system design. Finally, some
metric and evaluative issues are discussed.

[Two Processing Systems]{.underline}

The idea that cognitive processes can be partitioned into two main
families, intuition and reasoning, is not new. They are known as dual
process theories \[2\], \[8\], \[23\]. Kahneman and Frederick \[14\]
adopted the generic labels System 1 and System 2 from Stanovich and West
\[25\]. They invoked these two cognitive systems to organize seemingly
contradictory results in studies of judgment under uncertainty \[23\],
\[22\], \[24\]. System 1, termed Intuition, is fast, effortless,
associative, slow-learning, and emotional. System 2, termed Reasoning,
is slow, serial, controlled, effortful, rule-governed, flexible, and
neutral. Cognitive illusions as well as perceptual illusions are the
results of System 1 processing. Expertise is primarily a resident of
System 1. So is most of our skilled performance such as recognition,
speaking, driving, and many social interactions. System 2, on the other
hand, consists of conscious operations, such as what is commonly thought
of as thinking. System 1 is effective presumably due to evolutionary
forces, massive experience, and constrained context. Most of the time it
is quite effective. It is using unconscious heuristics to achieve these
efficiencies, so occasionally it errs and misfires. System 1 misfires
are responsible for perceptual and cognitive errors. One of the roles of
System 2 is to monitor the outputs of System 1 processes.
Phenomenologically this is what happens when an idea flashes to mind,
but one stops to think about it. There are obvious costs to processing
efficiency here, but this quality control mechanism is necessary to
reduce the number of System 1 processing errors. It is worthwhile to
note that Schacter in his *Seven Sins of Memory* \[22\], while not
invoking the System 1/System 2 distinction *per se*, was making a
similar argument in that these memory sins were actually the result of
adaptive processes that were the product of evolutionary forces.
Anderson made a similar argument in his *Adaptive Character of Thought*
\[1\].

[Expertise]{.underline}

It is instructive to consider the concept of expertise within the System
1/System 2 framework. Here the Dreyfuses \[4\] have provided a
convenient perspective from which to view expertise. They have
characterized the development of expertise in the following five stages:
The first stage is that of the *novice* where the individual is learning
to recognize various objective facts and features relevant to the skill
as well as basic rules for performance of a task. Stage 2 is that of the
*advanced beginner* that is achieved after the novice has had
considerable experience in real world settings and has learned to
consider more context-free facts and more sophisticated rules. Stage 3,
*competence*, comes with yet more experience and the number of
recognizable context-free and situational elements present in real-world
circumstances becomes overwhelming. Stage 4, *proficiency*, is
characterized by the transition from rule-based behavior to more
intuitive (System 1) processing. At Stage 5 *true expertise* is achieved
wherein the skill has become so much a part of the individual that there
is no awareness of it. At Stage 5 behavior is arational. Arational
behavior refers to action without conscious analytic decomposition and
recombination. Performance is rational through Stage 3. That is, it is
characterized by rational processing. The transition to arational
behavior, and intuitive processing, occurs during Stage 4. Experts act
arationally, according to the Dreyfuses.

When viewed in terms of System 1 vs. System 2 processes, stages 1
through 3 involve System 2 processes solely. Stage 4, proficiency, is
characterized by the transition from System 2 to System 1 processes. One
might conclude from the statement that Stage 5 arational behavior occurs
without conscious thought and that expert behavior consists solely of
System 1 processing. Clearly this is in error as it implies that experts
function like automatons, with no conscious awareness. A more plausible
conception is that expertise consists of an interaction between System 1
and System 2 processing. System 2 processes are those of which the
expert is conscious and which can be accessed through conventional
knowledge engineering. System 1 processes are those that have become
automated through thousands of hours of experience and are not available
to consciousness.

Typically, expertise is thought of in terms of unique individuals with
highly specialized skill, *e.g.,* chess grand masters. However,
virtually every member of *Homo sapiens* has expertise that is fairly
general across the species. For example, most perceptual processes and
language should be regarded as expertise. These skills have been honed
through thousands of hours of practice and operate primarily below the
level of conscious awareness, *i.e.,* they are System 1 Processes. *This
point is critical for considerations of human/computer neo-symbiosis.*

System 1 processes are those which humans do well and quickly. The
Dreyfuses would argue that some of these processes are unique to humans
and cannot be done by computers. On the other hand, System 2 processes
are performed slowly by humans and are prone to error. System 1
processes can also misfire. These misfirings of System 1 processes
provide the bases for perceptual and cognitive illusions according to
Kahneman. Another role of System 2 processes is to monitor System 1
processes. Perceptual and cognitive illusions are the result of System 1
misfirings and the failure of System 2 to catch the System 1 misfirings.

[Neo-symbiotic Design]{.underline}

At a top level, neo-symbiotic design employs the computer to supplement
human System 2 processing and to leverage and monitor System 1
processing. Thus, both speed and accuracy are enhanced via computer
support, while information processing biases and illusions are mitigated
by computer monitoring. On the other hand, the brittleness and
limitations of the computer are compensated for by the human.
Brittleness refers to the tendency of computers to lock up or crash at
inopportune times. Whether there are fundamental limitations regarding
computer capabilities is an empirical issue, ultimately. However, there
are practical limitations for any given state of technology for which
the human needs to compesate. Neo-symbiosis should lead to Licklider's
vision , *i.e.,*"...human brains and computing machines will be coupled
together very tightly, and that the resulting partnership will think as
no human brain has ever thought and process data in a way not approached
by the information-handling machines we know..." \[19\].

Of course, as is usually the case, the top level is the easy part. It is
the lower implementation levels that present difficulties.

Earlier it was mentioned that usability was a necessary, but not a
sufficient condition, for neo-symbiosis. Usability is, however, a
necessary condition, so if it is not achieved, there is little sense in
going further. Much has been written about usability, so little more
will be written here. It should be clear, however, that it is the
computer that needs to accommodate the human rather than, as is too
often the case, the human that must make accommodations for the
computer. Symbiosis has connotations of two biological organisms living
together with mutual benefit, with no position taken on the involvement
of conscious awareness. However, neo-symbiosis has the human in the
super-ordinate position with conscious awareness. Whether computers can
have conscious awareness is a debate for another forum.

Neo-symbiosis can best be implemented by some sort of avatar or software
agent. Again, one role for this agent is to supplement System 2
processes. According to Kahneman, one of the roles of System 2 is to
monitor System 1 for processing errors. Fortunately, the literature on
human information processing shortcomings is voluminous. Indeed, much of
the field of human factors is concerned with modeling and compensating
for human information processing limitations. *How We Know What Isn't
So* by Gilovich \[5\] provides an interesting compendium of entertaining
cognitive errors. Although Heuer's *Psychology of Intelligence Analysis*
\[10\] addressed the effects of information processing shortcomings and
biases on intelligence analysis in particular, the volume is relevant to
knowledge workers in general. One way of viewing these volumes is as a
compendia of System 1 errors.

Neo-symbiosis dictates the need for two complementary types of user
models. A nomothetic user model is needed to specify generally human
information processing capabilities. Included here are information
processing biases and cognitive and perceptual illusions. Given the
extensive literature on these topics, this is, by far, the easier of the
two models. The second type of user model is an idiographic model.
Idiographic models are unique to each individual user. Although an
idiographic model needs to account for weaknesses and biases specific to
an individual user, its major purpose is to capture information
processing strengths and areas of expertise that can be leveraged. More
will be written about the idiographic model later in the paper.\
Having an inventory of human information processing shortcomings is one
matter. Having the computer technology to identify them is another. At a
minimum, the agent needs information on the task and goals the human is
trying to achieve. Then the agent could issue alerts at the beginning of
a task and a summary of computer supports that could be supplied. As the
task nears conclusion, the agent could offer checklists to help the
human find possible errors that might have been committed. This is what
could be offered at a primitive level. At an advanced level, documents
could be checked for common information processing errors much as spell
checkers or style checkers do today. The human could either accept or
reject the suggestions made by the agent.

Ideally, this agent could intervene during a task to prevent errors and
unnecessary effort. The problem here, of course, is knowing when to
intervene. If this is done too frequently and at inappropriate times,
then the agent becomes more of a hindrance than a help. Microsoft's
Clippy has developed a reputation for popping up when it is not wanted
and provides a ready negative exemplar of what is to be avoided. The
agent needs a learning mechanism that would adjust interventions on the
basis of the frequency with which the interventions have been
successful. The human should also be able to set the sensitivity of the
threshold for agent interventions. There is also no requirement that the
agent intervene directly. Rather a signal can be presented indicating
that the agent has a message to convey to its human colleague.

To this point the term agent has been employed rather than avatar, as
avatar has connotations of a graphical image. This intervening agent
need not have a graphical representation. It could effectively
communicate solely through text messages. Whether text messaging would
be sufficient or if a graphical image would be preferred is an empirical
question. However, the likely answer to this empirical question is that
different people have different preferences. Ultimately it is likely
that users would have a choice of text or image, or perhaps even a set
of images, or a build your own avatar kit.

There are two views of the role of the computer in a neo-symbiotic
system. One view is as an information processing prosthetic. The other
role is of the computer as coach. This involves the monitoring of
intuitive processes. Intuitive processes are those processes that either
can be done better by the human, or that cannot be effectively
accomplished via computer. Unfortunately, this delineation of tasks
between human and computer is not that simple. Intuitive processes can
misfire resulting in cognitive and perceptual illusions. As a coach, one
role of the computer is to monitor the information processing of the
human and intervene when a miscomputation is suspected. Another means by
which symbiosis can be achieved is through context elaboration and
enhancement. Humans are context constrained, particularly with respect
to intuitive processing. However, computers are, or can be, context
free. By providing additional contexts, not only are the capabilities of
the humans enhanced, but the likelihood of perceptual and cognitive
illusions should be decreased. As cognitive and perceptual illusions are
the result of heuristics that constrain context, the provision of
additional contexts can mitigate or eliminate these illusions.

Context is an important concept. Communication among humans often fails
due to the lack of a shared context. One can also profitably regard
human/computer interaction problems as being due to a lack of shared
context. Microsoft and others in the computer industry are of the
mistaken opinion that improved speech recognition capabilities will make
computers more usable when, in fact, it is the difficulty in penetrating
the logic or context within the software that is the primary problem.
Having a shared context is critical to effective human/computer
interaction just as it is to effective human/human interaction. The
development of protocols to define a mutual context is critical to
effective communication among humans, and is also critical to achieving
neo-symbiosis.

Another issue for neo-symbiosis is how to leverage human information
processing strengths. Although certain skills run across the species,
*e.g.,* perceptual and linguistic skills, most areas of expertise are
idiosyncratic to individuals. Here the idiographic model needs to be
developed. Of course, systems can be developed for particular areas of
expertise. Then the computer agent can play the role of coach. If there
are digital records of the expert's performance, these can be analyzed
by the computer coach to identify areas of weakness, which need to be
compensated for and developed, and areas of strength, which need to be
leveraged and exploited. Klein \[16\] notes that when multiple options
are perceived by the expert, the expert will run mental simulations to
decide upon the course of action. The sophistication of these mental
simulations can be significantly enhanced with computer tools. Computers
will also allow for the generation of many more simulations than a human
could generate so more options can be evaluated with more precision.

There are a variety of options for building the idiographic model,
*i.e.,* assessing the expertise of an individual user. On-line
questionnaires or interviews provide one means. On-line testing is
another approach. However, nonintrusive techniques might also be
possible. By monitoring the interactions of the human, *e.g.,* level of
prose, level of computations, sources searched, it might be possible to
provide an initial characterization of the user's capabilities that
would be further refined as the result of subsequent interactions.

Neo-symbiotic systems have the definite potential of enhancing the
performance of experts as long as experts are aware of their own
information processing shortcomings. Experts are subject to the same
perceptual and cognitive biases elucidated by Kahneman and others. For
example, Dawes and Corrigan \[3\] reviewed a range of forecasting
problems and found that the experts performed poorer than simple linear
models.

It is interesting to speculate what the nature of the interaction would
be in a highly symbiotic system. The interesting problems are those that
are open ended and deal with uncertainty. For these problems there is no
*a priori* correct solution. The correctness of the solution awaits some
outcome or series of outcomes. Suppose that the human has a strong hunch
as to the outcome courtesy of System 1 processing. However, the
symbiotic counterpart has offered a different, or perhaps several
different, alternative solutions. So what does the human do? Go with the
hunch or accept, largely on faith, an alternative offered by the
symbiotic counterpart. Myers \[20\] *Intuition: Its Powers and Perils*
provides a splendid review of the general problem, whether to accept
System 1 processing or whether to subject the problem to further
consideration. An analogous dilemma has already appeared in mathematics
when computers are incorporated into mathematical proofs. The proof is
too difficult to formulate without the aid of the computer. That being
the case, how can it be checked adequately? A certain element of faith
is involved. This is not unusual in everyday life as we know many things
that we cannot explain. Sometimes the explanations were never known.
Sometimes they were learned in school but forgotten. Although the risk
is one of an infinite regress, intuition about intuitions needs to be
developed. Klein \[15\] contends that intuition can be explicitly
developed and has developed training programs to that end. One option
that can be explored in the symbiotic computer situation is that
simulations can be run regarding the different options, so a choice can
be made on the basis of risk, award, or some cost/benefit function.

[An Example of a Neo-symbiotic System]{.underline}

To make more concrete the concepts being discussed, a brief overview of
a system being developed for analysts who deal with massive, complex,
and sometimes unreliable data will be presented. The system is called
MAUI NITE[^1]. Key to the achievement of human/computer symbiosis is an
avatar to implement the symbiosis of human and machine. The Steward
Agent, henceforth referred to as Steward, serves this function in MAUI
NITE. The Steward monitors the user in an attempt to anticipate the
user' needs and to circumvent any shortfalls or biases in the user. The
Steward stands ready to summon Staff agents, to support the System 2
(Controlled, Deliberative) Processing of the user. It is these processes
for which computer tools are most needed. Staff agents, in turn, are
supported by Scout agents that address the problem of filtering text,
selecting a small subset of potentially relevant documents from a much
larger corpus, and extracting structured information from unstructured
documents (data). Computers are capable of processing much, much more
data than humans are capable of processing. However, there is not even
additivity, much less symbiosis, when data are not transformed into
*information*. Properly designed visualization tools can help, but if
the graphics are too busy, the basic problem remains the same.

Tacit knowledge can be regarded as a product of automatic (intuitive)
processing. MAUI NITE tools provide an example of how a computer system
can employ context to make tacit knowledge explicit. MAUI provides tools
for marshalling evidence. It allows the analyst to apply tacit knowledge
directly against data. When information is presented in a given way, the
analyst has a tacit expectation of how information ought to behave in
that context. When anomalous behavior is observed, the marshalling
strategy draws the analyst's attention to facts that are potentially
important. As MAUI NITE's Concept Index (CI) stores information in a way
that is transparent to both human users and software agents, it should
support a wide variety of marshalling strategies.

Humans have difficulty with quantitative processes. There is a general
lack of facility in computational reasoning. The Steward uses a
cognitive, stylized form of uncertainty to make decisions
algorithmically. This algorithm, called Quantitative Option Assessment
(QOA), enables a software agent to choose among competing hypotheses
about a current or future situation.

![](media/image1.png){width="0.7708661417322835in"
height="0.6688976377952756in"} The Steward has access to other tools for
supporting the controlled, deliberative processes of the user. There is
a structured argumentation tool that complements the System 1
(automatic, intuitive) processing of the analyst. The analyst might be
good at quickly generating an intuitive hypothesis that apparently
works. However, this facility might result in failing to generate or
consider other viable hypotheses. The intelligence analyst is able to
construct a variety of arguments and consider multiple hypotheses with
the aid of this tool.

[Evaluative Metrics]{.underline}

At a conceptual level, it is easy to define neo-symbiosis in terms of
the General Linear Model, *i.e.,*

Computer + Human + Human\*Computer = Performance

where the coefficient for the interaction is positive. This is stating
that the whole is greater than the sum of its parts.

At a practical level, however, it would be difficult to implement a test
in which the individual effects of the computer, the human, and the
interaction between the two could be independently estimated. Even if
such a test could be devised, it might not be plausible and might not
present a compelling case. There are tasks that are performed solely by
humans, but it is likely that these tasks are too difficult or costly to
automate. There are also tasks that are performed solely by computers
with the human being performing only a monitoring role. Unfortunately,
monitoring is not a task humans do well, so such automated systems are
not neo-symbiotic. Most tasks involve the interaction between humans and
computers. Perhaps this means that some level of symbiosis has already
been achieved, but surely not at the level envisioned by Licklider.

Perhaps the most compelling evidence for neo-symbiosis would be if
experts chose to use computer systems to leverage their expertise. That
is, that they used the computer as a coach or associate rather than
simply providing administrative support. Today we have chess
grandmasters playing against IBM's Big Blue. That is human expertise
versus the brute computing force of Big Blue. One can question the
objective of such competitions. It would seem to accomplish little in
the way of advancing computer science. Consider, however, a
neo-symbiotic competition, say a chess grandmaster playing without a
computer associate versus a chess master playing with the computer
associate. Who would win? How far might the different levels of
expertise be pushed? Could a novice playing with a computer associate
defeat a chess grand master? This would appear to be a more interesting
challenge than the chess grand master versus Big Blue. It also would do
more to advance computer science and the neo-symbiotic design of
systems.

One can envision a day where there are two types of chess competitions:
conventional and computer assisted. The computer assisted world
championship would pit two grand masters against each other each playing
with the aid of a personalized computer coach.

[Neo-symbiosis as a Design Goal]{.underline}

Even should it prove difficult to measure, neo-symbiosis clearly
represents a desirable design goal. It provides the means of getting the
most from humans and computer technology. A common reaction to
automation by organized labor and society is that automation will put
people out of work and that there will be a loss of jobs. The reply
usually offered is that automation can result in job loss in the short
term, but in a job increase in the long term. According to the
neo-symbiosis philosophy this short term job loss need not always occur.
First of all, humans should not be replaced by technology. Otherwise
there is the nightmarish prospective future offered by Bill Joy (2000)
in his paper "Why the Future Doesn't Need Us". In Kurzweil's \[17\]
optimistic depiction of the future in *The Age of Spiritual Machines:
When Computers Exceed Human Intelligence,* certain assumptions need to
be made for this future to be optimistic. Either machine intelligence is
assumed to be beneficent, or humans remain in a controlling, not
controlled, status. Given the tenuous nature of the first assumption,
the second assumption is the more prudent. Thus automation needs to be
done with care and in a manner that always gives ultimate control to the
human. Otherwise the danger is the prospect of the bleak and terrifying
future portrayed by Joy.

*According to the neo-symbiosis concept, technology should not replace
humans, but instead produce job enrichment and upgrades* \[7\]. A simple
example of this is the job of travel agent. Travel agents have
effectively been automated out of their jobs. Travelers can readily book
their own travel. The travel agents position can be upgraded to a travel
consultant or facilitator and add value to the tasks performed by the
computer. The travel consultant can provide additional information not
readily found on the internet and provide a human face and voice, to say
nothing of reassurance, to the wary traveler. Thus, neo-symbiosis should
not only result in an overall enhancement in system performance, but it
should also result in the job enhancement of the individual user(s).
Neo-symbiosis does not automate humans out of their jobs, but rather
promotes them into a better job. Of course, this cannot be achieved
without some initiative on the part of the human. Often this upgrade is
to supervisor, even if the human is only supervising machines. Typically
job upgrades require training and appropriately designed technology
should include the necessary training to support the job upgrade. Many
will be effectively promoted from labor to management. They will be
managing a staff of software agents that not only perform the tasks
previously done by the user, but also perform additional tasks that
could not be done previously. Others will be relieved of tedious work
and presented with tasks that are more challenging and enriching. There
is the promise of considerable time savings, which should free people
for educational and recreational opportunities. The idea that human
factors and ergonomics should go beyond usability issues is gaining
increasing currency *e.g.,* \[9\], \[21\].

*Automation, to the exclusion of human involvement and control should
always be avoided.* Rather the objective should be to see how technology
can relieve humans from tedious and error prone tasks and leverage the
unique capabilities of the human to enhance productivity. This is a
win/win situation. Productivity is enhanced and jobs are enriched.
Moreover, it can be delivered incrementally as technology evolves and as
we learn more about implementing neo-symbiosis.

[References]{.underline}

\[1\] Anderson, J.R. *The Adaptive Character of Thought*. Hillsdale, NJ:
Lawrence Erlbaum Associates. (1990).

\[2\] Chaiken , S., & Trope, Y. *Dual-process theories in social
psychology.* New York: Guilford Press. (1999).

\[3\] Dawes, R.M. & Corrigan, B.. Linear models in decision making.
*Psychological Bulletin, 81,* 95-106. (1974)

\[4\] Dreyfus, H.L. & Dreyfus, S.E. *Mind over machine: the power of
human intuition and expertise in the era of the computer.* New York: The
Free Press. (1986).

\[5\] Gilovich, T.. *How we know what isn't so: the fallibility of human
reason in everyday life.* New York: The Free Press. (1991)

\[6\] Griffith, D. Neo-symbiosis: a tool for diversity and enrichment.
*CybErg 2005.cyberg.wits.ac.za* (2005).

\[7\] Griffith, D. Beyond usability: the new symbiosis. *Ergonomics in
Design, 13, 3 (2005).* (2005).

\[8\] Hammond, , K.R.. *Human judgment and social policy: irreducible
uncertaintly, inevitable error, unavoidable injustice.* New York:
Oxford. (1996)

\[9\] Hancock, P.A., Pepe, A.A., Murphy, L. . Hedonomics: the power of
positive and pleasurable ergonomics. *Ergonomics in Design, 13,* 8-14.
(2005)

\[10\] Heuer, Jr., R.J. *Psychology of intelligence analysis.* CIA:
Center for the Study of Intelligence Analysis. (1999)

\[11\] Joy, B. Why the future doesn't need us. *Wired, Issue 8.04,
April.* (2000).

\[12\] Kahneman, D.. *Maps of bounded rationality: a perspective on
intuitive judgment and choice.* Nobel Prize lecture, December 8. (2002)

\[13\] Kahneman, D. A perspective on judgment and choice: mapping
bounded rationality. *American Psychologist, 58,* 697-720. (2003).

\[14\] Kahneman, D. & Frederick, S. Representativeness revisited:
Attribute substitution in intuitive judgment. In T. Gilovich, D.
Griffin, & D. Kahneman (Eds.) *Heuristics and biases* (pp.49-81). New
York; Cambridge University Press. (2002).

\[15\] Klein, G.. *Intuition at work.* New York: Doubleday.

Klein, G. (1998). *Sources of power: How people make decisions.*
Cambridge, MA: MIT Press. (2002)

\[16\] Klein, G. *Sources of Power: How people make decisions*.
Cambridge, MA, MA: MIT Press (1998).

\[17\] Kurzweil, R. *The age of spiritual machines: when computers
exceed human intelligence.* New York: Penguin Group. (1999).

\[19\] Licklider, J.C.R. Man-computer symbiosis. *IRE Transactions on
Human Factors in Electronics.*, *HFE ,*4-11. (1960).

\[20\] Myers, D.G.. *Intuition: its powers and perils.* New Haven, CT:
Yale University Press. (2004)

\[21\] Norman, D.A. *Emotional design: Why do we love (or hate) everyday
things.* New York: Basic Books. (2004).

\[22\] Schacter, D.L. *The Seven Sins of Memory: How the Mind Forgets
and Remembers.* Boston: Houghton Mifflin. (2001).

\[22\] Sloman, S.A.. Two systems of reasoning. . In T. Gilovich, D.
Griffin, & D. Kahneman (Eds.) *Heuristics and biases* (pp.379-396). New
York; Cambridge University Press. (2002)

\[23\] Sloman, S.A. The empirical case for two systems of reasoning.
*Psychological Bulletin*, *119,* 3-22. (1996).

\[24\] Stanovich, K.E. *Who is rational: Studies of individual
differences in reasoning.* Mahway, NJ: Erlbaum. (1999).

\[25\] Stanovich, K.E. & West, R.F. Individual differences in reasoning.
Implications for the rationality debate. In T. Gilovich, D. Griffin, &
D. Kahneman (Eds.) *Heuristics and biases* (pp.421-440). New York;
Cambridge University Press. (2002).

[^1]: Model-based Agent-guided Unexpected Inferences for a Novel
    Intelligence Test Environment. This research has been funded in part
    by contract \# MDA-904-C-0406 to Russell Richardson Vane, Ph.D. from
    the Advanced Research and Development Activity, Novel Intelligence
    from Massive Data Program.