A TAXONOMY OF INTELLIGENCE

Comparatively litte work has been done
comparing structured techniques to intuition.
Robert Folker’s work is one of the exceptions; it
compares the effectiveness of a modified form of
alternative competing hypotheses with intuition
in a controlled experimental design. His study is
unique in the field and demonstrates that
experimental methods are possible. Geraldine
Krotow’s research, on the other hand, looks at
differing forms of cognitive feedback during the
analytic process and makes recommendations to
improve intelligence decisionmaking.*?

Idiosyncratic Variables

Variables in the third column are those that
influence individuals and their analytic perfor-
mance. These include the sum of life experiences
and enculturation—familial, cultural, ethnic, reli-
gious, linguistic, and political affiliations—that
identify an individual as a member of a group. I
have used the German word Weltanschauung
(customarily rendered in English as “world view”)
to denote this concept. These idiosyncratic vari-
ables also encompass such psychological factors
as biases, personality profiles, cognitive styles and
processing, cognitive loads,* expertise, approach
to problem-solving, decisionmaking style, and
reaction to stress. Finally, there are such domain
variables as education, training, and the readiness
to apply knowledge, skills, and abilities to the task
at hand.

Idiosyncratic Variables
Weltanschauung (world-
view)

Affiliation

Familial
Cultural
Ethnic
Religious
Social
Linguistic
Political
Psychology
Bias
Personality Profile
Security Trust
Cognitive Processing
Learning Style
Information Acquisition
Information Processing
Expertise
Problem-solving
Decisionmaking
Cognitive Load
Speed/Accuracy
Stress Effects
Education
Domain
Location
Mentor
Training
Organizational
Domain
Procedural
Readiness
Resources
Facilities

The relevant psychological literature is extensive. Amos Tversky and Daniel
Kahneman began to examine psychological biases in the early 1970s.TM Their
work has found its way into the intelligence literature through Butterfield,

» Geraldine Krotow, The Impact of Cognitive Feedback on the Performance of Intelligence Ana-

dysts, 176.

2 “Cognitive loads” are the amount/number of cognitive tasks weighed against available cogni-

tive processing power.

TM Amos Tversky and Daniel Kahneman, “The Belief in the ‘Law of Small Numbers"” and “Judg-

ment Under Uncertainty: Heuristics and Biases.”

41



ORGANIZATIONAL CULTURE

employees.?! The job preview allows candidates to make an informed decision
to continue with the recruitment process or to withdraw from it if they feel the
job is not appropriate. Realistic previews also lower unrealistically high
expectations. A particularly good example of such an effort can be found on
the CIA’s Office of General Counsel Web site. This Web site includes a sec-
tion titled “Misconceptions about working for the CIA,” which tries to dispel
prejudices and biases about employment at the CIA by addressing them in a
straightforward manner.” In addition, the authors explain the benefits of hav-
ing work experiences with the CIA for future employment endeavors in other
areas.

Interview screenings of applicants should be reviewed and improved where
needed. Hiring interviews are not very effective predictors of job perfor-
mance; even so, there are ways to improve their reliability and validity.
Numerous cognitive measurement instruments are available that help predict a
match between an individual’s knowledge, skills, and abilities and specific
behavioral, cognitive, and psychomotor tasks.” In addition, the use of struc-
tured interviewing - posing the same questions to all applicants - is more
effective than unstructured interviewing. Structured interviews allow for con-
sistent comparisons among applicants.** Organizations should also consider
using panel interviews. Differences among individual interviewers may result
in inaccurate judgment of an applicant, but the overall decision of a team of
evaluators may improve reliability?

The use of situational exercises should be included in the recruitment pro-
cess. These exercises usually consist of approximations of specific aspects of
a job. They can be used to evaluate candidates’ job abilities and to provide
candidates with simulated work tasks. The former can facilitate organizational
evaluations of candidates’ performance on a job-related task; the latter may
help candidates to decide whether the job would be a good match.?6

A desirable additional step would be the creation and expansion of aca-
demic degree programs with a focus on intelligence and intelligence analysis.
Further, an enhanced effort to improve public awareness and understanding of
the Intelligence Community through greater community outreach, internships,

21 Glenn M. McEvoy and Wayne F. Cascit
Meta-analysis.”

2 See Central Intelligence Agency, Office of General Counsel Web site.

® The Buros Institute of Mental Measurements tracks and reports the statistical validity and reli-
ability of thousands of measurement instruments.

TM Richard D. Arvey and James E. Campion, “The Employment Interview: A Summary of Recent
Research.”

25 PL. Roth and James E. Campion, “An Analy:
and Pre-Employment Tests.”

2% Wayne F. Cascio, Applied Psychology in Human Resource Management.

“Strategies for Reducing Employee Turnover: A

of the Predictive Power of the Panel Interview

105



DEFINITIONS

Error is reporting the wrong thing.
A failure is something catastrophic, and an error is just a mistake.
Error is about facts; failure is about surprise.

Error is when nobody notices, and failure is when everybody
notices.

Some responses disavowed the existence of intelligence error and failure;
some placed the terms in the broader context of policy and decisionmaking;
some interchanged the two terms at random; some defined the terms accord-
ing to their outcomes or consequences. Despite the variability of the
responses, two trends emerged: novice analysts tended to worry about being
factually inaccurate; senior analysts, managers, and consumers, tended to
worry about being surprised. Often, participants’ responses were not defini-
tions at all but statements meant to represent familiar historical examples:

The attack on Pearl Harbor.

The Chinese sending combat troops into Korea.
The Tet Offensive.

The Soviet invasion of Afghanistan.

The collapse of the Soviet Union.

The Indian nuclear test.

September Eleventh.

The danger of defining by example is that each case is contextually unique
and can be argued ad infinitum, What is important about these examples as a
whole is that they all indicate one central and recurring theme. Specifically, all
these examples signify surprise—in some cases, intelligence surprise; in other
cases, military, civil, and political surprise. Even if the Intelligence Commu-
nity itself was not surprised by one of these events, it was unable to convince
the military, civil, and political consumers of intelligence that these events
might occur; in which case, the failure was one of communication and persua-
sion.

When I began this study, my own definition of error and failure derived
from the psychological and cognitive disciplines. Specifically, I took it that
human error and failure are related to measures of cognitive and psychomotor
accuracy, commission of error being at one end of the accuracy scale and
omission or not performing the correct action being at the other.®

© See Appendix A for a list of literature on error.



APPENDIX

Hede Helfrich, “Human Reliability from a Social-Psychological Perspective,”
International Journal Human-Computer Studies 50 (1999): 193-212.

Erik Hollnagel, “The Phenotype of Erroneous Actions,” International Journal
of Man-Machine Studies 39 (1993): 1-32.

Daniel Kahneman and Amos Tversky, “Prospect Theory: An Analysis of
Decision Under Risk,” Econometrica 47, no. 2 (1979): 263-91.

Duncan Luce and Howard Raiffa, “Utility Theory” in Paul Moser, ed. Ratio-
nality in Action: Contemporary Approaches (New York: Cambridge Univer-
sity Press, 1990).

Judith Orasanu and Lynne Martin, “Errors in Aviation Decision Making: A
Factor in Accidents and Incidents” in Proceedings of the Working Group on
Human Error, Safety, and System Development (Seattle, WA: International
Federation for Information Processing, 1998, 100-107).

James Reason, Human Error (New York: Cambridge University Press, 1990).

John Senders and Neville Moray, Human Error: Cause, Prediction and
Reduction (Hillsdale, NJ: Earlbaum, 1991).

Neville Stanton and Sarah Stevenage, “Learning to Predict Human Error:
Issues of Acceptability, Reliability and Validity,” Ergonomics 41, no. 11
(1998): 1737-56.

Language and Cognition

Terry K. Au, “Chinese and English Counterfactuals: The Sapir-Whorf
Hypothesis Revisited,” Cognition 15 (1983): 155-187.

Alfred Bloom, The Linguistic Shaping of Thought: A Study in the Impact of
Language on Thinking in China and the West (Hillsdale, NJ: Lawrence
Erlbaum Associates, 1981).

Lera Boroditsky, “Does Language Shape Thought? Mandarin and English
Speakers’ Conceptions of Time,” Cognitive Psychology 43 (2001):1-22.

Emest Cassirer, “Le Langage et le Monde des Objets,” Journal de Psycholo-
gie Nomale et Pathologique 30, 1(4) (1933): 18-44,

John Gumperz and Dell Hymes, Directions in Sociolinguistics: The Ethnogra-
phy of Communication (Oxford: B. Blackwell, 1986).

, and Stephen Levinson, Rethinking Linguistic Relativity (Cambridge:
Cambridge University Press, 1996).

Harry Hoijer, “The Relation of Language to Culture” in Alfred Kroeber, ed.
Anthropology Today (Chicago: University of Chicago Press, 1953).

Wilhelm von Humboldt “Uber die Verschiedenheit des Menschlichen Sprach-
baues” in B. Bohr, ed. Gesammelte Schriften (Berlin: Verlag, 1907).

Earl Hunt and Franca Agnoli, “The Whorfian Hypothesis. A Cognitive Psy-
chology Perspective,” Psychological Review 98, no. 3 (1991): 377-89.

136



CHAPTER TWELVE

0.15 and for interactive videodiscs, the effect sizes range from 0.17 to 0.66
depending on the population.’ The effect size for flight simulation is 0.54 and
the effect size for tutorials range from 0.25 to 0.41 depending on the presenta-
tion of the tutorial material.?

Although the effect sizes for instructional technology range from 0.15 to 0.66
standard deviations, they all report favorable findings when compared to conven-
tional _ instruction.
There are many pos-
sible explanations for
the differences in
instructional technol-
ogy effectiveness; it
might be the result of
population —differ-
ences, system differ-
ences, interactivity
or individualization.
From a purely utili-
tarian point of view,
the reason may not
be all that important.
Tf, at the very least,

ga
if
H i using instructional
Z

Instructional Technology Effectiveness

‘AoHN-OAI
Ae MI-NIS

technology forces the
producer to rethink
the content of the
course to match the
delivery system, then
revisiting the pedagogy may be enough to produce the positive effect sizes. What-
ever reason for the changes in effectiveness, the use of instructional technology
saves instructional time, overhead costs, and results in a higher level of achieve-
ment for the students in a variety of domains,

7 The abbreviations in figure one: CBT=Computer Based Training, DI=Distance Instruction,
IVD=Interactive Video Disc, SIM=Simulation. More than 300 research studies were used to
develop these effect sizes, see Chen-Lin Kulik., James Kulik and Barbara Shwalb, “Effectiveness
of Computer-Based Adult Education: A Meta-Analysis”; Chen-Lin Kulik and James Kulik,
“Bffectiveness of Computer-Based Education in Colleges”; Rob Johnston and J, Dexter Fletcher,
A Meta-Analysis of the Effectiveness of Computer-Based Training for Military Instruction.

* Godwin Chu and Wilbur Schramm, Learning from Television; J. Dexter Fletcher Effectiveness
and Cost of Interactive Videodise Instruction in Defense Training and Education; J. Dexter
Fletcher, “Computer-Based Instruction: Costs and Effectiveness.”

°R.T. Hays, J. W. Jacobs, C. Prince and E. Salas, “Flight Simulator Training Effectiveness: A Meta-
Analysis”; Peter Cohen, James Kulik and Chen-Lin Kulik, “Educational Outcomes of Tutoring.”

131



All statements of fact, opinion, or analysis expressed in this book
are those of the authors. They do not necessarily reflect official
positions of the Central Intelligence Agency or any other US gov-
ernment entity, past or present. Nothing in the contents should be
construed as asserting or implying US government endorsement
of the authors’ factual statements and interpretations.

The Center for the Study of Intelligence

The Center for the Study of Intelligence (CSI) was founded in 1974 in
response to Director of Central Intelligence James Schlesinger’s desire to cre-
ate within CIA an organization that could “think through the functions of
intelligence and bring the best intellects available to bear on intelligence prob-
lems.” The Center, comprising professional historians and experienced practi-
tioners, attempts to document lessons learned from past operations, explore
the needs and expectations of intelligence consumers, and stimulate serious
debate on current and future intelligence challenges.

To support these activities, CSI publishes Studies in Intelligence and books
and monographs addressing historical, operational, doctrinal, and theoretical
aspects of the intelligence profession. It also administers the CIA Museum
and maintains the Agency’s Historical Intelligence Collection.

Comments and questions may be addressed to:

Center for the Study of Intelligence
Central Intelligence Agency
Washington, DC 20505

Printed copies of this book are available to requesters outside the
US government from:

Government Printing Office (GPO)
Superintendent of Documents
P.O. Box 391954
Pittsburgh, PA 15250-7954
Phone: (202) 512-1800
E-mail: orders @ gpo.gov

ISBN: 1-929667-13-2



ORGANIZATIONAL CULTURE

realistic expectations helps to ease the accommodation stage of the socializa-
tion process.

Consequences of Culture Mismatch

When I got here, I felt like a rabbit stuck in headlights, Now, I feel
like a deer.

It took me a while to figure out that this place runs more like a
newspaper than a university.

It’s pretty solitary work. I spend all day in my head. I really wasn't
expecting that.

There are several consequences of a cultural mismatch between an
employee and an organization. Among these consequences are culture shock,
low job satisfaction, low employee morale, increased absenteeism, increased
turnover, and increased costs.

Culture Shock. People often have to be confronted with different cultures
before they become conscious of their own culture. In fact, when people are
faced with new cultures, it is not unusual for them to become confused and
disoriented, a phenomenon commonly referred to as culture shock.

Beryl Hesketh and Stephen Bochner, among others, have observed that the
process of adjusting to another culture generally follows a U-shaped curve."
At first, people are optimistic about learning a new culture. This excitement is
followed by frustration and confusion as they struggle to learn the new cul-
ture. After six months or so with the organization, people adjust to their new
cultures, become more accepting of them, and are more satisfied by them. For
those who enter a mismatched culture, the productivity issue is clear: the sev-
eral months required to adjust and accept the new work style results in several
months of even lower productivity than is obtainable with those who fit in
right away.

Job Satisfaction. Job satisfaction is defined by one scholar as “people’s pos-
itive or negative feelings about their jobs.” It is hardly surprising that dissat-
isfied employees may try to find ways of reducing their exposure to their jobs.
This is especially significant when one considers that people spend roughly
one-third of their lives at work.

" Beryl Hesketh and Stephen Bochner, “Technological Change in a Multicultural Context: [mpli-
cations for Training and Career Planning”; Maddy Janssens, “Interculture Interaction: A Burden
on International Managers?”

® Edwin A. Locke, “The Nature and Causes of Job Satisfaction.”

101



CHAPTER FOUR

provides an effective way of showing who is responsible for what, and neither
reflects the impact of the work on the individuals responsible for producing
the reports—particularly the analyst—nor the reliance of the analyst on a vari-
ety of factors beyond his or her control.

In sum, this brief evaluation of the Intelligence Cycle with respect to its
inputs, processes, and outputs shows us that the traditional model:

* assumes the process works the same way for all objectives, regardless of
complexity and cognitive demands;

* does not represent the iterative nature of the process required for meeting
objectives;

* does not identify responsibilities for completing steps and allows for mis-
conceptions in this regard;

* does not accurately represent the impact of resource availability on analysts.

To better understand these limitations and the relationships among elements
in the process, it is necessary to step back and take a longer view of the pro-
cess, using a different method of analysis.

Systemic Analysis

If we think of the phenomenon that is being described by the Intelligence
Cycle as a system and perform a systems analysis, we may be able to derive a
greater understanding of process relationships, a better representation of the
variables affecting the process, and a greater level of detail regarding the pro-
cess itself.

The premise that underlies systems analysis as a basis for understanding
phenomena is that the whole is greater than the sum of its parts. A systems
analysis allows for the inclusion of a variety of influences and for the identifi-
cation of outliers that are obfuscated in other types of analyses but that often
play major roles. A systems analysis is accomplished through the examination
of phenomena as cause-and-effect patterns of behavior. This approach is
called a “closed feedback loop” in systems analysis. It requires a close exami-
nation of relationships and their influences, provides a longer view of these
relationships, and often reveals new insights based on trends rather than on
discrete events.°

The systems model diagrammed below is a visual representation of the pro-
cess. The elements of the Intelligence Cycle are identified in terms of their

9° Fritjof Capra, “Criteria of Systems Thinking”; David L. Kaufman, Jr., Introduction to Systems
Thinking.

50



APPENDIX

Jonathan Lockwood and K. Lockwood, “The Lockwood Analytical Method
for Prediction (LAMP),” Defense Intelligence Journal 3, no. 2 (1994): 47—
74.

Douglas J. MacEachin, et al., The Tradecraft of Analysis: Challenge and
Change in the CIA (Washington, DC: Consortium for the Study of Intelli-
gence, 1994).

Don McDowell, Strategic Intelligence: A Handbook for Practitioners, Man-
agers and Users (Canberra, Australia: Istana Enterprises, 1998).

Marilyn Peterson, Applications in Criminal Analysis (Westport, CT: Green-
wood Press, 1995).

, Bob Morehouse, and Richard Wright, eds. Intelligence 2000: Revis-
ing the Basic Elements (Sacramento CA: LEIU and IALEIA, 2000).

John Pierce, “Some Mathematical Methods for Intelligence Analysis,” Studies
in Intelligence 21 (Summer 1977, declassified): 1-19.

John Prescott and Stephen Miller, eds. Proven Strategies in Competitive Intel-
ligence (New York: John Wiley & Sons, 2001).

Edwin Sapp, “Decision Trees,” Studies in Intelligence 18 (Winter 1974,
declassified): 45-57.

David Schum, The Evidential Foundations of Probabilistic Reasoning (Evan-
ston, IL: Northwestern University Press, 1994).

Jack Zlotnick, “Bayes’ Theorem for Intelligence Analysis,” (1972), in H.
Bradford Westerfield, ed. Inside CIA's Private World: Declassified Articles
from the Agency's Internal Journal (New Haven: Yale University Press,
1995),

Cognitive Processes and Intelligence

Edgar Johnson, Effects of Data Source Reliability on Inference (Alexandria,
VA: Army Research Institute for the Behavioral and Social Sciences, Tech-
nical Paper #251, 1974).

Robert Mathams, “The Intelligence Analyst’s Notebook,” in Douglas Dearth
and R. Thomas Goodden, eds., Strategic Intelligence: Theory and Applica-
tion, 2nd ed. (Washington, DC: Joint Military Intelligence College, 1995).

Avi Shlaim, “Failures in National Intelligence Estimates: The Case of the Yom
Kippur War,” World Politics 28, no. 3 (1976): 348-80.

Tools and Techniques as Cognitive Processes

Bruce D. Berkowitz and Allan E. Goodman, Best Truth: Intelligence in the
Information Age (New Haven: Yale University Press, 2000).

, and Allan E. Goodman, Strategic Intelligence for American National
Security (Princeton: Princeton University Press, 1989).

134



CHAPTER FIVE
Integrating Methodologists into Teams of Experts!

ES

Intelligence analysis, like other complex tasks, demands considerable
expertise. It requires individuals who can recognize patterns in large data sets,
solve complex problems, and make predictions about future behavior or
events. To perform these tasks successfully, analysts must dedicate years to
researching specific topics, processes, and geographic regions.

Paradoxically, it is the specificity of expertise that makes expert forecasts
unreliable. While experts outperform novices and machines in pattern recog-
nition and problem solving, expert predictions of future behavior or events are
seldom as accurate as Bayesian probabilities.” This is due, in part, to cognitive
biases and processing-time constraints and, in part, to the nature of expertise
itself and the process by which one becomes an expert.

Becoming an Expert

Expertise is commitment coupled with creativity. By this, I mean the com-
mitment of time, energy, and resources to a relatively narrow field of study
and the creative energy necessary to generate new knowledge in that field. It
takes a great deal of time and regular exposure to a large number of cases to
become an expert.

"A version of this chapter originally appeared as “Integrating Methodologists into Teams of Sub-
stantive Experts in Studies in Intelligence 47, no. 1 (2003): 57-65.

2 Method for estimating the probability of a given outcome developed by Thomas Bayes (1702-
61), an English mathematician, See Thomas Bayes, “An Essay Toward Solving a Problem In the
Doctrine of Chances.”

61



INTRODUCTION

munity’s performance needs to come from within. Once the determination has
been found and the necessary policy guidelines put in place, it is incumbent
upon the Intelligence Community to find and utilize the internal and external
resources necessary to create a performance improvement infrastructure.

Scope

This project was designed explicitly as an applied research program. In
many respects, it resembles an assessment of organizational needs and a gap
analysis, in that it was intended to identify and describe conditions and vari-
ables that affect intelligence analysis and then to identify needs, specifica-
tions, and requirements for the development of tools, techniques, and
procedures to reduce analytic error. Based on these findings, I was to make
recommendations to improve analytic performance.

In previous human performance-related research conducted in the military,
medical, and astronautic fields, I have found in place—especially in the mili-
tary—a large social science literature, an elaborate training doctrine, and well-
developed quantitative and qualitative research programs. In addition to
research literature and programs, these three disciplines have substantial per-
formance improvement programs. This was not the case with the Intelligence
Community.

This is not to say that an intelligence literature does not exist but rather that
the literature that does exist has been focused to a greater extent on case stud-
ies than on the actual process of intelligence analysis.5 The vast majority of
the available literature is about history, international relations, and political
science. Texts that address analytic methodology do exist, and it is worth not-
ing that there are quantitative studies, such as that by Robert Folker, that com-
pare the effectiveness of different analytic methods for solving a given
analytic problem. Folker’s study demonstrates that objective, quantitative, and
controlled research to determine the effectiveness of analytic methods is pos-
sible.*

The literature that deals with the process of intelligence analysis tends to be
personal and idiosyncratic, reflecting an individualistic approach to problem
solving. This is not surprising. The Intelligence Community is made up of a
variety of disciplines, each with its own analytic methodology. The organiza-
tional assumption has been that, in a multidisciplinary environment, intelli-

5 There are exceptions. See the appendix.
© MSgt. Robert D. Folker, /nelligence Analysis in Theater Joint Intelligence Centers. Folkec's
study contains a methodological flaw in that it does not describe one of the independent variables
intuitive method), leaving the dependent variable (test scores) in doubt.

xvii



CHAPTER NINE

For intelligence analysts, the question may be put as, “What is an analytic
morbidity and mortality ratio?” The process of describing and identifying
morbidity and mortality, or error and failure in analytical terms, is a necessary
step in identifying mechanisms to develop, test, and implement performance
improvement interventions. There was little consensus among the participants
in this study about what comprises failure, or even if failure was possible.
There was greater consensus regarding the nature of analytic error, which was
generally thought to be a consequence of analytic inaccuracy.

Metrics. One could reasonably conclude that compounded errors lead to
analytic failure. Conversely, one could conclude that failure is the result of
analytic surprise, that its causes are different from the causes of error, and that
it needs to be treated as a separate measurement. This subject is open to
debate and will require further research. It is still possible, however, to use
both accuracy and surprise as metrics in evaluating analytic performance on a
case-by-case basis.

The advantage of an error and failure metric is that it is observable in a
grounded state separate from the analytic process. Any analytic product can
be reviewed to determine levels of accuracy, and any unexpected event can be
traced back through analytic products to determine if there was an instance of
surprise.

Once levels of error and failure are calculated, along with measures of out-
put, it is possible to determine expert levels of performance and to derive per-
formance models based on successful processes. In any organization, there
will be those individuals with the greatest output—in this case, the greatest
number of written products. There will also be individuals with the highest
levels of accuracy—in this case, factual consistency. There will also be indi-
viduals who have the lowest incident of surprise—in this case, those who gen-
erate the greatest number of potential scenarios and track and report
probabilities most reliably. Using data-driven metrics means that expertise is
not a function of tenure; rather, it is a function of performance.

Once expert performers are identified, it is possible to capture their work
processes and to develop performance models based on peak efficiency and
effectiveness within the Intelligence Community. Through the use of cogni-
tive, behavioral, and linguistic task analyses, ethnography, and controlled
experiments, it is possible to generate process metrics to identify analytic
methods that are more effective for specific tasks than other methods. This is
not to say that there is one analytic method that is the most effective for intel-

2 Grounded Theory is the development of theoretical constructs that result from performing interpre-
tive analysis on qualitative data rather than relying on a priori insights, The theory is then derived
from some grounded data set. Barney Glaser and Anselm Strauss, Discovery of Grounded Theory;
Barney Glaser, Theoretical Sensitivity; Barney Glaser, Basics of Grounded Theory Analysis.

110



CHAPTER NINE

agents or agencies of change and manage their organization’s culture in the
same way they manage physical and financial resources. An organization's
culture shapes individual behavior by establishing norms and taboos and, ulti-
mately, determines the quality and character of an organization’s products.
Culture and product are inseparable, and one cannot be changed without
affecting the other. The choice confronting any organization is to manage its
institutional culture or to be managed by it.

There is no single path to carrying out the research recommended in this
work. It could be performed at a single center or coordinated through several
specific centers; it could be purely internal to the Intelligence Community; a
cooperative effort among the community, academe, and national laboratories;
or some combination of these. What is most important to effective implemen-
tation is that there be regular and open access among researchers and the Intel-
ligence Community. This may appear simple enough, but access equals trust,
and trust is difficult to establish in any domain. This is especially the case
within the Intelligence Community. The Intelligence Community needs to
increase its commitment to community outreach efforts. This study is one
such effort.

During the course of my research, the value of access and the premium the
community places on trust quickly became evident. At agency after agency,
physical access restrictions, security clearances, forms, interviews, phone
calls, questions, vetting, and more vetting were all signs of the value, not of
secrecy per se, but of trust and access. Without this sort of cooperation, this
research would have been impossible, and this is an important lesson that
ought to inform future research programs.

116



sealed, world of intelligence. Rob’s work also bears witness to the tremendous
passion among intelligence professionals to understand better what they do, why
they do it, and how their work could be improved. There is, in this community, a
palpable desire to do better.

Since the tragedy of 9/11 and the bitter controversies surrounding Iraqi WMD,
the world of intelligence analysis has been scrutinized with an intensity of almost
unprecedented dimensions. The focus of scrutiny, however, has been on the
results, not on the process by which the results were produced and certainly not
on the largely anonymous corps of civil servants whose work was at the heart of
the issue. Those people, and how and why they do what they do, are at the heart
of Rob’s important study. If we are ever to make the improvements that must be
made in the quality of our intelligence work, then we must begin with a more
mature and nuanced understanding of who actually does the work, how, and why.
There is a context within which the work is done, a definite culture with values,
traditions, and procedures that help shape important outcomes.

Rob has characterized a world that I find all too familiar, It is a world in which
rewards and incentives are weighted heavily in favor of filling in gaps in conven-
tional knowledge rather than in leading the way to alternative points of view. It is
a world in which confirming evidence is welcome and rewarded and disconfir-
matory evidence is, at best, unwelcome and, at worst, discounted. It is a world in
which the legitimate and often necessary resort to secrecy has served, all too
often, to limit debate and discussion. It is a world in which the most fundamen-
tally important questions—what if and why not—are too often seen as distrac-
tions and not as invitations to rethink basic premises and assumptions.

Much of the recent discussion concerning the performance of intelligence
organizations has been conducted in almost mechanical terms. “Connecting
the dots,” “mining the nuggets” are phrases offered as a way of understanding
the exquisitely subtle, complex business of making intelligence judgments.
As Rob Johnston’s book makes abundantly clear, this is first and foremost a
human enterprise. All of the intellectual power, biases, and paradigms that
inform the thinking of the people who actually do the work need to be under-
stood in the organizational context in which they do their work. In the finest
tradition of anthropological field research, Rob has observed, collected data
rigorously, reflected with deep insight upon it, and produced a study both
sophisticated and extremely useful.

I worked myself for more than 30 years in the clandestine operations area of
CIA, a part of the Intelligence Community that calls out for the same kind of
understanding, professional, and constructive scrutiny this book has devoted to
the analytic realm. My fervent hope is that the human intelligence service will
benefit from the same kind of rigor and constructive understanding the analytic
side has now experienced. My real hope is that Rob is available for the job.

160



CHAPTER FOUR

Further Study. The use of simulation allows us to determine flaws in the sys-
tem that basic informational models cannot address. A simulation moves the
image of the Intelligence Cycle from a picture that selectively and indiscrimi-
nately illustrates a series of events to a holistic and realistic representation of
events, responsibilities, processes, and their impact on each other. The simula-
tion of the Intelligence Cycle developed for this analysis is merely a first step.
Further work should be done with it to validate the representations, test for vul-
nerabilities, predict outcomes, and accurately recommend changes.

Lightening the Analyst’s Load. The systems model reveals a serious imbal-
ance in the work processes analysts can and cannot control. It is unrealistic
and unnecessary to consider reorganizing the process to correct this defect.
However, there are actions that could be taken to provide analysts more con-
trol over external factors without significantly altering their roles. These
actions would also reduce the amount of potential influence that one group
could have over the entire process.

First, analysts might be designated as reports or research analysts. The
former would prepare products that address short-term tasks, such as writing
for the PDB. As the process of collection and analysis is different for short-
and long-term products, this might be a responsibility assigned primarily to
more junior analysts. Research analysts might be those with more experience.
Freed from the obligation to prepare short-term reports, senior analysts would
be available for more intense research efforts, such as those required for an
NIE. In addition, cross-training or experience in creating both products and
the flexibility to switch from one process to another would provide greater
depth of personnel. If appropriate, movement to a long-term research position
could be viewed as professional development.

Second, personnel responsible for formatting and processing raw data
might be included on accounts. Through association with a particular group,
people in this role would have a reasonable idea of analysts’ requirements.
This would allow the preselection and preparation of data, so that analysts
could focus on “connecting the dots.” The skills requirement for this role
would be akin to those of a research librarian.

Third, tools to help the analyst identify, manage, and fuse relevant data
could be identified and deployed. These tools, which need not be limited to
those that are technology-based, should be used to support analysts’ labor-
intensive tasks, thereby freeing them to focus on the analysis of data.

Employ alternative methods for examining work processes. Just as we used
alternative methods to examine the Iintelligence Cycle, and as managers press
analysts to use alternative analyses in assessing their targets, so should managers
employ alternative methods for examining work processes. These methods

56



CHAPTER TWO

You know, someday somebody is bound to notice that velocity isn’t
a substitute for quality. We’ve gotten rid of the real analytic prod-
ucts that we use to make, and now we just report on current events.

Not all analysts indicated that time constraints and information load had a
negative effect on their performance. A minority indicated that there was suf-
ficient time to perform analytic duties and prepare analytic products.

This is a tactical shop. It’s all we do. Current reporting is our job.

I work a slow desk. I have plenty of time for self-initiated products
—maybe 60 percent or more.

1 multitask pretty well. I don’t really experience a time-crunch.

Maybe I just process better than other people, but I don't really feel
pressed for time. Besides, I’d rather be at a hot desk than at a cold
desk.

Analytic supervisors were more evenly mixed in their opinions about time
constraints. A slight majority of the managers interviewed said time con-
straints had negative effects on the work environment, work processes, and the
morale of their staff. A majority of them also put analytic time constraints in a
larger context of policy making. They indicated that the decision-cycle of pol-
icymakers was 24 hours a day and that their responsibility was to support that
decision cycle with current intelligence.

In discussing their perceptions of consumer demand, the managers’ views
of the nature of those demands were mixed.

I want my analysts to produce long-term products. [ want them
thinking through their subjects. The decision makers want well-
thought-out products, not just daily briefs.

Our customers want current production. They never complain about
the daily products and, frankly, I doubt they have time to read the
longer stuff.

My consumers like the bigger pieces. They like having the context
and broader picture. They don’t want to be spoon fed.

I’ve never had a customer tell me they want more to read.

Our customers want to avoid surprise. As long as we keep them
from being surprised, | don’t care if we do daily or long-term pro-
duction. I don’t think they care either.



COMBATING ETHNOCENTRISM

people in their own setting and on their own terms. Students would be encour-
aged to investigate the rituals, norms, taboos, kinship systems, and social net-
works of the cultures being studied. There would also be provision for
continuing on-line education and an on-line community of practice for men-
toring, problem solving, and peer-to-peer interaction.

In my view, a stand-alone training program would be insufficient to affect
analytic processes without specific follow-on programs. Retention of training
requires repetition, problem solving, application, and evaluation. People must
use what they learn and then determine if what they have learned can improve
the quality of their work. To this end, [ recommend a formal After Action
Review (AAR) process.

The AAR is used by the US Army to capture lessons learned after a training
exercise or a live operation. Unlike conventional postmortems and traditional
performance critiques, the AAR is used to evaluate successes as well as fail-
ures. Although failure generally receives more scrutiny and attention than suc-
cess, an approach that only examines failure results in sampling error. If one
only scrutinizes mistakes, otherwise effective methods may be blamed for the
errors. That those techniques were successful in 99 out of 100 cases can go
unnoticed, with the result that the failures receive disproportionate attention
and bias the statistical results of the postmortem. The AAR was specifically
designed to avoid this problem.

The AAR process was introduced in the mid 1970s, but it is based on the
oral history method of “after combat interviews” employed by S.L.A. Mar-
shall during World War II, the Korean War, and the Vietnam War. As soon as
possible after a battle, regardless of the outcome, Marshall would assemble
soldiers who were involved and, using a semistructured interview technique,
would engage them in a group discussion about their individual and team roles
and actions during combat.

The current AAR method also includes such objective data as tactics, logis-
tics, kill ratios, time-to-task, accuracy-of-task, and operational outcomes. '4
Informed by the objective data, a group discussion led by a facilitator trained
in the elicitation process ensues. The AAR, along with supporting documents,
such as historical studies and relevant doctrinal materials, is then stored in a
knowledge repository at the US Army’s Center for Army Lessons Learned
(CALL).'5

With some customization, an AAR process and a lessons learned repository
could be created for intelligence analysts. Although seemingly time-consuming
and cumbersome, with training and expert facilitators, the AAR process could be
modified and streamlined for use by analysts at the end of a production cycle. As

‘s John Morrison and Larry Meliza, Foundations of the After Action Review Process.



FINDINGS

selves, but in articulating those methods for the purpose of testing and validat-
ing them and then testing their effectiveness throughout the community. In the
long view, developing the science of intelligence analysis is easy; what is dif-
ficult is changing the perception of the analytic practitioners and managers
and, in turn, modifying the culture of tradecraft.

Finding: Confirmation Bias, Norms, and Taboos

Organization is key, because it sets up relationships among people
through allocation and control of resources and rewards, It draws
on tactical power to monopolize or parcel out liens and claims, to
channel action into certain pathways while interdicting the flow of
action into others. Some things become possible and likely; others
are rendered unlikely.

Eric Wolf

Time constraints affect both the general analytic production cycle and ana-
lytic methodology by contributing to and exacerbating cognitive biases.
Although there are any number of cognitive biases to which the human mind
is susceptible, one in particular became evident during the triangulation phase
and interpretive analysis of the interview and observation data of this study.
The cognitive bias identified most often was confirmation bias, which is the
tendency of individuals to select evidence that supports rather than refutes a
given hypothesis. '4

Although the psychological mechanism by which confirmation bias occurs
is in debate, confirmatory behavior is a consistent finding throughout the
experimental psychology and cognitive science literature. Rather than focus
on the mechanism and nomenclature, the term “confirmation bias” is used in
this work as a description of confirmatory behavior. This behavior was

© Eric Wolf was an anthropologist who focused on power, social structures, and the third world.
His work on power and the lives of peasants is considered a modern anthropological classic. Eric
Wolf, Pathways of Power.

1 There is a fair amount of disagreement in the psychological literature regarding the mechanism
by which an individual displays confirmatory behavior. Some researchers attribute it to motiva-
tional factors, for example, a desire to maintain respect within a group, Other researchers attribute
it to selectivity factors, an unconscious cognitive selection of data that confirms the current status
quo. Some researchers attribute it to social factors, a subspecies of groupthink (see Irving Janis,
Groupthink), Still others ascribe it to a misapplication of heuristics, whereby an individual learns
a set of rules that solves one problem and then begins using that same set of rules to try to solve
other types of problems. Although the literature is extensive, Karl Popper's The Logic of Scien-
tific Discovery provides a foundation for understanding the issue. Jonathan Evans’ Bias in Human
Reasoning: Causes and Consequences is still a useful and concise summary of the research
related to confirmation bias.

21



BIBLIOGRAPHY

Hedges, Larry. “Estimation of Effect Size From a Series of Independent
Experiments,” Psychological Bulletin, 7 (1982): 119-37.

, and Ingram Olkin. “Vote-Counting Methods in Research Synthesis,”

Psychological Bulletin 88 (1980): 359-69.

, and Ingram Olkin. Statistical Methods for Meta-Analysis. New York:
Academic Press, 1985.

Helfrich, Hede. “Human Reliability From a Social-Psychological Perspec-
tive,” International Journal Human-Computer Studies 50 (1999): 193-212.

Hesketh, Beryl, and Stephen Bochner. “Technological Change in a Multicul-
tural Context: Implications for Training and Career Planning” in Marvin
Dunnette and Leaetta Hough, eds. Handbook of Industrial and Organiza-
tional Psychology. Palo Alto, CA: Consulting Psychologists Press, 1994.

Heuer, Richards J., Jr. Psychology of Intelligence Analysis. Washington, DC:
Center for the Study of Intelligence, 1999.

Hewitt, John P. Self and Society, 7th ed. Boston: Allyn and Bacon, 1997.

Hinchman, Lewis P., and Sandra K. Hinchman, eds. Memory, Identity, Com-
munity. Albany, NY: State University of New York Press, 1997.

Hoijer, Harry. “The Relation of Language to Culture” in Alfred Kroeber, ed.
Anthropology Today. Chicago: University of Chicago Press, 1953.

Hollnagel, Erik. “The Phenotype of Erroneous Actions.” International Jour-
nal of Man-Machine Studies 39 (1993): 1-32.

Hopkins, R. Warnings of Revolution: A Case Study of El Salvador. Washing-
ton, DC: Center for the Study of Intelligence, 1980, TR 80-100012.

House, Ernest. Evaluating with Validity. Beverly Hills, CA: Sage, 1980.

Hubble, Edwin. The Realm of the Nebulae. New Haven, CT: Yale University
Press, 1936.

Hulnick, Arthur S. “The Intelligence Producer-Policy Consumer Linkage: A
Theoretical Approach,” Intelligence and National Security 1, no. 2 (May
1986): 212-33.

Hunt, David. Complexity and Planning in the 21st Century: Intelligence
Requirements to Unlock the Mystery. Newport, RI: Naval War College,
2000.

Hunt, Earl, and Franca Agnoli. “The Whorfian Hypothesis. A Cognitive Psy-
chology Perspective,” Psychological Review 98, no. 3 (1991): 377-89.

Hymes, Dell, ed. Language in Culture and Society: A Reader in Linguistics
and Anthropology. New York: Harper and Row, 1977.

Iaffaldano, Michelle T., and Paul M. Muchinsky. “Job satisfaction and Job
Performance: A Meta-Analysis,” Psychological Bulletin 97 (1985): 251-73.

James, William. Principles of Psychology: Volume I. New York: Dover Press,
1890/1950.

Janis, Irving. Groupthink, 2nd ed. Boston: Houghton Mifflin, 1982.

Janssens, Maddy. “Interculture Interaction: A Burden on International Manag-
ers?” Journal of Organizational Behavior 16 (1995): 155-67.

146,



CHAPTER FOUR

the entire effort and involves, in particular, determining collection requirements
based on customer requests. Collection refers to the gathering of raw data to
meet the collection requirements. These data can be derived from any number
and type of open and secret sources. Processing refers to the conversion of raw
data into a format analysts can use. Analysis and production describes the pro-
cess of evaluating data for reliability, validity, and relevance; integrating and
analyzing it; and converting the product of this effort into a meaningful whole,
which includes assessments of events and implications of the information col-
lected. Finally, the product is disseminated to its intended audience.*
In some ways, this
The Traditional Intelligence Cycle process resembles many
other production cycles. It
Planning is prescriptive, structured,

and direction made up of discrete steps,
and expected to yield a
specific product. The
traditional depiction of the

‘ Dissemination : P
Collection " process in the Intelligence

Cycle, however, is not an
accurate representation of
the way intelligence is
produced. The notion of a

Processing * Analysis cycle assumes that the steps
and production . -

will proceed in the

prescribed order and that

the process will repeat itself continuously with reliable results. This type of

representation gives the impression that all inputs are constant and flow

automatically, but it does not address elements that may influence the
movement of the cycle, positively or negatively.

The most significant assumption about the Intelligence Cycle model, that it
provides a means for helping managers and analysts deliver a reliable product,
should be examined at the outset. This can be accomplished through two types
of analyses. The first is a systematic examination of the elements of the pro-
cess, the inputs it relies on, and the outcomes that can be expected. The second
uses a systemic approach to identifying the relationships of the elements in the
process and their influence on each other.

+ Central Intelligence Agency, A Consumer's Guide to Intelligence.
4 Central Intelligence Agency, Factbook on Intelligence.



CHAPTER TWELVE

Effect size is usually defined as the difference between the means of two
groups divided by the standard deviation of the control group XX 2

7

Effect sizes calculated in this way estimate the difference between two
group means measured in control group standard deviations as seen in the fig-
ure above. Glass et al. suggest that the choice of the denominator is critical
and that choices other than the control group standard deviation are defensi-
ble.3 However, they endorse the standard choice of using the control group
standard deviation.

Alternatively, Hedges and Olkin show that, for every effect size, both the
bias and variance of its estimate are smaller when standard deviation is
obtained by pooling the sample variance of two groups instead of using the
control group standard deviation by itself.* An effect size based on a pooled
standard deviation estimates the difference between two group means mea-
sured in standard deviations estimated for the full population from which both

Xe - Xe

experimental and control groups are drawn: | g = a 5 where

S is the pooled standard deviation:

(e-1)(Se)?#(Ne-1)(Se)? §
Nes Ne-2 :

Most commentators suggest that effect sizes can be treated as descriptive
statistics and entered into standard tests for statistical significance. Hedges
and Olkin have shown that the error variance around estimates of effect size is
inversely proportional to the sample size of the studies from which the effect
sizes are drawn. If the effect size in any review is drawn from studies employ-
ing widely different sample sizes, then the heterogeneity of variance among
effect sizes prohibits their use in conventional t-tests, analyses of variance,
and other inferential tests. This is the case in most of these reviews; therefore,
effect sizes reported in this study are treated only with descriptive statistics.

The effect sizes for computer-based training range from 0.20 to 0.46 depend-
ing on the population.’ The effect size for distance instruction (television) is

2 A =Glass's Effect Size, X, Experimental Mean, X,, — Control Mean, Ge = Control
Standard Deviation

+ Gene Glass, Barry McGaw and Mary Lee Smith, Meta-Analysis in Social Research

4 Larry Hedges and Ingram Olkin, Statistical Methods for Meta-Analysis,

5 g=Hedge's Effect Size, S=Hedge's Pooled Standard Deviation

® Ne=Number of experimental subjects, Nc=Number of control subjects, Se=Standard deviation
of experimental group, Sc=Standard deviation of control group

130



FINDINGS

production cycle, information load, a shift in product focus, and organiza-
tional norms regarding promotion have had an impact on analytic work and
intelligence analysis itself.

Finding: “Tradecraft” Versus Scientific Methodology

Human beings do not live in the objective world alone, nor alone in
the world of social activity as ordinarily understood, but are very
much at the mercy of the particular language which has become the
medium of expression for their society...The fact of the matter is that
the “real world” is to a large extent unconsciously built upon the
language habits of the group...We see and hear and otherwise expe-
rience very largely as we do because the language habits of our
community predispose certain choices of interpretation.

Edward Sapir®

The Intelligence Community, in its culture and mythos and in its literature,
tends to focus on intelligence operations rather than on intelligence analysis.
Open literature about the community certainly does so. Along with time con-
straints and the analytic production cycle, the private and public focus on
operations has had an effect on intelligence analysts and analytic methodol-
ogy. The principal effect is the spread of the concept of “tradecraft” within the
analytic community.

Community members quite often used the word “tradecraft” to describe
intelligence analysis during the interviews, observations, training programs,
workshops, and actual analytic tasks that I performed for this study. Analysts,
managers, instructors, and academic researchers employed the word “trade-
craft” as a catchall for the often-idiosyncratic methods and techniques
required to perform analysis. Although the intelligence literature often refers
to tradecraft, the works tend to be a collection of suggestions and tips for writ-
ing and communicating with co-workers, supervisors, and consumers instead
of focusing on a thorough examination of the analytic process and techniques.

The notion that intelligence operations involve tradecraft, which I define as
practiced skill in a trade or art, may be appropriate, but the analytic commu-
nity’s adoption of the concept to describe analysis and analytic methods is not.
The obvious logical flaw with adopting the idea of tradecraft as a standard of
practice for analytic methodology is that, ultimately, analysis is neither craft
nor art. Analysis, I contend, is part of a scientific process. This is an important

* Edward Sapir is best known for the Sapir-Whorf hypothesis, which asserts linguistic/cognitive
relativity (language and thought are inseparable; therefore, different languages mean different
ways of thinking). Edward Sapir, Language.



TESTING THE INTELLIGENCE CYCLE

an especially important point. Assuming that the actions identified in the
“Processes” column are ultimately the responsibility of the intelligence ana-
lyst, the steps of the process move from a heavy reliance on information com-
ing in from sources outside the analyst’s control to a heavy reliance on the
analyst to produce and manage the final submission of the product.

Another important defect in this analysis is that steps in the cycle do not
accurately represent the differences in the cognitive complexity involved in
preparing a long-range assessment or a national intelligence estimate and that
required for a two-paragraph brief on a current situation. The same can be said
about the process required to develop each of the products.

The Intelligence
Cycle depicts =a
sequential process and
does not provide for ‘Tasting and
iterations between Soon
steps. This is not an
accurate reflection of
what happens, particu- Tmaaene har
larly in the collection infers needs Infeligence
and production steps,
where the challenges
of defining  policy-
maker needs and shap-
ing collection
necessitate Tepeated Poy cies Proceeng.
refinement of require- ane reacts Analysis:
ments by policymakers
or of inferences by the
Intelligence Commu-
nity. A more accurate picture of the steps in the process and their iterative ten-
dencies may be seen in Greg Treverton’s model, which he terms the “Real”
Intelligence Cycle (above).”

Mark Lowenthal proposes another model.* Although presented in a more
linear fashion than Treverton’s, it focuses on the areas where revisions and
reconsiderations take place, representing iteration in a slightly different light.
Both models provide a more realistic view of the entire process. In addition,
assuming that the analyst’s role is represented by the “Processing, Analysis”
box, the Treverton model allows us to focus visually and conceptually on the
demands that the process can place on the analyst. However, neither model

Treverton’s “Real” Intelligence Cycle

7 Gregory F. Treverton, Reshaping National Intelligence in an Age of Information.
® Mark W. Lowenthal, intelligence: From Secrets to Policy.

49



CHAPTER SIX

a practical matter, the process would be used mostly with longer works, such as
assessments or estimates. The intelligence product, along with AAR notes, would
then be incorporated in a community knowledge repository. This knowledge
repository would also help in the development and refinement of advanced ana-
lytic courses by providing course developers with baseline analytic data. In short,
the repository becomes a tool for continuous educational needs analysis and links
training directly to the actual work practices of analysts. These data can be used
as a test bed for research on the effectiveness of analytic methodology. In this
way, the lessons learned are not lost to future generations of analysts.

'S See the US Army Center for Army Lessons Learned Web site, which has links to numerous
other repositories. Although each organization has customized the concept to meet its unique
needs, all of the US military services, the National Aeronautics and Space Administration, the
Department of Energy, the Environmental Protection Agency, the North Atlantic Treaty Organi-
zation, the United Nations, and the ministries of defense of Australia and Canada, currently have
Lessons Learned repositories.

86



BIBLIOGRAPHY

, and Barbara Shwalb, “Effectiveness of Computer-Based Adult Edu-
cation: A Meta-Analysis,” Journal of Educational Computing Research 2,
no. 2 (1986): 235-52.

Kulik, James. “Meta-Analytic Studies of Findings on Computer-Based
Instruction” in Eva Baker and Harold O'Neil, eds. Technology Assessment in
Education and Training. Hillsdale, NJ: LEA Publishers, 1994.

Lacan, Jacques. Ecrits. Translated by Alan Sheridan. New York: W.W.
Norton, 1977.

Laqueur, Walter A. The Uses and Limits of Intelligence. New Brunswick, NJ:
Transaction Publishers, 1993.

Larkin, Jill. “The Role of Problem Representation in Physics” in Debra Gent-
ner and Albert Stevens, eds. Mental Models. Hillsdale, NJ: Lawrence
Erlbaum Associates, 1983.

Lee, Penny. The Whorf Theory Complex: A Critical Reconstruction. Amster-
dam: Benjamins, 1996.

Leli, D., and S. Filskov. “Clinical-Actuarial Detection of and Description of
Brain Impairment with the Wechsler-Bellevue Form I,” Journal of Clinical
Psychology 37 (1981): 623-29.

Lesgold, Alan, et al. “Expertise in a Complex Skill: Diagnosing X-Ray Pic-
tures” in Michelene Chi, Robert Glaser and Marshall Farr, eds. The Nature of
Expertise. Hillsdale, NJ: Lawrence Erlbaum Associates, 1988.

Levi-Strauss, Claude. Structural Anthropology. New York: The Penguin Press,
1968. (Originally published in 1958.)

Levine, George, ed. Constructions of the Self. Brunswick, NJ: Rutgers Univer-
sity Press, 1992.

Levinson, David, and Melvin Ember, eds. American Immigrant Cultures:
Builders of a Nation. Vol. I and II. New York: Macmillan Reference, 1997.

Litwin, Mark. “How to Measure Survey Reliability and Validity” in Arlene
Fink, ed. The Survey Kit. Volume 7. Thousand Oaks, CA: Sage, 1995.

Liu, Lisa. “Reasoning Counterfactually in Chinese: Are There Any Obsta-
cles?” Cognition 21 (1985): 239-70.

Locke, Edwin A. “The Nature and Causes of Job Satisfaction” in Marvin Dun-
nette, ed. Handbook of Industrial and Organizational Psychology. Chicago:
Rand McNally, 1976.

Lockwood, Jonathan, and K. Lockwood. “The Lockwood Analytical Method
for Prediction (LAMP),” Defense Intelligence Journal 3, no. 2 (1994): 47—
74.

Lowenthal, Mark. Intelligence: From Secrets to Policy. Washington, DC: CQ,
2000.

Luce, Duncan, and Howard Raiffa. “Utility Theory.” Paul Moser, ed. Ratio-
nality in Action. Contemporary Approaches. New York. Cambridge Univer-
sity Press, 1990.

149



CHAPTER NINE

As the resources available to intelligence analysis are limited, it needs to be
determined if those resources are better spent on the reporting functions of the
Intelligence Community or on warning functions. It also needs to be deter-
mined whether these functions should be performed by the same analysts or if
they are two separate career tracks. To make this determination, the Intelli-
gence Community will need to invest in what I call a Performance Improve-
ment Infrastructure as well as basic and applied analytic research.

Performance Improvement Infrastructure

The first step in improving job or task-specific performance is the establish-
ment of a formal infrastructure designed explicitly to create an iterative per-
formance improvement process. Such a process would include:

* measuring actual analytic performance to create baseline data;
* determining ideal analytic performance and standards;

* comparing actual performance with ideal performance;

* identifying performance gaps;

* creating interventions to improve analytic performance;

* measuring actual analytic performance to evaluate the effectiveness of
interventions.

Several organizational, or infrastructure, assets should be developed to sup-
port this process. These should include:

* basic and applied research programs;

* knowledge repositories;

* communities of practice;

¢ development of performance improvement techniques.

The performance improvement process would be repeated throughout the
life cycle of an organization in order to encourage continuous improvement.
With the infrastructure and process in place, an organization would be capable
of adapting to new or changing environmental conditions.

Infrastructure Requirements

Institutional changes, such as corporate reorganizations, are often enacted
without a clear understanding of their potential or actual impact. What is most
often missing in such changes is a basic research plan or a systems approach

108



FINDINGS

The shift in focus to daily analytic products, the changes in the production
cycle, and a heterogeneously defined professional discipline have had an addi-
tional effect on the professional identity of analysts within the Intelligence
Community. Analysts often commented that they perceived their job and their
daily work routine as more akin to reporting than to analysis.

Basically, on a day-to-day basis, it’s like working at CNN, only
we're CNN with secrets. Actually, it’s more like CNN's Headline
News.

Imagine USA Today with spies—bullet points, short paragraphs,
the occasional picture. You know, short and simple.

I think of myself as a writer for the most important newspaper in the
world.

Many analysts expressed dissatisfaction with the shift in work processes
from long-term forecasts and toward current reporting and the subsequent
shift in their own professional identity within the Intelligence Community.
The current sentiment about identity was often contrasted against an idealized
past that was described as being freer of current production practices and
products.

About 15 years ago, I would have described myself as a scholar.
Now, I’m a reporter. I’ve got 15 people trying to change my work
into bullet points. Presumably, nobody has time to read anymore.

When I joined, it seemed that the word “analyst” was shorthand for
“problem solver.” Now, it’s shorthand for “reporter.”

I'm proud of where I work. I’m proud of the job that we do. But, it is
hard to take pride in one paragraph. I have to look at the big pic-
ture, or I would get discouraged.

I spend most of my waking hours doing this, but I still can’t really
say what an analyst is.

I'm not a reporter, and I’m not an academic. I'm somewhere in
between.

The heterogeneous descriptions and definitions of intelligence analysis as a
professional discipline were consistent findings during this study, indicating
that there needs to be a clear articulation and dissemination of the identity and
epistemology of intelligence analysis. A clearly defined professional identity
would help to promote group cohesion, establish interagency ties and relation-
ships, and reduce intra- and interagency communication barriers by establish-
ing a professional class throughout the Intelligence Community. At an
individual level, a clearly defined professional identity helps to reduce job dis-

27



INTEGRATING METHODOLOGISTS

cal infrastructure, chemical supplies, and research and development (the same
data that the economist analyzed in the previous example). The chemist recog-
nizes that these are the ingredients necessary for a nation to produce a specific
chemical agent, which could have a military application or could be benign.
The chemist then meshes the data with an existing pattern, stores the data as a
new pattern, or ignores the data as an anomaly.

The chemist, however, has no frame of reference regarding spending trends
in the country of interest. He does not know if the investment in chemical sup-
plies represents an increase, a decrease, or a static spending pattern—answers
the economist could supply immediately. There is no reason for the chemist to
know if a country’s ability to produce this chemical agent is a new phenome-
non. Perhaps the country in question has been producing the chemical agent
for years, and these data are part of some normal pattern of behavior.

If this analytic exercise is to begin to coalesce, neither expert must treat the
data set as an anomaly and both must report it as significant. In addition, each
expert’s analysis of the data—an increase in spending and the identification of
a specific chemical agent—must be brought together at some point. The prob-
lem is, at what point? Presumably, someone will get both of these reports
somewhere along the intelligence chain. Of course, the individual who gets
these reports will be subject to the same three complicating variables
described earlier—time context, pattern bias, and heuristic bias—and may not
be able to synthesize the information. Thus, the burden of putting the pieces
together will merely have been shifted to someone else in the organization.

In order to avoid shifting the problem from one expert to another, an actual
collaborative team could be built. Why not explicitly put the economist and
the chemist together to work on analyzing data? The utilitarian problems with
this strategy are obvious: not all economic problems are chemical, and not all
chemical problems are economic. Each expert would waste an inordinate
amount of time. Perhaps one case in 100 would be applicable to both experts,
but, during the rest of the day, they would drift back to their individual
domains, in part, because that is what they are best at and, in part, just to stay
busy.

Closer to the real world, the same example may also have social, political,
historical, and cultural aspects. Despite an increase in spending on a specific
chemical agent, the country in question may not be inclined to use it in a
threatening way. For example, there may be social data unavailable to the
economist or the chemist indicating that the chemical agent will be used for a
benign purpose. In order for collaboration to work, each team would have to
have experts from many domains working together on the same data set.

Successful teams have very specific organizational and structural require-
ments. An effective team requires discrete and clearly stated goals that are

69





ANALYTIC CULTURE IN THE
US INTELLIGENCE COMMUNITY

AN ETHNOGRAPHIC STUDY

SEE

Dr. ROB JOHNSTON

Center for the Study of Intelligence

Lv
<

Central Intelligence Agency
Washington, DC
2005





CHAPTER TWO:

instead of academic and contradictory, and that message is transmitted to the
analysts. In response to the organizational norm, the analyst is inclined to
work the product line rather than change it.

Our products are company products, not individual products. When
you publish something here, it’s the official voice. It’s important for
us to speak with one voice.

It doesn't do us any good if people think we can't make up our mind.

Access matters; if people think you don’t know what you're talking
about, then they stop seeing you.

We already briefed one thing. I can’t go in there and change it now.
We'll look like idiots.

When I was new, I wrote a piece that disagreed with our line. Let's
just say, I’m more careful about that now.

Another organizational norm that contributes to confirmation bias in the
Intelligence Community is the selection and weighing of data according to
classification. Secrets carry the imprimatur of the organization and, in turn,
have more face validity than information collected through open sources. '

Most analysts indicated that they considered “secret” data collected by
covert means to be more important or meaningful than “open” or unclassified
data. Analysts said that they rely on open sources to help fill in missing pieces
of their mental models but that they test the model’s validity with secret infor-
mation. Choosing to rely on classified data as more meaningful to problem
solving and as a tool for testing the validity of their hypotheses serves to exac-
erbate the confirmation bias.

I’m an all-source analyst, so I use whatever I can get my hands on;
but, if the traffic comes from operations, I tend to pay more attention
to it than to information in the open literature.

There is something special about the word “secret” in my business.
It says that it must be important because people had to go and get it
rather than its just showing up in the news. We tend to weigh classi-
fied material as more important than other sources.

‘© In research methodology, face validity is the concept that a measurement instrument appears or
seems to measure what it is actually intended to measure and requires no theoretical supporting
material. In contrast, content validity depends on the content of the domain and established theo-
ries to determine its measures of validity. See David Brinberg and Joseph McGrath, Validity and
the Research Process, Edward Carmines and Richard Zeller, Reliability and Validity Assessment,
Jerome Kirk and Mare Miller, Reliability and Validity in Qualitative Research; Mark Litwin,
“How to measure survey reliability and validity”; William Trochim, The Research Methods
Knowledge Base.

24



BIBLIOGRAPHY

Endsley, M.R. “Design and Evaluation for Situation Awareness Enhance-
ment,” Proceedings of Human Factors Society 32nd Annual Meeting, Santa
Monica, CA: Human Factors Society, 1988.

Ervin, Alexander. Applied Anthropology. Tools and Perspectives for Contem-
porary Practice. Boston: Allyn and Bacon, 2000.

Evans, Jonathan St. B. T. Bias in Human Reasoning: Causes and Conse-
quences. Hove, UK: Lawrence Erlbaum Associates, 1989.

Fandt, Patricia, W. Richardson, and H. Conner. “The Impact of Goal Setting
on Team Simulation Experience,” Simulation and Gaming 21, no. 4 (1990):
411-22.

Feder, Stanley. “FACTIONS and Policon: New Ways to Analyze Politics,”
(1987), in H. Bradford Westerfield, ed. Inside CIA’s Private World: Declas-
sified Articles from the Agency’s Internal Journal. New Haven, CT: Yale
University Press, 1995.

Feldman, Daniel C. “The Multiple Socialization of Organization Members.”
Academy of Management Review 6, no. 2 (1981): 309-18.

Feldman, Tine, and Susan Assaf. Social Capital: Conceptual Frameworks and
Empirical Evidence. An Annotated Bibliography. Social Capital Initiative
Working Paper No. 5. Washington, DC: The World Bank, 1999.

Festinger, Leon. “Informal Social Communication” in Dorwin Cartwright and
Alvin Zander, eds. Group Dynamics: Research and Theory. New York:
Harper & Row, 1960.

Fishburn, Peter. Utility Theory for Decision Making. New York. John Wiley,
1970.

Fitts, Paul, and Michael Posner. Human Performance. Belmont, CA: Brooks
Cole, 1967.

Flavell, J.H. “Metacognitive Aspects of Problem Solving” in L. Resnick, ed.
The Nature of Intelligence. Hillsdale, NJ: Lawrence Erlbaum Associates,
1976.

Fleisher, Craig, and Babette Bensoussan. Strategic and Competitive Analysis.
Methods and Techniques for Analyzing Business Competition. Upper Saddle
River, NJ: Pearson Education, 2003.

Fleishman, Edwin, and Marilyn Quaintance. Taxonomies of Human Perfor-
mance. New York: Academic Press, 1984.

Fletcher, J. Dexter. “Computer-Based Instruction: Costs and Effectiveness” in
Andrew Sage, ed. Concise Encyclopedia of Information Processing in Sys-
tems and Organizations. Elmsford, NY: Pergamon Press, 1990.

. Effectiveness and Cost of Interactive Videodisc Instruction in Defense

Training and Education. Alexandria, VA: Institute for Defense Analyses,

1990, IDA P-2372.

. “Evidence for Learning From Technology-Assisted Instruction” in

Harold F. O'Neil, Jr. and Ray Perez, eds. Technology Applications in Educa-

tion: A Learning View. Hillsdale, NJ: Lawrence Erlbaum Associates, 2003.

143



TESTING THE INTELLIGENCE CYCLE

the organization; the amount of relevant, usable data and actions related to
transforming collected data to relevant, usable data; and the level of demand
on the analyst. Of these five factors, only one—the analyst’s capabilities—is
an internal factor and somewhat under the analyst’s control.'' Yet, even
though the other factors are out of the analyst’s control, the analyst must rely
on them to accomplish the goal and to meet the expectations of customers and
the organization. When the proportion of external factors to internal factors is
as unbalanced as the systems model of the Intelligence Cycle demonstrates,
the causes of stress in the analytic environment increase, as does the possibil-
ity that stress will occur.

In such a high stress environment, where the critical person is responsible
for delivering a product whose development relies on a great number of fac-
tors beyond his or her control, there is greater risk of error, with an increased
likelihood of incomplete or incorrect products. Tendencies to use shortcuts, to
avoid creative thinking, and to minimize the perceived impact of certain
events or actions become more apparent in this situation, especially if their
implementation means reducing the workload and the stressors. Results of
working in such an environment can include increased personnel turnover,
missed or undervalued information, lack of attention to detail, decreased moti-
vation, and a lack of creativity in approaching analysis. Moreover, with ana-
lysts so central to the process, their actions may have a widespread and, thus,
powerful influence on the entire system. This change can be positive or nega-
tive. Given the number of elements influencing the analyst that are out of his
or her control, however, it is unlikely that the changes would positively affect
the quality, accuracy, and number of intelligence products created.

Recommendations

Revisit the traditional intelligence model. The traditional Intelligence Cycle
model should either be redesigned to depict accurately the intended goal, or
care should be taken to discuss explicitly its limitations whenever it is used.
Teaching with an inaccurate aid merely leads to misconceptions that can result
in poor performance, confusion, and a need for unlearning and reteaching. If
the objective is to capture the entire intelligence process, from the request for
a product to its delivery, including the roles and responsibilities of Intelligence
Community members, then something more is required. This should be a
model that pays particular attention to representing accurately all the elements
of the process and the factors that influence them.

1" Byen the factors that contribute to the analyst’s capabilities, notably experience and training, may
be seen to be under the control of others when access to, and selection of, them are considered.

55



CHAPTER FIVE

the same objects according to their function or utility. Experts comprehend the
meaning of data better than novices, and they weigh variables with different
criteria within their domains better. Experts recognize variables that have the
largest influence on a particular problem and focus their attention on those
variables.

Experts have better domain-specific short-term and long-term memory than
do novices.° Moreover, experts perform tasks in their domains faster than nov-
ices and commit fewer errors while solving problems.'° Interestingly, experts
also go about solving problems differently. At the beginning of a task, experts
spend more time thinking about a problem than do novices, who immediately
seek to find a solution.'' Experts use their knowledge of previous cases as con-
text for creating mental models to solve given problems. '*

Because they are better at self-monitoring than novices, experts are more
aware of instances where they have committed errors or failed to understand a
problem.'3 They check their solutions more often and recognize when they
are missing information necessary for solving a problem.'* Experts are aware
of the limits of their knowledge and apply their domain’s heuristics to solve
problems that fall outside of their experience base.

The Paradox of Expertise

The strengths of expertise can also be weaknesses.!5 Although one would
expect experts to be good forecasters, they are not particularly good at it.
Researchers have been testing the ability of experts to make forecasts since
the 1930s.'* The performance of experts has been tested against Bayesian
probabilities to determine if they are better at making predictions than simple
statistical models. Seventy years later, after more than 200 hundred experi-
ments in different domains, it is clear that the answer is no.'? Supplied with an
equal amount of data about a particular case, Bayesian probability data are as

$M. Chi, P. Feltovich, and R. Glaser, “Categorization and Representation of Physics Problems by
Experts and Novices”; M. Weiser and J. Shertz, “Programming Problem Representation in Nov-
ice and Expert Programmers.”

9 W. Chase and K. Ericsson, “Skill and Working Memory.”

'© W. Chase, “Spatial Representations of Taxi Drivers.”

"J, Paige and H. Simon, “Cognition Processes in Solving Algebra Word Problems.”

"2 Voss and Post.

8M. Chi, R. Glaser, and E. Rees, “Expertise in Problem Solving”; D. Simon and H. Simon,
“Individual Differences in Solving Physics Problems.”

¥ J, Larkin, “The Role of Problem Representation in Physics.”

15.C, Camerer and E. Johnson, “The Process-Performance Paradox in Expert Judgment.”

6 H, Reichenbach, Experience and Prediction; T. Sarbin, “A Contribution to the Study of Actuar-
ial and Individual Methods of Prediction.”



ANALYTIC CULTURE ww THE
U.S. INTELLIGENCE COMMUNITY

An Ethnographic Study

boo i
e ‘uP onarerts
:

Am \Y
Dr. Ros JOHNSTON



CHAPTER SEVEN

age about 30 percent, as seen in the table below. The reduction in overhead
expenses averages 20-30 percent.'° Research has shown that the cost ratios
(calculated as the ratio of experimental intervention costs over the costs of a
control group) for interactive multimedia technology (computer-based instruc-
tion with enhanced audio, graphics, and/or video; CD-ROM and DVD-based
instruction; interactive video, etc.) favor it over conventional instruction along
with time savings of about 31 percent.!! Simulation of such systems as heli-
copters, tanks, and command-control systems for training combat skills has
also proven to be cost-effective.!? The operational costs for simulation are, on
average, 10 percent of the costs of using the actual systems to train.

Time Savings for Technology-Based Instruction

Study (Reference) Number of Studies Average Time Saved
Reviewed (Percent)
Military Training - 13 34
Orlansky
Higher Education - 8 31
Fletcher
Higher Education - 17 34
Kulik
Adult Education - Kulik 15 24

Meta-analysis Demonstrates the Effectiveness of Instructional
Technology

Researchers often use a meta-analytic approach to review and synthesize
quantitative research studies on a variety of issues, including instructional
effectiveness. '* This method involves a three-step process, which begins with
the collection of studies relevant to the issue using clearly defined procedures
that can be replicated. Next, a quantitative measure, “effect size,” is used to
tabulate the outcomes of all the collected studies, including those with results
that are not statistically significant. Finally, statistical procedures are used to

1 Jesse Orlansky and Joseph String, Cost-Effectiveness of Computer-Based Instruction in Mili-
tary Training; H. Solomon, Economic Issues in Cost-Effectiveness Analyses of Military Skill
Training; James Kulik, “Meta-Analytic Studies of Findings on Computer-Based Instruction”; Rob
Johnston, “The Effectiveness of Instructional Technology”; Ruth Phelps et al., “Effectiveness and
Costs of Distance Education Using Computer-Mediated Communication”; J. D, Fletcher Effec-
tiveness and Cost of Interactive Videodisc Instruction in Defense Training and Education.

1 J.D. Fletcher, “Computer-Based Instruction: Costs and Effectiveness.”

Jesse Orlanksy et al., The Cost and Effectiveness of the Multi-Service Distributed Training Test-
bed (MDT2) for Training Close Air Support.

'8 Jesse Orlansky et al., The Value of Simulation for Training.

' Gene Glass, “Primary, Secondary, and Meta-Analysis of Research.”

90



RECOMMENDATIONS

Intelligence analysis, like other fields of science, is a cognitive process.
Although tools and technologies may be available to assist cognitive pro-
cesses, such as measurement devices for physical scientists, technology is ulti-
mately merely a tool to be designed and developed using a human-centered
approach. As such, any new technology needs to be a passive tool, employed
by analysts to solve specific problems or answer specific questions, rather
than a restrictive reinterpretation of cognition according to the rules of binary
computation and artificial intelligence theorists.®

Analytic Psychology and Cognition. As evidenced by the work of Richards
Heuer and others, there is significant research to be conducted into the cogni-
tive mechanisms involved in intelligence analysis.” Understanding and defin-
ing the heuristics used in performing intelligence analysis, as well as
cognitive-load thresholds, multitasking requirements, mechanisms that gener-
ate cognitive biases, and the utilization of pattern recognition strategies and
anomaly detection methods are all areas that will prove fundamental to
improving analytic performance.

In addition to researching basic cognitive functions and intelligence analy-
sis, this area of research will be valuable for understanding how external vari-
ables, such as time constraints and analytic production methods, affect the
cognitive processing of individual analysts. Another result will be the devel-
opment of future employee screening and selection tools that will match the
specific cognitive requirements of intelligence analysis with each applicant’s
individual knowledge, skills, and abilities.

Analysts employ cognitive strategies that are time efficient in order to cope
with the demands of producing daily written products, but such strategies are
not necessarily the most effective analytic methods for increasing analytic
accuracy and decreasing the occurrence of analytic surprise. In fact, improv-
ing analytic accuracy and avoiding surprise may require mutually exclusive
analytic strategies. This line of inquiry will require baseline performance data
generated through the development of performance metrics and conducted in
conjunction with research in analytic methodology effectiveness. The results
would then be integrated into a knowledge repository.

These types of studies will require experimental psychologists and cogni-
tive scientists working in controlled laboratory environments with consistent
access to working professional analysts.

© See Chapter Five for a more detailed description of the limitations of technological solutions.
7 Richards J. Heuer, Jr., Psychology of Intelligence Analysis; William Brei, Getting Intelligence
Right; Isaac Ben-Israel, “Philosophy and Methodology of Intelligence: The Logic of Estimate
Process”; Klaus Knorr, Foreign Intelligence and the Social Sciences; Abraham Ben-Zvi, “The
Study of Surprise Attacks.” See also Marjorie Cline, Carla Christiansen and Judith Fontaine,
Scholar's Guide to Intelligence Literature.

3



BIBLIOGRAPHY

Camerer, Colin, and Eric Johnson. “The Process-Performance Paradox in
Expert Judgment: How Can Experts Know so Much and Predict so Badly?”
in K. Anders Ericsson and Jacqui Smith, eds. Toward a General Theory of
Expertise: Prospects and Limit. Cambridge: Cambridge University Press,
1991.

Cannon-Bowers, Janis, Eduardo Salas, and S. Converse. “Shared Mental
Models in Expert Team Decision Making” in J. N. John Castellan, ed. Cur-
rent Issues in Individual and Group Decision Making. Hillsdale, NY:
Lawrence Erlbaum Associates, 1983.

Capra, Fritjof. “Criteria of Systems Thinking,” Futures, October 1985: 475—
BB.

Carbonell, J. R. “Al in CAI: An Artificial Intelligence Approach to Computer-
Assisted Instruction” in IEEE Transactions on Man-Machine Systems, |1
(1970): 190-202; J. D. Fletcher and M. R. Rockway loc. cit.

Carmines, Edward, and Richard Zeller. Reliability and Validity Assessment.
Newbury Park, CA: Sage, 1991.

Cartwright, Dorwin, and Alvin Zander. Group Dynamics: Research and The-
ory. New York: Harper & Row, 1960.

Cascio, Wayne F. Applied Psychology in Human Resource Management. Sth
ed. Englewood Cliffs, NJ: Prentice Hall, 1998.

Cassirer, Ernest. “Le Langage et le Monde des Objets,” Journal de Psycholo-
gie Nomale et Pathologique 30, 1(4) (1933): 18-44.

Central Intelligence Agency. A Consumer’s Guide to Intelligence. Washing-
ton, DC: Central Intelligence Agency, 1993.

.. Factbook on Intelligence (http://www.cia.gov/cia/publications/fact-
tell/index.html.)

Chambers, Erve. Applied Anthropology: A Practical Guide. Prospect Heights,
IL: Waveland Press, 1989.

Chase, William G. “Spatial Representations of Taxi Drivers” in Don Rogers
and John Slobada, eds. Acquisition of Symbolic Skills. New York: Plenum,
1983.

, and Herbert Simon. “Perception in Chess,” Cognitive Psychology 4

(1973): 55-81.

, and K. Anders Ericsson. “Skill and Working Memory” in Gordon
Bower, ed. The Psychology of Learning and Motivation. New York: Aca-
demic Press, 1982.

Chi, Michelene, Robert Glaser, and Ernest Rees. “Expertise in Problem Solv-
ing” in Robert Sternberg, ed. Advances in the Psychology of Human Intelli-
gence. Hillsdale, NJ: Lawrence Erlbaum Associates, 1982.

, Paul Feltovich, and Robert Glaser. “Categorization and Representa-

tion of Physics Problems by Experts and Novices,” Cognitive Science 5

(1981): 121-25.

141



The Center for the Study of Intelligence
le
‘

Central Intelligence Agency
Washington, DC 20505

Library of Congress Cataloguing-in-Publication data
Johnston, Rob

Analytic Culture in the US Intelligence Community: An Ethnographic Study/
Dr. Rob Johnston

Includes bibliographic references.
ISBN 1-929667-13-2 (pbk.:alk paper)

1, Intelligence—United States. 2. Intelligence analysis.
3. Intelligence policy. 4. Intelligence training.

Typeset in Times and Ariel.
Printed by Imaging and Publication Support, CIA.

Cover design: Imaging and Publication Support, CIA.

The pensive subject of the statue is Karl Ernst von Baer (1792-1876), the
Prussian-Estonian pioneer of embryology, geography, ethnology, and physical
anthropology (Jane M. Oppenheimer, Encyclopedia Brittanica).



INSTRUCTIONAL TECHNOLOGY

synthesize the quantitative measures and describe the findings of the analysis.
Meta-analysis appears to be especially suited for synthesizing the results of
instructional research, and it has been widely used for this purpose since its
introduction in 1976.

Meta-analysis is still being developed as a technique, and some matters
concerning its use, notably the “file-drawer” problem and calculation of effect
size, remain unsettled. Chapter Twelve presents a more detailed explanation
and these considerations. Briefly, however, meta-analytic reviews of instruc-
tional technology effectiveness have found substantial results favoring its use
over traditional technologies of classroom instruction.

Overall, effect sizes for post-secondary school instruction average about
0.42, which is roughly equivalent to raising the achievement of 50th percentile
students to that of 66th percentile students.'> Reviews of more elaborate forms
of instructional technology, such as those using applied artificial intelligent
techniques, have found effect sizes in excess of 1.0, which is roughly equiva-
lent to raising the achievement of 50th percentile students to that of the 84th
percentile. It seems reasonable to conclude that the reduced costs and
reduced time to learn obtained in applications of instructional technology are
not achieved at the expense of instructional effectiveness.

Encouraging as these favorable results are, our ability to apply instructional
technology efficiently may be in its infancy. Findings thus far have been based
on instructional applications intended to teach facts (e.g., What is the capital
of Brazil? What is the Spanish word for chapel? Who was the first director of
the Central Intelligence Agency?) concepts (e.g., What is a mass spectrometer
used for? What is the difference between micro- and macro-economics? When
must you use a torque wrench?), and procedures (e.g., How do you record a
movie from television? How do you prepare a purchase requisition? How do
you calibrate a radar repeater?). All intelligence analysts must possess a reper-
toire of facts, concepts, and procedures to perform their craft, and instructional
technology holds great promise for increasing both the efficiency with which
they might develop this repertoire and their access to instructional resources
for doing so.

However, the capabilities analysts may seek through instruction are likely
to include more abstract, or “higher,” cognitive processes. For instance, in
addition to learning a procedure, analysts may need the capability to recognize

'S Chen-Lin Kulik., James Kulik and Barbara Shwalb, “Effectiveness of Computer-Based Adult
Education: A Meta-Analysis”; Chen-Lin Kulik and James Kulik, “Effectiveness of Computer-
Based Education in Colleges”; Rob Johnston and J. D. Fletcher, A Meta-Analysis of the Effecti
ness of Computer-Based Training for Military Instruction; J. D. Fletcher, “Evidence for Learning
from Technology-Assisted Instruction.”

' Sherrie P, Gott, R. $. Kane, and Alan Lesgold , Tutoring for Transfer of Technical Competence.

91



CHAPTER TWO.

Secrecy vs. Efficacy

a
CrEMfeciveress:

EMeiency

secrecy. I believe this secrecy-efficacy conflict can be stated as a theory, along
the following lines.®

The more open the system (where zero is perfect information access and
sharing on the X axis secrecy scale [as shown on the above graph]), the more
access an analyst has to all sources of information within the Intelligence
Community regarding an adversary. In addition, this openness encourages
interorganizational communication, interaction, and sharing of information
among analysts and increases the likelihood that an analyst will be more effi-
cient (in this case the Y axis efficiency scale) and therefore effective or accu-
rate in his or her assessment of a situation.

Conversely, counter-intelligence is negatively affected by zero-level
secrecy and perfect openness. The less open or more compartmentalized the
system, the more efficient and effective are counterintelligence activities.
Notionally, the two curves would meet somewhere in the tradeoff between
efficiency and secrecy. Where they meet would depend on program goals and
a clear definition of starting points and end-states.

The notional set of curves above illustrates the tradeoff between system
efficiency and system secrecy and the effect that the tradeoff has on perfor-
mance effectiveness, both positive and negative. In this case, the starting and
ending points of effectiveness for analysis and for counterintelligence are
arbitrary and could be positioned anywhere along a continuum between zero

© [would like to credit and thank Matthew Johnson at the Institute for Defense Analyses for his
help in formulating this theory.


INTEGRATING METHODOLOGISTS

Can Technology Help?

There are potential technological alternatives to multifaceted teams. For
example, an Electronic Performance Support System (EPSS) is a large data-
base that is used in conjunction with expert systems, intelligent agents, and
decision aids.*8 Although applying such a system to intelligence problems
might be a useful goal, at present, the notion of an integrated EPSS for large
complex data sets is more theory than practice.” In addition to questions
about the technological feasibility of such a system, there are fundamental
epistemological challenges. It is virtually inconceivable that a comprehensive
computational system could bypass the three complicating variables of exper-
tise described earlier.

An EPSS, or any other computational solution, is designed, programmed,
and implemented by a human expert from one domain only, that of computer
science. Historians will not design the “historical decision aid,” economists
will not program the “economic intelligent agent,” chemists will not create the
“chemical agent expert system.” Computer scientists may consult with various
experts during the design phase of such a system, but, when it is time to sit
down and write code, the programmer will follow the heuristics with which he
is familiar. In essence, one would be trading the heuristics of dozens of
domains for those that govern computer science. This would reduce the prob-
lem of processing time by simplifying and linking data, and it might reduce
pattern bias. It would not reduce heuristic bias, however; if anything, it might
exaggerate it by reducing all data to a binary state.*!

This skepticism is not simply a Luddite reaction to technology. Computa-
tional systems have had a remarkable, positive effect on processing time, stor-
age, and retrieval. They have also demonstrated utility in identifying patterns
within narrowly defined domains. However, intelligence analysis requires the
expertise of so many diverse fields of study and is not something a computa-
tional system handles well. Although an EPSS, or some other form of compu-
tational system, may be a useful tool for manipulating data, it is not a solution
to the paradox of expertise.

% An Expert System is a job-specific heuristic process that helps an expert narrow the range of
available choices. An Intelligent Agent is an automated program (bot) with built-in heuristics used
in Web searches. A Decision Aid is an expert system whose scope is limited to a particular task.
» R. Johnston, “Blectronic Performance Support Systems and Information Navigation.”

2, Johnston and J. Fletcher, A Meta-Analysis of the Effectiveness of Computer-Based Training
for Military Instruction,

3 J, Fletcher and R. Johnston, “Effectiveness and Cost Benefits of Computer-Based Decision
Aids for Equipment Maintenance.”

7



CHAPTER FIVE

to human biases to explain unreliable expert predictions.?! There is general
agreement that two types of bias exist:

* Pattern bias: looking for evidence that confirms rather than rejects a
hypothesis and/or filling in—perhaps inadvertently—missing data with
data from previous experiences;

* Heuristic bias: using inappropriate guidelines or rules to make predic-
tions.

Paradoxically, the very method by which one becomes an expert explains
why experts are much better than novices at describing, explaining, perform-
ing tasks, and solving problems within their domains but, with few excep-
tions, are worse at forecasting than are Bayesian probabilities based on
historical, statistical models. A given domain has specific heuristics for per-
forming tasks and solving problems, and these rules are a large part of what
makes up expertise. In addition, experts need to acquire and store tens of thou-
sands of cases in order to recognize patterns, generate and test hypotheses, and
contribute to the collective knowledge within their fields. In other words,
becoming an expert requires a significant number of years of viewing the
world through the lens of one specific domain. This concentration gives the
expert the power to recognize patterns, perform tasks, and solve problems, but
it also focuses the expert’s attention on one domain to the exclusion of others.
It should come as little surprise, then, that an expert would have difficulty
identifying and weighing variables in an interdisciplinary task, such as fore-
casting an adversary’s intentions. Put differently, an expert may know his spe-
cific domain, such as economics or leadership analysis, quite thoroughly, but
that may still not permit him to divine an adversary’s intention, which the
adversary may not himself know.

The Burden on Intelligence Analysts

Intelligence analysis is an amalgam of a number of highly specialized
domains. Within each, experts are tasked with assembling, analyzing, assign-
ing meaning to, and reporting on data, the goals being to describe an event or
observation, solve a problem, or make a forecast. Experts who encounter a
case outside their field repeat the steps they initially used to acquire their
expertise. Thus, they can try to make the new data fit a pattern previously
acquired; recognize that the case falls outside their expertise and turn to their
domain’s heuristics to try to give meaning to the data; acknowledge that the

21 J, Evans, Bias in Human Reasoning; R. Heuer, Psychology of Intelligence Analysis, D. Kahne-
man, P. Slovic, and A. Tversky, Judgment Under Uncertainty; A. Tversky and D, Kahneman,
“The Belief in the ‘Law of Small Numbers’.”

66



INTRODUCTION

ment, very open and honest; and this, too, may be attributable to 9/11. In any
case, that event is now part of the culture of the Intelligence Community, and
that includes whatever consequences or biases resulted from it.

Background

The opportunity to do this research presented itself, at least in part, as a
result of my participation in a multiyear research program on medical error
and failure for the Defense Advanced Research Projects Agency (DARPA).!
The DARPA research focused on team and individual error in minimally inva-
sive or laparoscopic surgical procedures. This research revealed that individ-
ual errors were cognitive rather than purely psychomotor or skill-based. For
example, some surgeons had trouble navigating three-dimensional anatomical
space using the existing laparoscopic technology, with the result that these
surgeons would identify anatomical structures incorrectly and perform a surgi-
cal procedure on the wrong body part.

Other individual errors were discovered during the DARPA studies, but, for
the most part, these were spatial navigation and recognition problems for
which there were technological solutions. Team errors, unlike individual
errors, proved to be more challenging. The formal and informal hierarchical
structures of operating rooms did not lend themselves to certain performance
interventions. Generally, junior surgical staff and support personnel were not
willing to confront a senior staff member who was committing, or was about
to commit, an error.

The culture of the operating room, coupled with the social and career struc-
ture of the surgical profession, created barriers to certain kinds of communica-
tion. For a surgical resident to inform a senior surgeon in front of the entire
operating room staff that he was about to cut the wrong organ could result in
career “suicide.” Such a confrontation could have been perceived by the
senior surgeon as a form of mutiny against his authority and expertise and a
challenge to the social order of the operating room. Although not universal,
this taboo is much more common than surgeons would care to admit. Unlike
individual errors, purely technological solutions were of little value in trying
to solve team errors in a surgical environment.

The DARPA surgical research was followed up by a multiyear study of
individual and team performance of astronauts at the National Aeronautics
and Space Administration’s (NASA) Johnson Space Center. Results of the
NASA study, also sponsored by DARPA, were similar to the surgical study

' Rob Johnston, J. Dexter Fletcher and Sunil Bhoyrul, The Use of Virtual Reality to Measure Sur-
gical Skill Levels.



CHAPTER THREE

that, “At the very least, analysis should fully describe the phenomenon under
study, accounting for as many relevant variables as possible. At the next
higher level of analysis, a thorough explanation of the phenomenon is
obtained, through interpretation of the significance and effects of its elements
on the whole.”'* In addition, several authors have written about individual ana-
lytic approaches. !5

Although the referenced works focus on methods and techniques, they do
not suggest that analysis is limited to these devices. The view that analysis is
both a process and a collection of specific techniques is explicit in the above
definitions. Analysis is seen as an action that incorporates a variety of tools to
solve a problem. Different analytic methods have something to offer different
analytic tasks.

Although largely implicit in the above definitions, analysis is also seen as a
product of cognition, and some authors directly link the two. Robert Mathams
defines analysis as: “[T]he breaking down of a large problem into a number of.
smaller problems and performing mental operations on the data in order to
arrive at a conclusion or generalization.” Avi Shlaim writes: “Since the facts
do not speak for themselves but need to be interpreted, it is inevitable that the
individual human propensities of an intelligence officer will enter into the pro-
cess of evaluation.”!7 Yet others describe analysis as a process whereby:
“{I]nformation is compared and collated with other data, and conclusions that
also incorporate the memory and judgment of the intelligence analyst are
derived from it.”'*

Several authors make the case that analysis is not just a product of cognition
but is itself a cognitive process. J. R. Thompson and colleagues write that
“[I]ntelligence analysis is an internal, concept-driven activity rather than an
external data-driven activity.”'? In his Psychology of Intelligence Analysis,
Heuer observes: “Intelligence analysis is fundamentally a mental process, but
understanding this process is hindered by the lack of conscious awareness of
the workings of our own minds.”? Ephraim Kam comments: “The process of
intelligence analysis and assessment is a very personal one. There is no
agreed-upon analytical schema, and the analyst must primarily use his belief

1 Bruce D. Berkowitz and Allan E. Goodman, Strategic Intelligence for American National Secu-
rity, 85. See Chapter Four for more on the intelligence cycle.

"6 Lisa Krizan, Intelligence Essentials for Everyone

15 See the apprendix for a listing of the literature.

‘© Robert Mathams, “The Intelligence Analyst’s Notebook.”

” Avi Shlaim, “Failures in National Intelligence Estimates: The Case of the Yom Kippur War.”

'8 John Quirk et al., The Central Intelligence Agency: A Photographie History.

'° J. R. Thompson, R. Hopf-Weichel, and R. Geiselman, The Cognitive Bases of intelligence
Analysis.

2 Richards J. Heuer, Jr., Psychology of Intelligence Analysis,

36



BIBLIOGRAPHY

. “The Prosperous Community: Social Capital and Public Life,” The
American Prospect 4, no. 13 (1993): 35-42.

Quirk, John, et al. The Central Intelligence Agency: A Photographic History.
Guilford, CT: Foreign Intelligence Press, 1986.

Reason, James. Human Error. New York: Cambridge University Press, 1990.

Reichenbach, Hans. Experience and Prediction. Chicago: University of Chi-
cago Press, 1938.

Richelson, Jeffrey. The U.S. Intelligence Community, 4th ed. Boulder, CO:
Westview Press, 1999.

Roethlisberger, Fritz J., and William J. Dickson. Management and the Worker:
An Account of a Research Program Conducted by the Western Electric Com-
pany, Hawthorne Works, Chicago, Boston: Harvard University Press, 1939.

Rosenberg, Marc J. “Performance Technology: Working the System,” Train-
ing, January 1990: 5-10.

Rossi, Peter, and Howard Freeman. Evaluation. A Systematic Approach, Sth
ed. Newbury Park, CA: Sage, 1993.

Roth, P. L. ,and James E. Campion. “An Analysis of the Predictive Power of
the Panel Interview and Pre-Employment Tests,” Journal of Occupational
and Organizational Psychology 65 (1992): 51-60.

Sacks, H. “Promises, Performance, and Principles: An Empirical Study of
Parole Decision-Making in Connecticut,” Connecticut Law Review 9 (1977):
349-422.

Salomon, G., and D.N. Perkins. “Rocky Roads to Transfer: Rethinking Mech-
anisms of a Neglected Phenomenon,” Educational Psychologist 24 (1998),

Sapir, Edward. Language. New York: Harcourt Brace, 1949.

Sapp, Edwin. “Decision Trees,” Studies in Intelligence 18 (Winter 1974)
(declassified): 45-57.

Sarbin, Theodore. “A Contribution to the Study of Actuarial and Individual
Methods of Prediction,” American Journal of Sociology 48 (1943): 593-602.

Saussure, Ferdinand. Course in General Linguistics. Transtated by W. Baskin.
New York: McGraw-Hill, 1959.

Sawyer, J. “Measurement and Prediction, Clinical and Statistical,” Psycholog-
ical Bulletin 66 (1966): 178-200.

Schensul, Jean, and Margaret LeCompte, eds. Ethnographer’s Toolkit. Vol. I-
Vol. VIE. Walnut Creek, CA: Alta Mira Press, 1999.

Schofield, W., and J. Garrard. “Longitudinal Study of Medical Students
Selected for Admission to Medical School by Actuarial and Committee
Methods,” British Journal of Medical Education 9 (1975): 86-90.

Schum, David. The Evidential Foundations of Probabilistic Reasoning. Evan-
ston, IL: Northwestern University Press, 1994.

Scriven, M. “Problems and Prospects for Individualization” in H. Talmage,
ed. Systems of Individualized Education. Berkley, CA: McCutchan, 1975.

153



CHAPTER SIX

As an observer, I missed the cultural context that was necessary to view the
events as an actual conflict and could not convince myself that a violent solu-
tion was a possibility. I had discounted the hypothesis that violence would
occur, because I could not imagine it occurring in the United States. This led
me to discount raw data that would have refuted a hypothesis that the two fac-
tions would reach a compromise. In addition, at that time, I had no formal
grounding in Chinese studies, nor had I been to China. Thus, I had not
acquired information that would have helped me create a meaningful context
for the event.

Years later, my wife and I were in China doing ethnographic fieldwork on
the socioeconomic effects of the spread of the English language and American
culture in urban and rural China.* While there, we spent a great deal of time
talking with others about the events of Tiananmen, and we decided to include
in our research questions about the student protests, if for no other reason than
to satisfy our own curiosity.

What we found stood in contrast to media reports and the opinions
expressed by many pundits and scholars in the US and the West. After hun-
dreds of interviews with a wide variety of people in and around Beijing, we
found a consistent preoccupation among the “silent majority.” That was the
Cultural Revolution, which had affected all of the people we interviewed.
They had been participants, observers, or survivors, and, often, all three.

In the mid-1960s, Mao Zedong sought to recapture power from reform-
minded opponents within the Communist Party. Using radical party leaders as
his instruments, he created the Red Guard, which was made up primarily of
college students (although others followed suit in time). The image of the Cul-
tural Revolution was not simply the image of Mao; it was also the image of
angry, violent, and powerful college students, who were the most visible pro-
ponents of the “Cult of Mao.” According to the people we interviewed, it was
the students who had chanted slogans, raised banners, paraded in public
spaces, resisted older forms of social control, and seized power. With that
power and the blessings of Mao, the youth and university students had com-
mitted many of the atrocities of the Cultural Revolution and plunged China
into a decade of chaos, during which many institutions, including schools,
were closed and many of the country’s cultural and historical artifacts were
destroyed.

« American anthropology is based on the ethnographic method and direct interaction with the peo-
ple who are being studied. This interaction includes direct and participant observation and inter-
views, or fieldwork, where one lives with the people being investigated. Continental European
schools of anthropology are not as obsessed with methodology and hands-on experience and tend
to the more theoretical.

7B



BIBLIOGRAPHY

, and Lynne Martin. “Errors in Aviation Decision Making: A Factor in
Accidents and Incidents,” Proceedings of the Working Group on Human
Error, Safety, and System Development. Seattle, WA: International Federa-
tion for Information Processing, 1998: 100-107.

Orlansky, Jesse, and Joseph String. Cost-Effectiveness of Computer-Based
Instruction in Military Training. Alexandria, VA: Institute for Defense Anal-
yses, 1979, IDA P-1375.

, et al. The Cost and Effectiveness of the Multi-Service Distributed

Training Testbed (MD12) for Training Close Air Support. Alexandria, VA:

Institute for Defense Analyses, 1997.

. The Value of Simulation for Training. Alexandria, VA: Institute for
Defense Analyses, 1994. IDA P-2982.

Paige, Jeffery, and Herbert Simon. “Cognition Processes in Solving Algebra
Word Problems” in Benjamin Kleinmuntz, ed. Problem Solving. New York:
Wiley, 1966.

Patton, Michael. Qualitative Evaluation and Research Methods, 2nd ed. Bev-
erly Hills, CA: Sage, 1990.

Perrow, Charles. Normal Accidents: Living with High Risk Technologies. Prin-
ceton, NJ: Princeton University Press, 1999.

. Written communications between Perrow and Johnston, 2002.

Peterson, Marilyn. Applications in Criminal Analysis. Westport, CT: Green-
wood Press, 1995.

, Bob Morehouse, and Richard Wright, eds. Intelligence 2000: Revis-
ing the Basic Elements. Sacramento, CA: LEIU and [ALEIA, 2000.

Phelps, Ruth, et al. “Effectiveness and Costs of Distance Education Using
Computer-Mediated Communication,” American Journal of Distance Edu-
cation 5, no. 3 (1991): 7-19.

Pierce, John. “Some Mathematical Methods for Intelligence Analysis,” Stud-
ies in Intelligence 21 (Summer 1977) (declassified): 1-19.

Popper, Karl. The Logic of Scientific Discovery. London: Hutchinson, 1959.

. Conjectures and Refutations: The Growth of Scientific Knowledge.
New York: Routledge, 2002 reprint.

Porter, Lyman W., Edward E. Lawler, and J. Richard Hackman. Behavior in
Organizations. New York: McGraw-Hill, 1975.

, et al. “Organizational Commitment, Job Satisfaction and Turnover
Among Psychiatric Technicians,” Journal of Applied Psychology 59 (1974):
603-609.

Prescott, John, and Stephen Miller, eds. Proven Strategies in Competitive
Intelligence. New York: John Wiley & Sons, 2001.

Price-Williams, Douglass. Explorations in Cross-Cultural Psychology. San
Francisco: Chandler and Sharp, 1975.

Putnam, Robert. Bowling Alone: The Collapse and Revival of American Com-
munity. New York: Simon and Schuster, 2000.

152



TESTING THE INTELLIGENCE CYCLE

should not simply test effectivenss; they should also identify vulnerabilities and
potential sources of other problems in the community's analytical methods.



CHAPTER TWO

distinction, for language is a key variable in anthropology and often reveals a
great deal about the cognition and culture of a community of interest.°

The adoption by members of the analytic community of an inappropriate
term for the processes and methods employed in their professional lives
obfuscates and complicates the reality of their work. The adoption of the
word “tradecraft” demonstrates the analytic community’s need to create a pro-
fessional identity separate and unique from other disciplines but tied directly
to the perceived prestige and cachet of intelligence operations. Adopting
“tradecraft” as a term of reference for explaining work practices and as a pro-
fessional identity marker may seem trivial. Yet the term, and its effect on the
community, has unanticipated consequences.

Tradecraft purposefully implies a mysterious process learned only by the
jated and acquired only through the elaborate rituals of professional indoc-
trination. It also implies that the methods and techniques of analysis are infor-
mal, idiosyncratic, unverifiable, and perhaps even unexplainable. “Good”
methods are simply those that survive, and then are passed on by “good” ana-
lysts to novice analysts. Unfortunately, “good” in both instances is not an
objective measure. That is, there is no formal system for measuring and track-
ing the validity or reliability of analytic methods, because they are both per-
ceived and employed within the context of idiosyncratic tradecraft. When
asked to describe the analytic process, analysts responded in a variety of ways.

First, I figure out what I know and what I don’t know about some
situation. Then, I look for information to fill the gap.

I have a model of the situation in my head. Whenever something new
comes in, I see if it fits with the model. If it does, I add it to the
model, if it doesn't, I try to figure out why.

I’ve found a system that lets me keep up. I just look for anomalies.
When I see any novel data, then I worry.

I'm always looking for anything strange or out of place. Then, 1
source it to see if it is meaningful.

The current data ought to fit a certain pattern. If it doesn’t, I know
something is wrong.

First, I print the daily traffic I’m concerned with, then I lay out all
of the relevant stuff in front of me on my desk or the floor; then I
start looking for threads.

° The literature on this subject is extensive. For a representative list, see the appendix.





ORGANIZATIONAL CULTURE

Anticipatory Socialization

Anticipatory socialization encompasses all of the learning that occurs prior
to a recruit’s entering on duty.’ At this stage, an individual forms expectations
about the job and makes decisions about the suitability of fit between himself
and the organization. What a person has heard about working for a particular
organization, such as an intelligence agency, provides an idea of what to
expect if hired. Conversely, individuals who do not believe they would fit in
may decide not to apply.

There are two variables that are particularly useful for tracking a potential
employee’s progress through the anticipatory stage: The first is realism, or the
extent to which an individual acquires an accurate picture of daily life in the
organization. Realism is influenced by the level of success recruits achieve
during the information-sharing and information-evaluation part of their
recruitment. The second is congruence, or the extent to which the organiza-
tion’s resources and the individual’s needs and skills are mutually satisfying.
Congruence is influenced by the level of success an individual has achieved in
making decisions about employment. Although it cannot directly influence
congruence, which is an inherently personal experience, an organization can
present relevant information in order to provide a realistic and accurate
description of the work performed and the work environment.

Organizations often use interviews to begin the socialization of new
recruits. For example, an interviewer will attempt to provide an accurate
description of what to expect from the job and the organization, the purpose
being to reduce the likelihood that a recruit will be disturbed by unanticipated
situations. Interviewing is also used to determine the degree to which there is a
match between the values of potential recruits and the values of the organiza-
tion. New recruits with personal values matching those of the organization
have been found to adjust to the organization’s culture more quickly than
recruits with nonmatching values.*

Organizations also send cultural messages to new recruits during inter-
views. When there are several rounds of interviews with progressively senior
members of the organization, for example, the message conveyed is that find-
ing the best person for the position is important. In contrast, hiring for a part-
time job at the lowest level of the organization is often accomplished quickly,
to the extent that a person having minimally acceptable qualifications may

7 This stage is termed “pre-arrival” in Lyman W, Porter, Edward E. Lawler, and J. Richard Hack-
man, Behavior in Organizations

* Jerald Greenberg and Robert A. Baron, Behavior in Organizations: Understanding the Human
Side of Work.

99



A TAXONOMY OF INTELLIGENCE

ing intelligence from other such activities.’ He argues that the behavior of the
subject of intelligence changes if the subject is aware of being observed or
analyzed. As discussed earlier, Warner’s argument is supported by a long his-
tory of psychological research, beginning with an experimental program
between 1927 and 1930 at Western Electric’s Hawthorne Works in Chicago.*

Intentional deception can occur outside intelligence—in connection with
certain law enforcement functions, for example—but most of the professional
literature treats this as the exception rather than the rule. In the case of intelli-
gence analysis, deception is the rule; the validity of the data is always in
doubt. Moreover, intelligence analysts are specifically trained to take decep-
tion into account as part of the analytic process—to look for anomalies and
outliers instead of focusing on the central tendencies of distribution.

The taxonomy being developed here requires a definition of intelligence
analysis that is specific to the field. Intelligence pioneer Sherman Kent, who
saw intelligence as a “special category of knowledge,” laid the foundation for
understanding the activities inherent in intelligence analysis by demonstrating
that the analytic process itself was subject to being analyzed. Kent’s approach
to analysis was to reduce the process to smaller functional components for
individual study.'° For example, he described intelligence analysis as having a
basic descriptive element, a current reporting clement, and an estimative ele-
ment.

Following suit, other authors focused attention on the process or method-
ological elements of intelligence analysis. In Intelligence Research Methodol-
ogy, Jerome Clauser and Sandra Weir followed Kent’s three functional areas
and went on to describe basic research foundations and the inductive and
deductive models for performing intelligence analysis.'' Garst’s Handbook of
Intelligence Analysis contains less background in basic research methods than
Clauser and Weir’s book, but it is more focused on the intelligence cycle.'?

Bruce Berkowitz and Allan Goodman highlight the process of strategic
intelligence and define intelligence analysis as: “[T]he process of evaluating
and transforming raw data into descriptions, explanations, and conclusions for
intelligence consumers.”'3 Lisa Krizan, too, focuses on process. She writes

? Michael Warner.

5 The Hawthorne Effect. See footnote 5 in Chapter Two.

9 Sherman Kent, Strategic Intelligence for American World Policy.

1 See Chapter Seven for a fuller discussion of this approach, now usually referred to as meta-
analysis.

» Jerome K. Clauser and Sandra M. Weir, intelligence Research Methodology.

"2 See also: Morgan Jones, The Thinker’s Toolkit. Jones's book is a popular version of the work of
Garst and Clauser and Weir in that it describes a collection of analytic methods and techniques for
problem-solving: however, the methods are not necessarily specific to intelligence.

35



CHAPTER TEN

research methodologies. Additionally, ethnography is used to identify and
describe the influence of different variables on cultural phenomena, again
with a focus on developing testable theory. Unlike more quantitative disci-
plines, cultural anthropology is not traditionally employed experimentally to
test theory or to generate predictive measures of statistical significance.

The findings in this work describe the data collected during this study, but
they do not indicate the weight or general statistical effect of any one variable
as opposed to any other variable. Although a single variable might have more
effect on the error or failure rate of intelligence analysis, further quantitative
research will be needed to determine those statistical values. Without addi-
tional quantitative support, it may not be possible to generalize from these
findings.

Methodology

This study used an applied anthropological methodology for the collection
and analysis of qualitative data. A traditional approach to ethnography, the
descriptive documentation of living cultures, was modified for use in post-
industrial organizational settings. This method included conducting inter-
views, directly observing analysts performing their jobs, participating in ana-
lytic tasks and training, and conducting focus groups. The settings for this
research included the 14 members of the Intelligence Community, related
government agencies, universities, think tanks, national laboratories, the
National Archives and related presidential libraries, and private and corporate
locations.

The background data were collected using a Q-sort literature review method,
which is discussed in more detail in Chapters Three and Eleven. This procedure
was followed by semi-structured interviews, direct observation, participant
observation, and focus groups. The Q-sort method was employed specifically
because of its utility for developing taxonomic categories.‘

The identity of the research participants will not be revealed. Participant
responses and observational data gathered during the research process have
been tabulated and made anonymous or aggregated according to context and

2 Erve Chambers, Applied Anthropology: A Practical Guide; Alexander Ervin, Applied Anthro-
pology: Tools and Perspectives for Contemporary Practice.

3 Russell Bernard, Research Methods in Anthropology: Qualitative and Quantitative Approaches;
Robert Bogdan, Participant Observation in Organizational Settings; Norman Denzin and Yvonna
Lincoln, Handbook of Qualitative Research; Jean Schensul and Margaret LeCompte, Ethnogra-
pher's Toolkit. Vol. I - Vol. VI; James Spradley, Participant Observation; Robert Yin, Case
Study Research: Design and Methods.

+ William Stephenson, The Study of Behavior: Q-Technique and its Methodology.

120



PART II
Ethnography of Analysis

a

31





CHAPTER FIVE

shared by each team member. Teams also require interdependence and
accountability, that is, the success of each individual depends on the success of
the team as a whole as well as on the individual success of every other team
member.

Effective teams require cohesion, formal and informal communication,
cooperation, shared mental models, and similar knowledge structures.?5 Put-
ting combinations such as this in place is not a trivial task. Creating shared
mental models may be fairly easy within an air crew or a tank crew, where an
individual's role is clearly identifiable as part of a clearly-defined, repetitive
team effort, such as landing a plane or acquiring and firing on a target. It is
more difficult within an intelligence team, given the vague nature of the goals,
the enormity of the task, and the diversity of individual expertise. Moreover,
the larger the number of team members, the more difficult it is to generate
cohesion, communication, and cooperation. Heterogeneity can also be a chal-
lenge; it has a positive effect on generating diverse viewpoints within a team,
but it requires more organizational structure than does a homogeneous team.”6

Without specific processes, organizing principles, and operational struc-
tures, interdisciplinary teams will quickly revert to being simply a room full of
experts who ultimately drift back to their previous work patterns. That is, the
experts will not be a team at all; they will be a group of experts individually
working in some general problem space.””

2 Dorwin Cartwright and Alvin Zander, Group Dynamics: Research and Theory; P. Fandt, W.
Richardson, and H. Conner, “The Impact of Goal Setting on Team Simulation Experience”: J.
Harvey and C, Boettger, “Improving Communication within a Managerial Workgroup.”

2 M, Deutsch, “The Effects of Cooperation and Competition Upon Group Process”; D. Johnson
and R. Johnson, “The Internal Dynamics of Cooperative Learning Groups”; D. Cartwright and A.
Zander, Group Dynamics: Research and Theory; David Johnson and Roger Johnson, “The Inter-
nal Dynamics of Cooperative Learning Groups”; D. Johnson et al, “Effects of Cooperative, Com-
petitive, and Individualistic Goal Structure on Achievement: A Meta-Analysis”; R. Slavin,
“Research on Cooperative Learning”; R. Slavin, Cooperative Learning.

2 J, Cannon-Bowers, E. Salas, S. Converse, “Shared Mental Models in Expert Team Decision
Making”; L. Coch and J. French, “Overcoming Resistance to Change”; M. Deutsch, “The Effects
of Cooperation and Competition Upon Group Process”; L. Festinger, “Informal Social Communi-
cation”; D. Johnson et al., “The Impact of Positive Goal and Resource Interdependence on
Achievement, Interaction, and Attitudes”; B. Mullen and C, Copper, “The Relation Between
Group Cohesiveness and Performance: An Integration”; W. Nijhof and P. Kommers, “An Analy-
sis of Cooperation in Relation to Cognitive Controversy”; J. Orasanu, “Shared Mental Models
and Crew Performance”; $. Seashore, Group Cohesiveness in the Industrial Work-group.

26 T, Mills, “Power Relations in Three-Person Groups”; L. Molm, “Linking Power Structure and
Power Use”; V. Nieva, E. Fleishman, and A. Rieck, Team Dimensions: Their Identity, Their Mea-
surement, and Their Relationships: G. Simmel, The Sociology of Georg Simmel

2 R. Johnston, Decision Making and Performance Error in Teams: Research Results; J. Meister,
“Individual Perceptions of Team Learning Experiences Using Video-Based or Virtual Reality
Environments.”

70



FINDINGS

We get all kinds of sourced material, but I think I trust technical col-
lection more than the other INTs.

I try to use everything we get, but, if we are jammed, I rely on
sources we collect.

Our value-added is classified sourcing. Everybody has access to the
Web and CNN.

All our customers are analysts these days. What we bring to the
party is information no one else has.

We're in the business of secrets. If you see that stamped on some-
thing, it must be there for a reason.

The over reliance on classified information for hypothesis testing creates a
situation in which the data are screened and sorted by the organization before
they are selected and tested by the analysts. Classified information comes
from very specific types of technical and human sources, and it is filtered
through very specific reporting channels. It also has a tendency to become
homogeneous because of the source types and reporting mechanisms. Because
it is generated and packaged in specific formats using specific processes, clas-
sified information lacks the diversity that is inherent in open information, and
this contributes to confirmation bias.

In sum, operating under difficult time constraints, trying to make new work
accord with previous products, trying to maintain the prestige and power of
the organization, and assigning greater weight to secret information than to
open information have a cumulative effect, and the analyst often finds himself
or herself trying to produce daily products using the most time-efficient strate-
gies available instead of generating or testing hypotheses by way of refutation.

The persistence of the notion of tradecraft, coupled with organizational
norms, promotes the use of disjointed analytic strategies by separating intelli-
gence analysts from other scientific disciplines. These conditions have had an
effect on the self-concept of analysts and have molded the way analysts per-
ceive their own identity.

Finding: Analytic Identity

The self is something which has a development; it is not initially
there, at birth, but arises in the process of social experience and
activity, that is, develops in the given individual as a result of his
relations to that process as a whole and to other individuals within
that process.

George Mead.'7

25



RECOMMENDATIONS

ligence analysis; rather, each type of task will have an analytic method that is
best suited to accomplishing it in an efficient and effective manner.

Developing these metrics is no small task, It is a job that will require
numerous researchers and research programs within, or with access to, the
Intelligence Community. These programs will need formal relationships with
human resource departments, analytic divisions, organizational leadership,
and developers of training and technology interventions in order to have a
positive effect on analytic performance.

Research Programs

The results of this research indicate that the Intelligence Community needs
to commit itself to performance research that is rigorous, valid (in that it mea-
sures what it proposes to measure), and replicable (in that the method is suffi-
ciently transparent that anyone can repeat it). Within some intelligence
organizations, this has been an ongoing process. The problem is that most of
the internal research has concentrated on historical case studies and the devel-
opment of technological innovations. What is missing is focused study of
human performance within the analytic components of the Intelligence Com-
munity. Questions about the psychology and basic cognitive aptitude of intel-
ligence analysts, the effectiveness of any analytic method, the effectiveness of
training interventions, group processes versus individual processes, environ-
mental conditions, and cultural-organizational effects need to be addressed.

This effort will require commitment. Researchers will have to be brought
into the Intelligence Community, facilities will have to be dedicated to
researching analytic performance, expert analysts will have to give some per-
centage of their time to participating in research studies, managers and super-
visors will have to dedicate time and resources to tracking analytic
performance within their departments, human resource staffs will have to ded-
icate time and resources to developing a performance repository, and there
will have to be formal interaction between researchers and the community.

Analytic Performance Research. In the previous section, I discussed the
need for analytic standards as part of the Performance Improvement Infra-
structure. In terms of a research program, this will require, as a first step, the
collection of baseline analytic performance data and a clear and measurable
description of ideal analytic behavior. Next, there should be a determined
effort by human performance researchers to develop, test, and validate ana-
lytic performance metrics and measurement systems. This will be a lengthy
process. The accuracy and surprise measures suggested in this text require
large historical and comparative data sets and are cumbersome and time con-
suming to perform. Conducting behavioral, cognitive, and linguistic task

WW



INSTRUCTIONAL TECHNOLOGY

« allow either the system or the student to ask open-ended questions and ini-
tiate instructional, “mixed-initiative” dialogue as needed or desired;

* generate instructional material and interactions on demand instead of
requiring developers to foresee and store all the materials and interactions
needed to meet all possible eventualities.

Mixed-initiative dialogue requires a language for information retrieval,
tools to assist decisionmaking, and instruction that is shared by both the sys-
tem and the student/user. The system must have the capability (referred to as
“generative capability”) to devise, on demand, interactions with students that
do not rely on predicted and prestored formats. This capability involves more
than generating problems tailored to each student’s needs. It must also provide
the interactions and presentations that simulate one-on-one tutorial instruc-
tion, including coaching, hints, and critiques of completed solutions.

Cost containment is one motivation for wanting to generate responses to all
possible student states and actions instead of attempting to anticipate and store
them. Another arises from basic research on human learning, memory, percep-
tion, and cognition. As documented by Neisser among others, during the
1960s and 1970s, the emphasis in basic research on human behavior and on
the way in which it is understood shifted from the strict logical positivism of
behavioral psychology, which focused on directly observable actions, to con-
sideration of the internal, cognitive processes that were needed to explain
empirically observed behavioral phenomena and are assumed to mediate and
enable human learning.**

The hallmark of this approach is the view that seeing, hearing, and remem-
bering are all acts of construction, making more or less use of the limited
stimulus information provided by our perceptual capabilities. Constructivist
approaches are the subject of much current and relevant discussion in instruc-
tional research circles, but they are firmly grounded in the foundations of sci-
entific psychology.*¢ For instance, in 1890, William James stated his General
Law of Perception: “Whilst part of what we perceive comes through our
senses from the object before us, another part (and it may be the larger part)
always comes out of our mind.”7

In this sense, the generative capability sought by intelligent instructional
systems is not merely something nice to have. It is essential if we are to

& JR, Carbonell, “Al in CAL: An Artificial Intelligence Approach to Computer-Assisted Instruc-
tion”; J. D. Fletcher & M. R. Rockway.

35 U. Neisser, Cognitive Psychology.

2 For example, T. M. Duffy, and D. H. Jonassen, Constructivism and the Technology of Instruc-
tion; 8. Tobias and L. T. Frase, “Educational psychology and training.”

¥ William James, Principles of Psychology: Volume I.

95



CHAPTER EIGHT

Conclusion and Recommendations

As noted, there is something of a disconnect between the largely fictional-
ized portrayal of the Intelligence Community in the popular media and the
actual experience of intelligence analysts. This disconnect can be exacerbated
once a recruit is on the job and can lead to negative consequences and behav-
iors, such as organizational culture shock, employee dissatisfaction, and
increased employee absenteeism and turnover. This has an obvious effect on
individual analysts, but it has a direct effect on the efficiency and effectiveness
of the Intelligence Community.

Since the September 2001 attacks, some members of the Intelligence Com-
munity have acted to change the socialization process by providing accurate
and realistic career information. One of the most widely used media for this is
the Internet. For example, the Central Intelligence Agency’s (CIA) Web site
contains a section on “Life at the CIA.”'8 This section contains information
about the Agency and its culture, several analyst profiles and job descriptions
written in the analyst’s own words, and information concerning employee
benefits and social and intellectual diversity. Although the “Employment”
section of the Federal Bureau of Investigation’s (FBI) Web site is less detailed
than the “Life at the CIA” section of the CIA Web site, it does illustrate a typ-
ical first assignment.'? In contrast, the “Careers” section of the Defense Intelli-
gence Agency’s (DIA) Web site contains detailed information on current job
openings and the application process, but it provides no information about the
actual work of a DIA analyst.” Steps such as these are encouraging, but they
are still insufficient. There are more active things that can be done to facilitate
the socialization of new employees.

To begin, Intelligence Community components should accept that what
most people know about a job is often false and that it is incumbent on the
organization and its recruiters to present accurate pictures and to work dili-
gently to dispel myths. This will help to counteract the effects of culture
shock. Instead of overselling a particular job or organization, recruiters should
focus on facilitating the anticipatory socialization of potential employees by
providing accurate information about the job and about the culture of the orga-
nization itself. Early in the selection process, applicants should be provided
with realistic job previews, presented in either written or oral form. Previews
should contain accurate information about the specific conditions within an
organization and the specific requirements of the job. Research has shown that
providing accurate descriptions of tasks is important in increasing job com-
mitment and job satisfaction, as well as decreasing initial turnover of new

"8 See Central Intelligence Agency Web site in Web Resources.
° See Federal Bureau of Investigation Web site in Web Resources
2 See Defense Intelligence Agency and US Intelligence Community Web sites in Web Resources.

104



RECOMMENDATIONS

to determine and predict the effect on organizational performance. The same
is true with the Intelligence Community. Although there have been numerous
proposals to reorganize the Intelligence Community—including those that
resulted from the hearings of the Kean 9/11 commission—few have addressed
the question of why one change would be any more effective than any other
change. Merely asserting, based on some a priori notion of effectiveness, that
organizational scheme X is more effective than organizational scheme Y is
insufficient evidence. What is needed is a posteriori data, such as case studies,
to support or refute the proposed change.!

Organizational Requirements. Many large organizations distribute perfor-
mance improvement responsibilities throughout the organization at a supervi-
sory or midlevel of management, but the group most often charged with
collecting and analyzing performance data is the human resources department.
This task generally involves developing task-specific performance standards
and metrics based on expert performance models and in accordance with cor-
porate policy.

The human resources department also becomes the central repository for
pre-, periodic, and post-performance measurements. As this department gen-
erally has contact with employees throughout their careers, this is the most
efficient way to manage, analyze, and inform senior leadership about aggre-
gate changes in performance over time. Although data are collected at the
individual level, it is the aggregation of performance data that allows leader-
ship to determine the effectiveness of any organizational change or job-related
intervention.

Baseline Data. Measuring actual analytic performance is essential to the
establishment of a data driven performance infrastructure. The analysts in this
study perceived their performance to be tied directly to the quantity of written
products they produced during each review period. Counting the number of
analytic publications is one metric, of course, but it is hardly indicative of ana-
lytic quality. Surgeons are a useful example of this problem.. They may
count the number of patients they treat, but this metric says more about system
throughput and salesmanship than it does about surgical performance. Unlike
the purely cognitive work of intelligence analysts, surgeons have the advan-
tage of multiple physical outputs, which makes measurement an easier task.
In particular, surgeons have patient outcomes, or morbidity and mortality
ratios, which become a grounded end-state for all measurements.” Other
things being equal, these data then ought to inform a prospective patient about
where to take his or her business.

| William Nolte, a deputy assistant director of central intelligence for analysis and production pro-
posed such an idea in “Preserving Central Intelligence: Assessment and Evaluation in Support of
the DCI” in Studies in Intelligence 48, no. 3 (2004): 21-25.

109



APPENDIX

Selected Literature

a

Intelligence Tools and Techniques

Jerome K. Clauser and Sandra M. Weir, Intelligence Research Methodology:
An Introduction to Techniques and Procedures for Conducting Research in
Defense Intelligence (Washington, DC: US Defense Intelligence School,
1976).

Stanley Feder, “FACTIONS and Policon: New Ways to Analyze Politics,”
(1987) in H. Bradford Westerfield, ed., Inside CIA's Private World: Declas-
sified Articles from the Agency’s Internal Journal (New Haven: Yale Univer-
sity Press, 1995).

Craig Fleisher and Babette Bensoussan, Strategic and Competitive Analysis.
Methods and Techniques for Analyzing Business Competition (Upper Saddle
River, NJ: Pearson Education, 2003).

Leonard Fuld, The New Competitor Intelligence. The Complete Resource for
Finding, Analyzing, and Using Information About Your Competitors (New
York: John Wiley & Sons, 1995).

Ronald Garst, ed., A Handbook of Intelligence Analysis, 2nd ed (Washington,
DC: Defense Intelligence College, 1989).

R. Hopkins, Warnings of Revolution: A Case Study of El Salvador (Washing-
ton, DC: Center for the Study of Intelligence, 1980) TR 80-100012.

Morgan Jones, The Thinker’s Toolkit: 14 Powerful Techniques for Problem
Solving (New York, NY: Times Business, 1998).

Larry Kahaner, Competitive Intelligence. How to Gather, Analyze, and Use
Information to Move Your Business to the Top (New York: Touchstone,
Simon and Schuster, 1996).

133



APPENDIX

Dell Hymes, ed. Language in Culture and Society: A Reader in Linguistics
and Anthropology (New York: Harper and Row, 1977).

Penny Lee, The Whorf Theory Complex: A Critical Reconstruction (Amster-
dam: Benjamins, 1996).

Lisa Liu, “Reasoning Counterfactually in Chinese: Are There any Obstacles?”
Cognition 21 (1985): 239-70.

John Lucy, Grammatical Categories and Cognition (Cambridge: Cambridge
University Press, 1992) and Language Diversity and Thought (Cambridge:
Cambridge University Press, 1992).

George Miller and Philip Johnson-Laird, Language and Perception (Cam-
bridge, MA: Harvard University Press, 1976).

Richard Nisbett and Ara Norenzayan, “Culture and Cognition” in Douglas
Medin and Hal Pashler, eds. Stevens’ Handbook of Experimental Psychol-
ogy, 3rd ed. (New York: John Wiley & Sons, 2002).

Edward Sapir, Language (New York: Harcourt Brace, 1949).

Jost Trier, Der deutsche Wortschatz im Sinnbezirk des Verstandes (Heidelberg:
Carl Winter, 1931).

Leo Weisgerber, Vom Weltbild der Deutschen Sprache (Dusseldorf: Padago-
gischer Verlag Schwann, 1953).

Benjamin Whorf, Language, Thought, and Reality (New York: John Wiley,
1956).

137



ANALYTIC CULTURE IN THE US
INTELLIGENCE COMMUNITY



CHAPTER ELEVEN
Q-Sort Methodology

a

Observations always involve theory.
Edwin Hubble!

As described earlier, this work reflects triangulation of the data derived
from the literature Q-sort, interview responses, and observations.? The data
include 489 interviews, direct and participant observation of 325 analysts per-
forming their jobs, participation in a variety of analytic tasks, and focus
groups conducted to generate the taxonomy of variables that guided this study.

The first Q-sort of the data was aggregated according to the function of
each intelligence organization, as listed in Table 1. The data were then ana-
lyzed to determine response context according to job type and to develop vari-
able categories.

The organizational Q-sort generated the broad variable groupings used to
create the second Q-sort parameters. The variable categories that emerged
during the interpretive analysis of the first Q-sort of the data were compiled
again, and a second Q-sort was performed based on those categories. The data
was then aggregated according to categorical or variable groupings of the sec-
ond Q-sort, Table 2.

The use of two separate Q-sort strategies generated the variables and then
de-contextualized the data in order to find consistent trends throughout the
Intelligence Community. That is, this strategy resulted in broad categories of
findings that apply across many agencies. In those cases where interview and

' Edwin Hubble discovered the first evidence to support the Big Bang theory that the universe is
expanding and that the Milky Way is not the only galaxy in the universe. He also developed the
Hubble Galaxy Classification System and Hubble’s Law (the farther away a galaxy is from Earth,
the faster its motion away from Earth). Edwin Hubble, The Realm of the Nebulae.
2 William Stephenson, The Study of Behavior: Q-Technique and its Methodology.

127



CHAPTER SEVEN:

Background

The argument for the use of instructional technology usually begins with a
comparative examination of the effectiveness of classroom instruction and
individual tutoring. For instance, the graph below illustrates the combined find-
ings of three dissertation studies that compared one-on-one tutoring with one-
on-many classroom instruction.*

It is not surprising that such comparisons would show that tutored students
learned more than those taught in classrooms. What is surprising is the magni-
tude of the difference. Overall, as the figure shows, it was two standard devia-

Individual Tutoring Compared to Classroom Instruction

Classroom
Students

# Students

Achievement

tions. This finding means, for example, that with instructional time held fairly
constant one-on-one tutoring raised the performance of 50th percentile students
to that of 98th percentile students. These, and similar empirical research find-
ings, suggest that differences between one-on-one tutoring and typical cl
room instruction are not only likely, but also very large.

Why then do we not provide these benefits to all students? The answer is
straightforward and obvious. With the exception of a few critical skills, such as
aircraft piloting and surgery, we cannot afford it. One-on-one tutoring has been
described as an educational imperative and an economic impossibility.*

* Benjamin S. Bloom, “The 2 Sigma Problem: The Search for Methods of Group Instruction as
Effective as One-to-One Tutoring.” The dissertation studies were performed under Bloom’s direc-

tion.
* M. Scriven, “Problems and Prospects for Individualization.”

88


CHAPTER FOUR

Inputs, Processes, and Outputs of the Intelligence Cycle

Policymaker and other Direction Data collection
stakeholder questions, requirements
requirements

Data collection Planning Task assignment,
requirements, assessment potential data sources,
of available resources and focus of analysis
capabilities

Open-source data: foreign | Collection Potentially relevant data

broadcasts, newspapers,
periodicals, books;
Classified data: case
officer, diplomatic, and
attaché reports,
electronics, satellite
photos

Potentially relevant data Processing: Reduction | Usable Data
of data in a variety of
formats to consistent
pieces of usable data
Usable data Analysis: Integration, Findings
evaluation, assessment
of reliability, validity,
and relevance of data

Analytic review Production: Peer Written briefs, studies,
review, supervisory long range assessments,
review short range assessments,

oral briefs, national
intelligence estimates
Written briefs, studies, Dissemination Appropriate product to
long-range assessments, address customer’s need
short-range assessments,
oral briefs, national
intelligence estimates

The table above depicts a more detailed input, process, and output analysis
and makes some relationships clearer—for example, the steps that include two
actions (planning and direction, analysis and production) have been separated
into distinct processes—but it sill leaves a number of questions unanswered. It
is difficult to see from this analysis specifically who is responsible for provid-
ing inputs, carrying out the processes, and producing outputs; and what
requirements are expected of the inputs and outputs.

An important issue that this analysis only partly clarifies is the role of ana-
lysts. Nor does it demonstrate how great a burden the process places on them,

48



CHAPTER THREE

edge and furthering the growth of individual disciplines. A taxonomy differ-
entiates domains by specifying the scope of inquiry, codifying naming
conventions, identifying areas of interest, helping to set research priorities,
and often leading to new theories. Taxonomies are signposts, indicating what
is known and what has yet to be discovered.

This chapter, to which more than 100 individuals contributed their time and
advice, proposes a taxonomy for the field of intelligence. It is my hope that the
resulting organized listing of variables will help practitioners strengthen their
understanding of the analytic process and point them in directions that need
additional attention.

Intelligence Analysis

We could have talked about the science of intelligence, but... the
science of intelligence is yet to be invented.

Charles Allen’

Developing an intelligence taxonomy is complicated by the fact that the lit-
erature in the field is episodic and reflects specialized areas of concern. Per-
haps it is best to begin with what appears to be a key distinction between
general analysis and intelligence analysis, that of solving a problem in the
public domain, and solving a problem in a private or secret domain.

Ronald Garst articulates two arguments that are used to support this distinc-
tion: intelligence analysis is more time sensitive than analysis in other
domains and it deals with information that intentionally may be deceptive.®
The notion that intelligence is uniquely time sensitive is questionable, how-
ever. Intelligence is not the only domain where time constraints can force
decisions to be made before data are complete. Time is always a key variable,
whether one is in an operating room or in a cockpit. To be sure, intelligence is
a life and death profession, but so are medicine and mass transportation. In
each instance, failure can mean casualties.

Garst’s point about intentional deception is more germane. With the possi-
ble exception of business and financial markets, analysts in other fields sel-
dom deal with intentional deception. As discussed in Chapter One, Michael
Warner makes a good case for secrecy being the primary variable distinguish-

4 See Benjamin S. Bloom, Taxonomy of Educational Objectives. Bloom’s taxonomy is a classifi-
cation of levels of intellectual behavior in learning, including knowledge, comprehension, appli-
cation, analysis, synthesis, and evaluation.

5 Comment made by the Associate Director of Central Intelligence for Collection at a public sem-
inar on intelligence at Harvard University, spring 2000.

© Ronald Garst, A Handbook of Intelligence Analysis.

34



BIBLIOGRAPHY

Thompson, J. R., R. Hopf-Weichel, and R. Geiselman. The Cognitive Bases of
Intelligence Analysis. Alexandria, VA: Army Research Institute, Research
Report 1362, 1984, AD-A146.

Tiemey, William G., and Robert A. Rhoads. Faculty Socialization as Cultural
Process: A Mirror of Institutional Commitment. Washington, DC: ASHE-
ERIC Higher Education Report No. 6, 1993.

Tobias, Sigmund. “When Do Instructional Methods Make a Difference?” Edu-
cational Researcher 11, no. 4 (1982): 4-9.

, and L. T. Frase. “Educational Psychology and Training” in S. Tobias
and J. D, Fletcher, eds. Training and Retraining: A Handbook for Business,
Industry, Government, and the Military. New York: Macmillan Library Ref-
erence, 2001.

Treverton, Gregory F. Reshaping National Intelligence in an Age of Informa-
tion. Cambridge: Cambridge University Press, 2001.

Trier, Jost. Der deutsche Wortschatz im Sinnbezirk des Verstandes. Heidel-
berg: Carl Winter, 1931.

Trochim, William. The Research Methods Knowledge Base. Cincinnati, OH:
Atomic Dog, 2001.

Tulving, E., and D.M. Thomson. “Encoding Specificity and Retrieval Pro-
cesses in Episodic Memory,” Psychological Review 80 (1973).

Tversky, Amos, and Daniel Kahneman. “The Belief in the ‘Law of Small
Numbers,” Psychological Bulletin 76 (1971): 105-10.

. “Judgment Under Uncertainty: Heuristics and Biases,” Science 185
(1974): 1124-31.

US Joint Forces Command. Department of Defense Dictionary of Military
and Associated Terms. USJFCOM Joint Publication 1-02. Amended through
June 5, 2003. (http://www.dtic.mil/doctrine/jel/doddict/index.hunl.)

von Humboldt, Wilhelm. “Uber die Verschiedenheit des Menschlichen
Sprachbaues” in B. Bohr, ed. Gesammelte Schriften. Berlin: Verlag, 1907.

Voss, James, and Timothy Post. “On the Solving of Ill-Structured Problems”
in Michelene Chi, Robert Glaser and Marshall Farr, eds. The Nature of
Expertise. Hillsdale, NJ: Lawrence Erlbaum Associates, 1988.

Vygotsky, Lev. Mind and Society, Cambridge: Harvard University Press,
1930.

Wanous, John P. Organizational Entry: Recruitment, Selection, Orientation,
and Socialization. Reading, MA: Addison-Wesley, 1992.

, etal. “The Effects of Met Expectations on Newcomer Attitudes and
Behavior: A Review and Meta-Analysis,” Journal of Applied Psychology 77
(1992): 288-97.

Warner, Michael. “Wanted: A Definition of ‘Intelligence,’” Studies in Intelli-
gence 46, no. 3 (2002):15-22.

155



COMBATING ETHNOCENTRISM

representative red team. There is the obvious problem of security. Someone
fully able to represent the adversary culturally would very likely be unable to
obtain requisite clearances for participation in a classified red team exercise.
In fact, even if it were possible to find someone both culturally representative
and sympathetic to the goals of the red team, such as the participant born in
the region, the conflicts triggered by that sympathy, cultural identity, and cul-
tural allegiance could well lead to unforeseen cognitive biases that would be
difficult to counteract.'°

An alternative is to find an ethnic American citizen with similarities to the
people of the region of interest, but simply finding a US citizen with the same
ethnicity as those of the region of interest does not guarantee any special
insight into their thinking. Ethnicity is not the same as sharing culture or iden-
tity. Not all ethnic groups in the US are isolated and self-perpetuating. Many,
in fact, put great effort into trying to assimilate into the larger “American cul-
ture” by distancing themselves from their culture of origin. These people often
struggle with their own concept of cultural identity and the broader issues of
community affiliation.!! Many immigrants and most first-generation offspring
have already begun the process of acculturation. More striking, their offspring
display a process of enculturation in the US by learning the language, attend-
ing the schools, assimilating local and national values, and establishing ties to
a diverse community outside of their own ethnic enclave. In fact, the children
of recent immigrants share many of the same cognitive filters as those who are
generations removed from migration. That said, there are American citizens
born in the region of interest, like the member of the red team in which I par-
ticipated, who do have insight into specific cultures, principally because their
enculturation was affected by being born in, and living in, a foreign region.

The participant in that red team was a foreign-born American citizen, but
foreign birth is not a necessary condition for enculturation.'? Living in a for-
eign region, speaking the language, interacting with the people, developing
community ties, and establishing an identity within that community are all
part of the acculturation process and allow one to alter the cognitive filters
through which one interprets the world. Time spent on a US military base, in a
US embassy, or in a Western hotel overseas does not lead to acculturation.

1” Philip Cushman, Constructing the Self, Constructing America; John Lucy, Language Diversity
and Thought, Douglass Price-Williams, Explorations in Cross-Cultural Psychology; Marshall
Segall, Cross-Cultural Psychology; Richard Shweder, Thinking Through Cultures; Yali Zou and
Enrique Trueba, Ethnic Identity and Power, and Benjamin Whorf.

1! David Levinson and Melvin Ember, American Immigrant Cultures. For raw data covering 186
cultural groups since 1937, including immigrants, see the Human Relations Area Files at Yale
University,

"2 Some anthropologists have argued that enculturation is specific to childhood but the evidence
supports that it is a lifelong process. See Segall.

83



CHAPTER ONE

During the interviews for this study, I soon found that the psychological
definition was insufficient. The psychological definition took into account the
cognitive and psychomotor components of task-structure, time-to-task, and
accuracy-of-task as measures of errors and error rates, but it did not fully take
into account the notion of surprise.’ Surprise is the occurrence of something
unexpected or unanticipated. It is not precisely commission or omission; it
indicates, rather, the absence of contravening cognitive processes. Measures
of accuracy may account for factual errors in the intelligence domain, but
measures of accuracy are insufficient to account for surprise events and intel-
ligence failure.

To put this in context, an analyst, while accounting successfully for an
adversary’s capability, may misjudge that adversary’s intention, not because
of what is cognitively available, but because of what is cognitively absent.
The failure to determine an adversary’s intention may simply be the result of
missing information or, just as likely, it may be the result of missing hypothe-
ses or mental models about an adversary’s potential behavior.

7 Sociological definitions are more akin to the definitions proposed in this study. Failure can occur
due to system complexity and missing data as well as through the accumulation of error. See
Charles Perrow, Normal Accidents. Living with High Risk Technologies. I'd like to thank Dr. Per-
row for his assistance with this work.



CHAPTER ONE

Definition 1: Intelligence is secret state or group activity to under-
stand or influence foreign or domestic entities.

The above definition of intelligence, as used in this text, is a slightly modi-
fied version of the one that appeared in Michael] Warner’s work in a recent
article in Studies In Intelligence.' Warner reviews and synthesizes a number
of previous attempts to define the discipline of intelligence and comes to the
conclusion that “Jntelligence is secret state activity to understand or influence
foreign entities.”

Warner’s synthesis seems to focus on strategic intelligence, but it is also
logically similar to actionable intelligence (both tactical and operational)
designed to influence the cognition or behavior of an adversary.’ This synthe-
sis captures most of the elements of actionable intelligence without being too
restrictive or too open-ended, and those I asked to define the word found its
elements, in one form or another, to be generally acceptable. The modified
version proposed here is based on Warner’s definition and the interview and
observation data collected among the law enforcement elements of the inteili-
gence agencies. These elements confront adversaries who are not nation states
or who may not be foreign entities. With this in mind, I chose to define intelli-
gence somewhat more broadly, to include nonstate actors and domestic intelli-
gence activities performed within the United States.

Definition 2: Intelligence analysis is the application of individual
and collective cognitive methods to weigh data and test hypotheses
within a secret socio-cultural context.

This meaning of intelligence analysis was harder to establish, and readers
will find a more comprehensive review in the following chapter on developing
an intelligence taxonomy. In short, the literature tends to divide intelligence
analysis into “how-to” tools and techniques or cognitive processes. This is not
to say that these items are mutually exclusive; many authors see the tools and
techniques of analysis as cognitive processes in themselves and are reluctant
to place them in different categories. Some authors tend to perceive intelli-
gence analysis as essentially an individual cognitive process or processes.?

My work during this study convinced me of the importance of making
explicit something that is not well described in the literature, namely, the very

' Michael Warner, “Wanted: A Definition of “Intelligence’,” Studies in Intelligence 46, no. 3
(2002): 15-22.

2 US Joint Forces Command, Department of Defense Dictionary of Military and Associated
Terms.

3 The appendix lists literature devoted to each of these areas.



CHAPTER SEVEN

Instructional Technology: Effectiveness and
Implications for the Intelligence Community

EE

J.D. Fletcher!
Rob Johnston

The Intelligence Community has begun to invest substantial resources in
the training and education of its analysts. With the exception of a few
advanced courses available through distance learning networks, this instruc-
tion is delivered using a conventional classroom model. This model possesses
a number of inherent inefficiencies, including inconsistent instruction, strict
ties to time and place of instruction, large student-to-instructor ratios, and lim-
ited active participation by students due to class size and scheduling.

Research suggests that significant improvements can be achieved through
the use of computer-based instructional technology. According to these stud-
ies, this technology can increase instructional effectiveness and reduce time
needed to learn. It can achieve these efficiencies, moreover, while both lower-
ing the cost of instruction and increasing its availability.? This chapter summa-
rizes evidence on the promise of instructional technology for intelligence
analysis training.

' Dr. J.D. Fletcher is a research staff member at the Institute for Defense Analyses, where he spe-
cializes in issues of manpower, personnel, and training. He holds graduate degrees in computer
science and educational psychology from Stanford University.

2 Because instructional technology makes few distinctions between formal education and profes-
sional training, the term “instruction” will be used for both in this chapter.

87



BIBLIOGRAPHY

. Technology, the Columbus Effect, and the Third Revolution in Learn-

ing. Alexandria, VA: Institute for Defense Analyses, 2001, D-2562.

, and Rob Johnston. “Effectiveness and Cost Benefits of Computer-

Based Decision Aids for Equipment Maintenance,” Computers in Human

Behavior 18 (2002): 717-28.

, and M. R. Rockway. “Computer-based Training in the Military” in J.
A. Ellis, ed. Military Contributions to Instructional Technology. New York:
Praeger Publishers, 1986.

Folker, Robert D. Intelligence Analysis in Theater Joint Intelligence Centers:
An Experiment in Applying Structured Methods. Washington, DC: Joint Mil-
itary Intelligence College, Occasional Paper #7, 2000.

Ford, Harold. Estimative Intelligence: The Purposes and Problems of
National Intelligence Estimating. Lanham, MD: University Press of Amer-
ica, 1993.

Freud, Sigmund. “A Philosophy of Life: Lecture 35,” New Introductory Lec-
tures on Psycho-Analysis. London: Hogarth Press, 1933.

Fries, J., et al. “Assessment of Radiologic Progression in Rheumatoid Arthri-
tis: A Randomized, Controlled Trial,” Arthritis Rheum 29, no. 1 (1986): 1-9.

Fuld, Leonard. The New Competitor Intelligence: The Complete Resource for
Finding, Analyzing, and Using Information About Your Competitors. New
York: John Wiley & Sons, 1995.

Galanter, E., ed. Automatic Teaching: The State of the Art. New York: John
Wiley & Sons, 1959.

Gall, Meredith, Walter Borg, and Joyce Gall. Educational Research, Sixth
Edition. White Plains, NY: Longman, 1996.

Garst, Ronald, ed. A Handbook of Intelligence Analysis, 2nd ed. Washington,
DC: Defense Intelligence College, 1989.

Geertz, Clifford. The Interpretation of Cultures. New York: Basic Books,
1973.

Giddens, Anthony. Modernity and Self-Identity. Stanford, CA: Stanford Uni-
versity Press, 1991.

Glaser, Barney, and Anselm Strauss. Discovery of Grounded Theory: Strate-
gies for Qualitative Research. Hawthorne, NY: Aldine de Gruyter, 1967.

. Theoretical Sensitivity: Advances in the Methodology of Grounded

Theory. Mill Valley, CA: Sociology Press, 1978.

. Basics of Grounded Theory Analysis: Emergence vs. Forcing. Mill
Valley, CA: Sociology Press, 1992.

Glass, Gene. “Primary, Secondary, and Meta-Analysis of Research.” Educa-
tional Researcher 5 (1976): 3-8.

, and Barry McGaw. “Choice of the Metric for Effect Size in Meta-

Analysis.” American Educational Research Journal 17 (1980): 325-7.

, Barry McGaw, and Mary Lee Smith. Meta-Analysis in Social

Research. New York: Sage, 1981.

144



BIBLIOGRAPHY

Jervis, Robert. Perception and Misperception in International Politics. Prince-
ton, NJ: Princeton University Press, 1977.

Johnson, David, and Roger Johnson. “The Internal Dynamics of Cooperative
Learning Groups” in Richard Slavin, et al., eds. Learning to Cooperate,
Cooperating to Learn. New York: Plenum, 1985.

, etal. “Effects of Cooperative, Competitive, and Individualistic Goal

Structure on Achievement: A Meta-Analysis,” Psychological Bulletin 89,

no. 1 (1981): 47-62.

, et al. “The Impact of Positive Goal and Resource Interdependence on
Achievement, Interaction, and Attitudes,” Journal of General Psychology
118, no. 4 (1996): 341-47.

Johnson, Edgar. Effects of Data Source Reliability on Inference. Alexandria,
VA: Army Research Institute for the Behavioral and Social Sciences, Tech-
nical Paper #251, 1974,

Johnston, Rob. Decision Making and Performance Error in Teams: Research
Results. Arlington, VA: Defense Advanced Research Projects Agency, 1997.

. “The Effectiveness of Instructional Technology,” Proceedings of the

Virtual Reality in Medicine and Developers’ Exposition. Cambridge, MA:

VRS, 1995.

. “Electronic Performance Support Systems and Information Naviga-

tion,” Thread 2, no, 2 (1994): 5-7.

, and Sunil Bhoyrul. “A Preliminary Report of a Task Analysis of Lap-

aroscopic Cholecystectomy Used to Assess the Effectiveness of a Virtual

Reality Surgical Simulator,” Journal of Virtual Reality in Medicine 2, no. |

(1996): 16-19.

, et al. “Assessing a Virtual Reality Surgical Skills Simulator” in

Suzanne Weghorst, Hans Sieburg, and Karen Morgan, eds. Health Care in

the Information Age: Future Tools for Transforming Medicine. Washington,

DC: IOS Press, 1996.

, and J. Dexter Fletcher. A Meta-Analysis of the Effectiveness of Com-

puter-Based Training for Military Instruction. Alexandria, VA: Institute for

Defense Analyses, 1998.

, J. Dexter Fletcher, and Sunil Bhoyrul. The Use of Virtual Reality to
Measure Surgical Skill Levels. Alexandria, VA: Institute for Defense Analy-
ses, 1998.

Jones, Morgan. The Thinker’s Toolkit: 14 Powerful Techniques for Problem
Solving. New York: Times Business, 1998.

Jung, Carl. The Undiscovered Self. Translated by R.F.C. Hull. Princeton, NJ:
Princeton University Press, 1990.

Kahaner, Larry. Competitive Intelligence. How to Gather, Analyze, and Use
Information to Move Your Business to the Top. New York: Touchstone,
Simon and Schuster, 1996.

147



BIBLIOGRAPHY

Miller, George, and Philip Johnson-Laird. Language and Perception. Cam-
bridge: Harvard University Press, 1976.

Mills, Theodore. “Power Relations in Three-Person Groups” in Dorwin Cart-
wright and Alvin Zander, Group Dynamics: Research and Theory. New
York: Harper & Row, 1960.

Min Qi, Zhongguo Zhengzhi Wenhua [Chinese Political Culture]. Kunming,
China: Yunnan People’s Press, 1989.

Minsky, Marvin, and Seymour Papert. Artificial Intelligence. Eugene: Oregon
State System of Higher Education, 1974.

Moln, Linda. “Linking Power Structure and Power Use” in Karen Cook, ed.
Social Exchange Theory. Newbury Park, CA: Sage, 1987.

Moore, David. “Creating Intelligence: Evidence and Inference in the Analysis
Process.” Unpublished Masters Thesis. Washington, DC: Joint Military
Intelligence College, 2002.

Morris, C.D., J.D. Bransford, and J.J. Franks. “Level of Processing Versus
Transfer-Appropriate Processing,” Journal of Verbal Learning and Verbal
Behavior 16 (1977).

Morrison, J.E., and J.D. Fletcher. Cognitive Readiness. Alexandria, VA: Insti-
tute for Defense Analyses, 2001, IDA Paper P-3735.

, and Larry Meliza. Foundations of the After-Action Review Process.
Alexandria, VA: Institute for Defense Analyses, 1999.

Mullen, Brian, and Carolyn Copper. “The Relation Between Group Cohesive-
ness and Performance: An Integration,” Psychological Bulletin, 115 (1994):
210-27.

National Aeronautics and Space Administration. Astronaut Fact Book. NP-
2000-09-008JSC. Washington, DC: NASA, 2000.

Neisser, U. Cognitive Psychology. New York: Appleton, Century Crofts, 1967.

Nieva, V., E. Fleishman, and A. Rieck. Team Dimensions: Their Identity,
Their Measurement, and Their Relationships, RN 85-12. Alexandria, VA:
US Army Research Institute for the Behavioral and Social Sciences, 1985.

Nijhof, W., and P. Kommers. “An Analysis of Cooperation in Relation to Cog-
nitive Controversy” in Richard Slavin, Shlomo Sharan, Spencer Kagan, et
al., eds. Learning to Cooperate, Cooperating to Learn. New York: Plenum,
1985.

Nisbett, Richard, and Ara Norenzayan. “Culture and Cognition” in Douglas
Medin and Hal Pashler, eds. Stevens’ Handbook of Experimental Psychol-
ogy, 3rd ed., New York: John Wiley & Sons, 2002.

Nolte, William. “Preserving Central Intelligence: Assessment and Evaluation
in Support of the DCI, ” Studies in Intelligence 48, no. 3 (2005): 21-25.

Orasanu, Judith. “Shared Mental Models and Crew Performance.” Paper pre-
sented at the 34th annual meeting of the Human Factors Society, Orlando,
FL, 1990.

151



CHAPTER TWO

Finding: Rewards and Incentives

The shift in the analytic production cycle is not only reflected in the prod-
ucts and processes but also in the way analysts perceive the system by which
intelligence organizations reward and promote employees. Employees see
their opportunities for promotion as being tied directly to the number of daily
products they generate and the amount of social capital or direct consumer
influence they amass, most often when their work is recognized by senior pol-
icymakers.”

In any given week, I could devote about 20 percent of my time to
longer think pieces, but why should I? You can write all the think
pieces you want, but, if you don’t write for the daily briefs, you
aren't going to move into management. These days the only thing
that matters is getting to the customers.

If l write a 12-page self-directed piece that goes out as a community
product, and somebody else writes one paragraph with two bullet
points that goes into a daily brief, the guy who got in the daily brief
is going to get the recognition. Why waste my time with the big
products?

It isn't really official policy, but the reality is that sheer production
equals promotion. People talk about quality, but, in the end, the
only measurable thing is quantity.

Our group has a “team award” of 5,000 bucks. Last year, they gave
it to the one guy who published the most. I’m not sure how that one
guy won a “team award,” but there you go.

Technically, I have four bosses. The only thing that seems to keep
them all happy is volume. It’s like piece work.

Quality? How do you measure quality? Quantity—now that’s some-
thing you can count.

Promotion is based on production—pure and simple.

In sum, aside from specific tactical groups, staff positions that generate lim-
ited social capital, and individual cognitive differences, there is a majority
sentiment among the analysts interviewed that the combination of a shorter

7 Social capital refers to the set of norms, social networks, and organizations through which peo-
ple gain access to power, resources, and reciprocity and through which decisionmaking and pol-
icy creation occur. In other words, whom you know is just as important as what you know. Pierre
Bourdieu, “The Forms of Capital”; Robert Putnam, “The Prosperous Community” and Bowling
Alone. See also the empirical work on social capital summarized in Tine Feldman and Susan
Assaf, Social Capital: Conceptual Frameworks and Empirical Evidence.



BIBLIOGRAPHY

Web resources

Buros Institute of Mental Measurements (http://www.unl.edu/buros/).

Central Intelligence Agency main website (http://www.cia.gov).

Central Intelligence Agency Office of General Council (http://www.cia.gov/
oge/best.htm).

Central Intelligence Agency Officer in Residence Program (http://
www.cia.gov//csi/officer.html).

Defense Intelligence Agency (http://www.dia.mil).

Federal Bureau of Investigation (http://www.fbijobs.com or http://
www.fbi.gov).

Human Relations Area Files. Yale University. (http://www. yale.edu/hraf/).

National Aeronautics and Space Administration-Johnson Space Center career
astronaut biography. (http://www. jsc.nasa.gov/Bios/astrobio
_activemgmt.html).

National Imagery and Mapping Agency (http://www.nima.mil/).

National Reconnaissance Office (http://www.nro.gov/).

National Security Agency (http://www.nsa.gov/).

US Air Force (http://aia.lackland.af.mil/aia/).

US Army (http://usaic.hua.army.mil/index.htm).

U.S. Army Center for Army Lessons Learned (http://call.army.mil/).

US Coast Guard (http://www.uscg.mil/USCGshtm).

US Department of Energy (http://www.energy.gov/engine/content.do).

US Department of Homeland Security (http://www.dhs.gov/dhspublic/).

US Department of State (http://www.state.gov/s/int/).

US Department of the Treasury (http://www.treasury.gov/).

US Intelligence Community (http://www. intelligence.gov/index.shtml).

US Marine Corps (http://www.usme.mil/).

US Navy (http://www.nmic.navy.mil/).

157



CHAPTER EIGHT

Organizational Culture: Anticipatory Socialization and
Intelligence Analysts

———

Stephen H. Konya!
Rob Johnston

I know it sounds silly, but I had this image of James Bond before I
started working here. The truth is, I just sit in a cubicle, and I write
reports.

Every organization has a unique culture that is defined partly by its individ-
ual members and partly by its structure, history, and policies. For that culture
to endure, it must be transmitted from current members to new members. This
process, known as organizational socialization, is especially important in
organizations with strong, insular cultures, as those with weak cultures have
less to transmit and will tend to experience culture changes as members come
and go.

Although socialization begins prior to a person’s first day on the job and is
a continuous process, it is experienced most intensely by new employees. The
cultural symbols acquired and interpreted during their initial interaction with
the institution create potent and lasting impressions.” For them, socialization

‘ Stephen Konya is a Research Associate at the Institute for Defense Analyses, currently examin-
ing multimodal interfaces for the dismounted for the DARPA/Army Future Combat Systems
program. He holds an MS in industrial and organizational psychology from Rensselaer Polytech-
nic Institute.

2 Umberto Eco, A Theory of Semiotics; Clifford Geertz, The Interpretation of Cultures; Jacques
Lacan, Eerits; Ferdinand de Saussure, Course in General Linguistics.

97





AFTERWORD
Joseph Hayes!

EEE

The unexamined life is not worth living
Socrates.

A very popular error: having the courage of one’s convictions;
rather, it is a matter of having the courage for an attack on one’s
convictions!

Nietzsche

Rob Johnston has written a superb book, a study of intelligence as it is actu-
ally practiced. Rob’s book is alive with specific, practical recommendations
about how the practice of intelligence could be made better. The literature of
intelligence is overwhelmingly devoted either to studies which, however rig-
orous in their academic structure, fail to convey the humanness of the enter-
prise or to books and articles, too often self congratulatory or self promoting,
which are little more than assemblages of entertaining anecdotes. Rob’s study
deserves a place of honor on the very small bookshelf reserved for analytically
sound, deeply insightful works on the conduct of intelligence. Any serious
discussion of reform or significant change in the ways in which US intelli-
gence is organized, structured, and carried out will need to take this book as a
starting point.

Rob’s work bears witness to the imagination and commitment to excellence
on the part of the senior intelligence officials who made it possible for a cultural
anthropologist to carry out his field work in the secret, sometimes hermetically

1 Joseph Hayes is a retired senior officer of the Central Intelligence Agency Directorate of Opera-
tions. He served more than 30 years in the clandestine service.

159



(CHAPTER THREE

‘Systematic Variables The Systemic Variables category also focuses on
User Requirements environmental variables. These include such exter-
Operations nal influences on the organization as consumer

Information Acquisition | needs and requirements, time limitations, and meth-
Collection Methods a : jon: .,
ods for using the information; and the consumer’s

even organization, political constraints, and security
Information Reliability | issues. The works of Betts, Hulnick, Hunt, Kam,
Reproducible and Laqueur address the environmental and con-
Consistent sumer issues that affect intelligence analysis.%°

Information Validity
Historical
Single Source

Case studies that touch on various systemic v:
ables include: Allison, on the Cuban missile crisis;

Dual Source Betts, on surprise attacks; Kirkpatrick, on World
Triangulation War II tactical intelligence operations; Shiels, on
Information Archive government failures; Wirtz, on the Tet offensive in
Siorage Vietnam; and Wohlstetter, on Pearl Harbor.*!
Correlation
Retrieval
Analytical Methodology Systematic Variables
Approach
Intuitive The Systematic Variables are those that affect
Structured the process of analysis itself. They include the
Semi-structured user’s specific requirements, how the information

Information Processing
Historical Information
Current Information

was acquired, the information’s reliability and
validity, how the information is stored, the pre-

Decision Strategies scribed methods for analyzing and processing the
Estimative information, specific strategies for making deci-
R Predictive sions about the information, and the methods used
eporting . .
Verbal Methods to report the information to consumers.
Written Methods A number of authors have written about the

analytic tools and techniques used in intelligence,
among them Clauser and Weir, on intelligence research methods; Jones, on
analytic techniques; and Heuer, on alternative competing hypotheses.

» Graham T. Allison, Essence of Decision; Bruce D. Berkowitz and Allan E. Goodman, Best Truth;
Dan Elkins, An Intelligence Resource Manager's Guide; Harold Ford, Estimative Intelligence: Roy
Godson, Comparing Foreign Intelligence; Jeffrey Richelson, The U.S. Intelligence Community.
% Richard K. Betts, “Policy-makers and Intelligence Analysts: Love, Hate or Indifference”; Arthur
S. Hulnick, “The Intelligence Producer-Policy Consumer Linkage: A Theoretical Approach”; David
Hunt, Complexity and Planning in the 21st Century; Kam, Surprise Attack; Walter A. Laqueur, The
Uses and Limits of Intelligence.

» Allison; Richard K. Betts, Surprise Attack; Lyman B. Kirkpatrick, Jr, Captains Without Eyes:
Intelligence Failures in World War It; Frederick L. Shiels, Preventable Disasters: Why Governments
Fail; James J. Wirtz, The Tet Offensive: Intelligence Failure in War, Roberta Wohlstetter, Pear! Har-
bor: Warning and Decision.

40



CHAPTER SEVEN

includes the ability to devise plans and actions that differ from and.
improve upon “school solutions.” Capabilities that widen the range of
options can be taught, but higher levels of creativity are more likely to be
inborn than trained.

The above review suggests, first, that the creative processes needed by ana-
lysts can, to some extent, be broken down into components, and second, that
these components can, again to some extent, be taught. Instructional technol-
ogy can now substantially aid analysts in acquiring the facts, concepts, and
procedures needed to perform their craft. However, it must become increas-
ingly “intelligent” if it is to compress the years of experience analysts now
need to become proficient and help them more rapidly acquire the advanced
cognitive capabilities—those higher in Bloom’s hierarchy—that they also
need. To do this successfully, instruction must be tailored to the specific back-
ground, abilities, goals, and interests of the individual student or user. Instruc-
tional technology must provide what has been called “articulate expertise.”
Not only must it supply helpful and relevant guidance in these more advanced
levels of knowledge, skills, and abilities, it must do so in a way that learners
and users with varying levels of knowledge and skill can understand.

Discussion
At this point, it may be worth reviewing the capabilities provided by “non-
intelligent” instructional technology since the 1950s. It has been able to:3?
* accommodate the rate of progress of individual students, allowing as
much or as little time as each needs to reach instructional objectives;

* tailor both the content and the sequence of instructional content to each
student’s needs;

+ make the instruction easy or difficult, specific or abstract, applied or theo-
retical as necessary;

+ adjust to students’ most efficient learning styles (collaborative or individ-
ual, verbal or visual, etc.).

Intelligent tutoring systems are a different matter. They require quite spe-
cific capabilities that were first targeted in the 1960s.** Two key capabilities
are that intelligent tutoring systems must:

® E. Galanter, Automatic Teaching; R. C. Atkinson and H. A. Wilson, Computer-Assisted
Instruction; P. Suppes and M. Morningstar, Computer-assisted Instruction at Stanford 1966-68; J
D. Fletcher and M. R. Rockway, “Computer-based Training in the Military.”

8 J. S. Brown, R. R. Burton, and J, DeKleer, “Pedagogical, Natural Language and Knowledge
Engineering in SOPHIE I, Il, and III.”

94





CHAPTER TWO

improvement infrastructures and that the consequences of intelligence error
and failure are disproportionately high in comparison with other domains.

It is evident from the literature that intelligence organizations recognize the
need to improve their performance and that it is possible to make the domain
of intelligence analysis into a coherent scientific discipline. The first step in
this transition is to identify and describe performance gaps.* Once gaps have
been identified, it will be possible to introduce performance improvement
methods systematically and to measure the effectiveness of the results. This
work is intended to further research toward creating intelligence organizations
that are more effective.

The Problem of Bias

Although a researcher might pretend to be neutral and unbiased in present-
ing his findings and conclusions, personal biases can creep into a finished
product. The methods ethnographers employ to collect raw data and the use of
interpretational analysis to extract meaning and generate theory virtually guar-
antee it. In my view, one should be candid about this possibility. I noted in
Chapter One that ethnographers bring a certain amount of experiential bag-
gage to their work, myself included. At this point, before discussing analytical
difficulties and problems I identified during my research, I want to make the
readers aware of an additional personal bias that has developed from observ-
ing the Intelligence Community.

During my research, I developed a great deal of empathy for individual ana-
lysts and the problems they face in trying to perform their jobs. The reason for
this is straightforward and something every anthropologist recognizes. It is
part of the process that anthropologists reach a point where they can modify
their own identity in order to gain insight into a different culture. The risk is
that empathy and identity modification will induce the researcher to “go
native” and produce bias in his findings.

Although I may empathize with analysts personally, it is critical for theory
development to avoid parroting the views, kudos, or complaints of individual
analysts, who may or may not be dissatisfied with their unique professional
experience. In order to counteract the empathy bias, I employed multiple data
collection techniques and then used those data to refute or confirm each cate-
gorical finding. Triangulation is not an infallible system, however, and the
reader is advised to approach these findings with both a critical eye and the

» Performance gaps are the difference or distance between ideal (perfect) organizational perfor-
mance and actual organizational performance. In this case, ideal performance includes complete
data sets, reportorial accuracy, and the ability to avoid strategic, operational, and tactical surprise.



CHAPTER SIX

have different histories, languages, customs, educational practices, and cul-
tures and, therefore, must think differently from one another.

The problem, however, is that the cognitive process of understanding or
even recognizing that there are cultural and cognitive differences is not intui-
tive at all. Intuition is the act of immediate cognition, that is, perceiving some-
thing directly through the use of culturally dependent heuristics and cognitive
patterns accumulated through a lifetime without requiring the use of rational
or formal processes. This effort appears doomed to failure, because “trying to
think like them” all too often results in applying the logic of one’s own culture
and experience to try to understand the actions of others, without knowing that
one is using the logic of one’s own culture. This, however, does not have to be
the case. Through acculturation and the use of specific strategies, tools, and
techniques, it is possible to combat the effects of ethnocentrism without trying
to “think like them.” This text includes two short case studies on failures to
recognize ethnocentrism, both drawn from the author’s own experience and
told from his perspective. These failures are then examined with the goal of
developing strategies and techniques to combat ethnocentric bias.

Case Study One: Tiananmen Square

At the time of the prodemocracy protests of the Chinese students and, to a
lesser extent, workers, between April and June of 1989, I too was a college
student. I mention this because American college students and Chinese col-
lege students tend to perceive themselves in very different ways, and they are
perceived by their societies as having very different social roles. Chinese stu-
dents perceive themselves as having moral authority, and they are perceived
as controlling social capital and possessing public status. There is a cultural
norm in China that students, as the future elite, have a morally superior role in
society. I remember thinking at the time that, with the obvious exception of
those in power, who risked losing their privileged positions, any “right-
minded” person in China would support democracy. A movement for demo-
cratic reform would liberalize the policies of a repressive regime, encourage
personal freedom, and give the Chinese people a voice in their lives.

When the university students went on strike and took over Tiananmen
Square, the popular view in the United States, reflected in the US media, was
that they were college students protesting for democratic reform. There were
images of thousands of students rallying and camping out on and around the
statue of the People’s Heroes. Throughout the square, banners and posters
from universities supported democracy and freedom. The statue of the God-
dess of Democracy erected by the demonstrators looked very much like our
Statue of Liberty. Labor groups offered to join the students, people paraded in
front of the Great Hall of the People, and citizens donated blankets and food.

16



FOREWORD
Gregory F. Treverton

a

It is a rare season when the intelligence story in the news concerns intelli-
gence analysis, not secret operations abroad. The United States is having such
a season as it debates whether intelligence failed in the run-up to both Septem-
ber 11 and the second Iraq war, and so Rob Johnston’s wonderful book is per-
fectly timed to provide the back-story to those headlines. The CIA’s Center
for the Study of Intelligence is to be commended for having the good sense to
find Johnston and the courage to support his work, even though his conclu-
sions are not what many in the world of intelligence analysis would like to
hear.

He reaches those conclusions through the careful procedures of an anthro-
pologist—conducting literally hundreds of interviews and observing and par-
ticipating in dozens of work groups in intelligence analysis—and so they
cannot easily be dismissed as mere opinion, still less as the bitter mutterings
of those who have lost out in the bureaucratic wars. His findings constitute not
just a strong indictment of the way American intelligence performs analysis,
but also, and happily, a guide for how to do better.

Johnston finds no baseline standard analytic method. Instead, the most com-
mon practice is to conduct limited brainstorming on the basis of previous analy-
sis, thus producing a bias toward confirming earlier views. The validating of
data is questionable—for instance, the Directorate of Operation’s (DO) “clean-
ing” of spy reports doesn’t permit testing of their validity—reinforcing the ten-
dency to look for data that confirms, not refutes, prevailing hypotheses. The
process is risk averse, with considerable managerial conservatism. There is
much more emphasis on avoiding error than on imagining surprises. The ana-
lytic process is driven by current intelligence, especially the CIA’s crown jewel
analytic product, the President's Daily Brief (PDB), which might be caricatured

xi



Part III

Potential Areas for Improvement

a

59



INTEGRATING METHODOLOGISTS.

good as, or better than, an expert at making calls about the future. In fact, the
expert does not tend to outperform the actuarial table, even if given more spe-
cific case information than is available to the statistical model.'*

There are few exceptions to these research findings, but these are informa-
tive. When experts are given the results of the Bayesian probabilities, for
example, they tend to score as well as the statistical model if they use the sta-
tistical information in making their own predictions.'? In addition, if experts
have privileged information that is not reflected in the statistical table, they
will actually perform better than does the table. A classic example is the case
of the judge’s broken leg. Judge X has gone to the theater every Friday night
for the past 10 years. Based on a Bayesian analysis, one would predict, with
some certainty, that this Friday night would be no different. An expert knows,
however, that the judge broke her leg Thursday afternoon and is expected to
be in the hospital until Saturday. Knowing this key variable allows the expert
to predict that the judge will not attend the theater this Friday.

Although having a single variable as the determining factor makes this case
easy to grasp, analysis is seldom, if ever, this simple. Forecasting is a com-
plex, interdisciplinary, dynamic, and multivariate task wherein many variables
interact, weight and value change, and other variables are introduced or omit-
ted.

During the past 30 years, researchers have categorized, experimented, and
theorized about the cognitive aspects of forecasting and have sought to
explain why experts are less accurate forecasters than statistical models.
Despite such efforts, the literature shows little consensus regarding the causes
or manifestations of human bias. Some have argued that experts, like all
humans, are inconsistent when using mental models to make predictions. That
is, the model an expert uses for predicting X in one month is different from the
model used for predicting X in a later month, although precisely the same case
and same data set are used in both instances.° A number of researchers point

7 R. Dawes, D. Faust, and P. Meehl, “Clinical Versus Actuarial Judgment”; W. Grove and P.
Meehl, “Comparative Efficiency of Informal (Subjective, Impressionistic) and Formal (Mechani-
cal, Algorithmic) Prediction Procedures.”

18 R. Dawes, “A Case Study of Graduate Admissions”; Grove and Meehl; H. Sacks, “Promises,
Performance, and Principles”; T. Sarbin, “A Contribution to the Study of Actuarial and Individual
Methods of Prediction”; J. Sawyer, “Measurement and Prediction, Clinical and Statistical”; W.
Schofield and J. Garrard, “Longitudinal Study of Medical Students Selected for Admission to
Medical School by Actuarial and Committee Methor
 L. Goldberg, “Simple Models or Simple Processes?”; L. Goldberg, “Man versus Model of
Man”; D. Leli and S. Filskov, “Clinical-Actuarial Detection of and Description of Brain Impair-
ment with the Wechsler-Bellevue Form I.”

20 J, Fries, et al., “Assessment of Radiologic Progression in Rheumatoid Arthritis.”

65



CONTENTS
Foreword by Gregory F. Treverton ...

Introduction

Background .
Scope ....
A Work in Progress

Part I: Research Findings

Chapter One: Definitions ....

Chapter Two: Findings..

The Problem of Bias
Finding: Secrecy Versus Efficacy .
Finding: Time Constraints .....
Finding: Focus on Current Production .
Finding: Rewards and Incentives
Finding: “Tradecraft” Versus Scientific Methodology
Finding: Confirmation Bias, Norms, and Taboos
Finding: Analytic Identity
Finding: Analytic Training .

Part I: Ethnography of Analysis

Chapter Three: A Taxonomy of Intelligence Variables ......

Intelligence Analysis
Developing the Taxonomy
Systemic Variables ......

Systematic Variables
Idiosyncratic Variables
Communicative Variables
Conclusion ..

Chapter Four: Testing the Intelligence Cycle
Through Systems Modeling and Simulation

The Traditional Intelligence Cycle
Systematic Analysis .....
Findings Based on Systematic Analysis ..
Systemic Analysis ....

vii



INSTRUCTIONAL TECHNOLOGY

enhance the former. Instruction in developing mental abstractions will
enhance the latter.

* Metacognition—the executive functions of thought, more specifically,
those needed to monitor, assess, and regulate one’s own cognitive pro-
cesses.25 Meta-cognitive skills can be enhanced by exercises designed to
increase awareness of self-regulatory processes.”

* Pattern Recognition—the ability to distinguish the familiar from the unfa-
miliar. It may be accomplished by “template matching,” which involves
comparing retained images with incoming sensory impressions; or by
“feature comparison,” which involves recognizing and generalizing from
distinctive features of a structure held in memory with incoming sensory
impressions?’ Pattern recognition can be taught through a combination of
extensive practice, with feedback, and instruction in forming abstractions.

* Automaticity—processes that require only limited conscious attention.”
Automaticity can be taught by providing extensive practice, with feedback.

* Problem Solving—the ability to analyze a situation and identify a goal or
goals that flow from it, identify tasks and subtasks leading to the goal,
develop a plan to achieve them, and apply the resources needed to carry out
the plan. Practice, with feedback, and overlearning can enhance problem-
solving ability in many tasks. Techniques for problem solving can be suc-
cessfully taught, as can the knowledge base needed to implement them.”

* Decisionmaking—a component of problem solving, but the emphasis in
decisionmaking is on recognizing learned patterns, reviewing courses of
action, assessing their impact, selecting one, and allocating resources to
it.° Instruction in assessing courses of action has been shown to improve
decisionmaking, but some aspects of successful decisionmaking are more
likely to be inborn than trained.

* Mental Flexibility and Creativity—the ability to generate and modify
courses of action rapidly in response to changing circumstances.*! It

2G Salomon and D. N. Perkins “Rocky Roads to Transfer: Rethinking Mechanisms of a
Neglected Phenomenon.”

25 J. H. Flavell, “Metacognitive Aspects of Problem Solving.”

2 D, J. Hacker, Metacognition: Definitions and Empirical Foundations {On-line Report].

” M.H. Ashcraft, Fundamentals of Cognition.

2 R. M. Shiffrin and W. Schneider, W. “Controlled and Automatic Human Information Process-
ing: II. Perceptual Learning.”

» JR. Hayes, The Complete Problem Solver.

» P. Slovic, S. Lichtenstein, and B. Fischoff, “Decision-making.”

MD, Klahr, & H. A. Simon, “What Have Psychologists (and Others) Discovered About the Pro-
cess of Scientific Discovery?”

93



INTEGRATING METHODOLOGISTS

ing chess. On average, a chess master acquires 50,000 different chess pat-
terns.3

Similarly, a diagnostic radiologist spends eight years in full-time medical
training - four years of medical school and four years of residency—before
being qualified to take a national board exam and begin independent practice.*
According to a 1988 study, the average diagnostic radiology resident sees 40
cases per day, or around 12,000 cases per year. At the end of a residency, a
diagnostic radiologist has acquired an average of 48,000 cases.

Psychologists and cognitive scientists agree that the time it takes to become
an expert depends on the complexity of the task and the number of cases, or
patterns, to which an individual is exposed. The more complex the task, the
longer it takes to build expertise, or, more accurately, the longer it takes to
experience a large number of cases or patterns.

The Power of Expertise

Experts are individuals with specialized knowledge suited to perform the
specific tasks for which they are trained, but that expertise does not necessar-
ily transfer to other domains.* A master chess player cannot apply chess
expertise in a game of poker; although both chess and poker are games, a
chess master who has never played poker is a novice poker player. Similarly, a
biochemist is not qualified to perform neurosurgery, even though both bio-
chemists and neurosurgeons study human physiology. In other words, the
more complex a task, the more specialized and exclusive is the knowledge
required to perform that task.

Experts perceive meaningful patterns in their domains better than do non-
experts. Where a novice perceives random or disconnected data points, an
expert connects regular patterns within and among cases. This ability to iden-
tify patterns is not an innate perceptual skill; rather, it reflects the organization
of knowledge after exposure to and experience with thousands of cases.”

Experts have a deeper understanding of their domains than do novices, and
they utilize higher-order principles to solve problems.* A novice, for example,
might group objects together by color or size, whereas an expert would group

+ W. Chase and H. Simon, “Perception in Chess.”
+ American College of Radiology. Personal communication, 2002.

5 A. Lesgold et al., “Expertise in a Complex Skill: Diagnosing X-Ray Pictures.”

®M. Minsky and S. Papert, Artificial Intelligence; J. Voss and T. Post, “On the Solving of Ill-
Structured Problems.”

70. Akin, Models of Architectural Knowledge; D. Egan and B. Schwartz. “Chunking in Recall of
Symbolic Drawings”; K. McKeithen et al., “Knowledge Organization and Skill Differences in
Computer Programmers.



INSTRUCTIONAL TECHNOLOGY

The success of one-on-one tutoring may be explained by two factors. First,
measured in terms of questions asked and answered, tutors and their students
engage in many more instructional interactions per unit of time than is possi-
ble in a classroom. Second, one-on-one tutoring can overcome the substantial
spread of ability, measured by the time needed to reach minimal proficiency,
that is found in practically every classroom. Tutoring reduces time-to-learn by
adapting each interaction to the needs of each student. Less time is spent on
material the student has already learned, and more time is spent on material
remaining to be mastered.

To investigate the intensity of instructional interactions, Art Graesser and
Natalie Person compared questioning and answering in classrooms with those
in tutorial settings.’ They found that classroom groups of students ask about
three questions an hour and that any single student in a classroom asks about
0.11 questions per hour. In contrast, they found that students in individual tuto-
tial sessions asked 20-30 questions an hour and were required to answer 117—
146 questions per hour. Reviews of the intensity of interaction that occurs in
technology-based instruction have found even more active student response
levels.

Differences in the time needed by individuals in any classroom to meet
instructional objectives are also substantial. Studies on this issue have reported
ratios varying from 1:3 to 1:7 in the times the fastest learners need to learn com-
pared to the times needed by the slowest learners. Although these differences
may be due initially to ability, these studies suggest that such ability is quickly
overtaken by prior knowledge of the subject matter.’ This effect is particularly
evident in instruction for post-secondary-school students, because prior knowl-
edge rapidly increases with age and experience. Technology-based instruction
has long been recognized for its ability to adjust the pace of instruction to indi-
vidual needs, advancing through instructional material as quickly or as slowly as
required. The overall result has been substantial savings in the time required to
meet given instructional objectives.®

It should be emphasized that these benefits are not achieved at the expense
of instructional quality. Research has found that many instructional technolo-
gies have a positive impact on learning across a wide variety of student popu-
lations, settings, and instructional subject matters.?

This research suggests that technology-based instruction results in substan-
tial savings of time and money. Studies have shown that the times saved aver-

5 Art Graesser and Natalie Person, “Question-Asking During Tutoring.”
6 J.D. Fletcher, Technology, the Columbus Effect, and the Third Revolution in Learning.
7 Sigmund Tobias, “When Do Instructional Methods Make a Difference?”

* J.D. Fletcher, “Evidence for Learning From Technology-Assisted Instruction.”

° Ken Spencer, “Modes, Media and Methods: The Search for Educational Effectiveness.”

89





;
i
é
i
Hl
g
:
i

if assertion that war is too serious a matter to be left to military
men is generally ascribed to France’s premier during World War |,
Georges Clemenceau. | have heard it said only in jest that intelligence
is too serious a matter to be left to intelligence professionals, but
there may well be something serious in the thought. At the Center

for the Study of Intelligence, we think increasingly that observers from
other disciplines will have something to say about the practice of our
profession and that it is important to reach out to them. If we are
successful, | might add, readers of CSI products are likely to see

more work that is not aimed exclusively at an intelligence audience.

It is in this spirit of outreach that CSI presents this study by Dr. Rob
Johnston, an anthropologist who writes onthe basis of an intimate
knowledge of the world of intelligence. An example of the interdisci-
plinary give-and-take that we hope to encourage, Rob's ethnographic
approach to the study of the world of intelligence has yielded a number

of useful insights that intelligence careerists will find well worth their study.

Paul Johnson

Director

GOR.

<a



CHAPTER THREE

Davis, Goldgeier, and Heuer, among others.3> Decisionmaking and problem-
solving have been studied since the early 1920s, and these topics are reflected in
Heuer’s work as well.** Personality-profiling, too,
Communicative Variables | is well understood and has had an impact on recent

Formal intelligence practices and theory.*”
Inter-organization
Hierarchical Other well-researched areas, however, have yet
Inter-division to be studied in the context of intelligence. Accul-
Inter-group turation and enculturation, educational factors,
Intra-organization and training strategies, for example, may yet yield
Hierarchical interesting results and insights into the field of
Intra-division interesting i 8
Intra-group intelligence.
Individual
Hierarchical
Inter-division Communicative Variables
Intra-group
Informal The fourth category contains variables that
Inter organization affect interaction within and among groups.
Interalvision Because communication is the vital link within
Inter-group the system—among processes and among individ-
Intra-organization uals—this group of variables logically could be
Hierarchical included in each of the other three categories. Its
ne ision broad relevance, however, makes it seem reason-
Individest able to isolate it as a distinct area of variability.
Hierarchical The Communicative Variables include formal and
Inter-group informal communications within an organization
Intra-group (from products to e-mails), among organizations,
Technology ; and between individuals and the social networks
Networked Analysis

they create. In his essay on estimative probability,
Kent highlights this area by describing the diff
culty that producers of intelligence have in com-
municating the likelihood of an event to their consumers.*® In addition to
addressing organizational issues, case studies by Wohlstetter and others touch
on communication and social networks and the impact that communication

Collaboration

* Alexander Butterfield, The Accuracy of Intelligence Assessment, Jack Davis, “Combating
Mindset”; James M. Goldgeier, “Psychology and Secur ’; Heuer.

%6 Frank H. Knight, Risk, Uncertainty and Profit.

»' Caroline Ziemke, Philippe Loustaunau, and Amy Alrich, Strategic Personality and the Effec-
tiveness of Nuclear Deterrence.

% Acculturation is the cultural change that occurs in response to extended firsthand contact
between two or more previously autonomous groups. It can result in cultural changes in groups as
well as individuals.
% Sherman Kent,

‘Words of Estimative Probability.”

42



CHAPTER THREE.

load model, David Meister’s behavioral model, and the cognitive process
model of Gary Klein and his colleagues.?* Each model focuses on a different
aspect of human performance: White’s examines the actual task and task
requirements; Meister’s looks at the behavior of individuals performing a
task; and Klein’s uses verbal protocols to identify the cognitive processes of
individuals performing a task.

Surveying the literature. My research began with a review of the literature,
both for background information and for the identification of variables. The
intelligence literature produced by academics and practitioners tends to be
episodic, or case-based. This is not unique to the field of intelligence. A num-
ber of disciplines—medicine, business, and law, for example—are also case-
based. Many of the texts were general or theoretical rather than episodic.
Again, this is not an uncommon phenomenon. The review yielded 2,432 case
studies, journal articles, technical reports, transcripts of public speeches, and
books related to the topic. I then narrowed the list to 374 pertinent texts on
which a taxonomy of intelligence analysis could be built, and I analyzed them
to identify individual variables and categories of variables that affect intelli-
gence analysis.”6

Using a methodology known as “Q-Sort,” by which variables are sorted and
categorized according to type, I read each text and recorded the variables that
each author identified.27 These variables were then sorted by similarity into
groups. Four broad categories of analytic variables emerged from this pro-
cess.°8

Refining the prototype. Next, | used the preliminary taxonomy derived from
my reading of the literature to structure interviews with 51 substantive experts
and 39 intelligence novices. In tandem, I conducted two focus group sessions,
with five individuals in each group. As a result of the interviews and focus
group discussions, I added some variables to each category, moved some to
different categories, and removed some that appeared redundant.

Testing in a controlled setting. Finally, to compare the taxonomy with spe-
cific analytic behaviors, I watched participants in a controlled intelligence
analysis—training environment. Trainees were given information on specific

% Alexander Ervin, Applied Anthropology. See Chapter One, note 4 for a definition of triangula-
tion.

5 Robert White, Task Analysis Methods; David Meister, Behavioral Analysis and Measurement
Methods; G. Klein, R. Calderwood, and A. Clinton-Cirocco, Rapid Decision Making on the Fire
Ground.

% A copy of the list and search criteria is available from the author,

2 William Stephenson, The Study of Behavior: Q-Technique and its Methodology. See Chapter
Eleven for additional information on this methodology.

2 | would like to credit Dr. Forrest Frank of the Institute for Defense Analyses for his suggestions
regarding the naming convention for the categories of variables in the accompanying chart.

38



INTRODUCTION

1 mention Mead not only because my work draws heavily on hers, but also
because of her impact on the discipline and its direction. She moved from tra-
ditional cultural anthropological fieldwork in the South Pacific to problem-
oriented applied anthropology during World War II. She was the founder of
the Institute for Intercultural Studies and a major contributor to the Cold War
RAND series that attempted to describe the Soviet character. She also pio-
neered many of the research methods that are used in applied anthropology
today. I mention her work also as an illustrative point. After two years of field
research in the South Pacific, she wrote at least five books and could possibly
have written more.

As I look over the stacks of documentation for this study, it occurs to me
that, given the various constraints of the fellowship, there is more material
here than I will be able to address in any one text. There are the notes from
489 interviews, direct observations, participant observations, and focus
groups; there are personal letters, e-mail exchanges, and archival material; and
there are my own notes tracking the progress of the work. Moreover, the field-
work continues. As I write this, I am scheduling more interviews, more obser-
vations, and yet more fieldwork.

This text, then, is more a progress report than a final report in any tradi-
tional sense. It reflects findings and recommendations to date and is in no way
comprehensive. Finally, based as it is on my own research interests and
research opportunities, it is but one piece of a much larger puzzle.

© Boas (1858-1942) developed the linguistic and cultural components of ethnology. His most
notable work was Race, Language, and Culture (1940).

Xx



FINDINGS

We have a basic training program, but it is different from the other
agencies. Our mission is different. The problem is that we talk past
each other all the time.

When I got hired, I had an advanced degree. People assumed that, if
Thad a Masters, I could just figure out what I was supposed to do.

The focus of training within the community varies widely and is shaped by
the mission of the agency, such as technical, tactical, and operational. Many
spend a considerable amount of time teaching new analysts how to prepare
briefings, write papers, and perform administrative functions unique to their
agency. This is logical from the perspective of agency managers, who natu-
rally believe that investments made in personnel, training, and readiness ought
to be tailored specifically for their own organizations.

The problem with an agency-centric view is that, without a general commu-
nitywide training program for intelligence analysts, agencies and their ana-
lysts have difficulty finding, communicating, and interacting with one
another.”! Analysts often said they were disinclined to draw on resources out-
side of their own agency, indicating that either they do not know whom to
contact or their experience in the past has been influenced by a strict organiza-
tional focus.

The media keep talking about intelligence failures and communica-
tion breakdowns in the Intelligence Community. What do they
expect? We don’t even speak the same language.

It’s taken me 15 years to build my own network. If I didn’t have my
own contacts, I wouldn't know who to call.

1 don't bother going outside. Our focus is different here.

We have official channels, but it only really works if you trust the
person on the other end of the phone. That’s hard to do if you don't
know them.

Without an inclusive communitywide basic training program, differentia-
tion between the intelligence analysis discipline, as a whole, and other fields
of study is unlikely. A community of practitioners will have difficulty interact-
ing with one another, communicating between and within organizations, and
establishing a professional identity, which is a key ingredient in the develop-
ment of a professional discipline.

2 Stephen Marrin, CIA's Kent School: A Step in the Right Direction and “Improving CIA Analysis
by Overcoming Institutional Obstacles.”

29



CHAPTER THREE

A Taxonomy of Intelligence Variables!

a

Science is organized knowledge.

Herbert Spencer?

Aristotle may be the father of scientific classification, but it was biologist
Carolus Linnaeus who introduced the first formal taxonomy—kingdom, class,
order, genera, and species—in his Systema Naturae in 1735. By codifying the
naming conventions in biology, Linnaeus’s work provided a reference point
for future discoveries. Darwin’s development of an evolutionary theory, for
example, benefited greatly from Linnaeus’s creation of a hierarchical group-
ing of related organisms. The Systema Naturae taxonomy was not a fixed
product but rather a living document. Linnaeus himself revised it through 10
editions, and later biologists have continued to modify it.

In response to new discoveries and the development of new research meth-
‘ods in other domains, taxonomies were created to help organize those disci-
plines and to assist researchers in identifying variables that required additional
study. The development of specific taxonomies—from highly structured sys-
tems, such as the periodic table of chemical elements, to less structured
approaches, such as Bloom’s Taxonomy*—is a key step in organizing knowl-

1 A version of this chapter, “Developing a Taxonomy of Intelligence Analysis Variables,” origi-
nally appeared in Studies in Intelligence 47, no. 3 (2003): 61-71.

2? Herbert Spencer's The Study of Sociology, published in 1874, set the stage for the emergence of
sociology as a discipline.

3 Emst Haeckel introduced phylum to include related classes and family to include related genera in
1866. The Linnaeus taxonomy is currently being revised to accommodate genomic mapping data.

33



CHAPTER FIVE

Entering a field of study as a novice, an individual needs to learn the heuris-
tics and constraints—that is, the guiding principles and rules—of a given task
in order to perform that task. Concurrently, the novice needs to be exposed to
specific cases that test the reliability of such heuristics. Generally, novices
find mentors to guide them through the process of acquiring new knowledge.
A fairly simple example would be someone learning to play chess. The novice
chess player seeks a mentor who can explain the object of the game, the num-
ber of spaces, the names of the pieces, the function of each piece, how each
piece is moved, and the necessary conditions for winning or losing a game.

In time, and with much practice, the novice begins to recognize patterns of
behavior within cases and, thus, becomes a journeyman. With more practice
and exposure to increasingly complex cases, the journeyman finds patterns not
only within but also among cases and, more important, learns that these pat-
terns often repeat themselves. Throughout, the journeyman still maintains reg-
ular contact with a mentor to solve specific problems and to learn more
complex strategies. Returning to the example of the chess player, the individ-
ual begins to learn patterns of opening moves, offensive and defensive strate-
gies, and patterns of victory and defeat.

The next stage begins when a journeyman makes and tests hypotheses
about future behavior based on past experiences. Once he creatively generates
knowledge, rather than simply matching patterns, he becomes an expert. At
this point, he becomes responsible for his own knowledge and no longer needs
a mentor. In the chess example, once a journeyman begins competing against
experts, makes predictions based on patterns, and tests those predictions
against actual behavior, he is generating new knowledge and a deeper under-
standing of the game. He is creating his own cases rather than relying on the
cases of others.

The chess example in the preceding paragraphs is a concise description of
an apprenticeship model. Apprenticeship may seem to many a restrictive, old-
fashioned mode of education, but it remains a standard method of training for
many complex tasks. In fact, academic doctoral programs are based on an
apprenticeship model, as are such fields as law, music, engineering, and medi-
cine. Graduate students enter fields of study, find mentors, and begin the long
process of becoming independent experts and generating new knowledge in
their respective domains.

To some, playing chess may appear rather trivial when compared, for
example, with making medical diagnoses, but both are highly complex tasks.
Chess heuristics are well-defined, whereas medical diagnoses seem more open
ended and variable. In both instances, however, there are tens of thousands of
potential patterns. A research study discovered that chess masters had spent
between 10,000 and 20,000 hours, or more than 10 years, studying and play-

62



CHAPTER TEN

analytic units (or units of meaning), developing categories, coding the analytic
units into content areas, and grouping the analytic units into categories. From
these categories, general trends and specific instances can be identified. As
noted, the direct and participant observational data were analyzed separately
in order to triangulate the findings from the interview data. The purpose of
using multiple data sources for triangulation is to uncover internal incons
tencies in the data, to cross-check those inconsistencies with the available lit-
erature, and to verify the content validity for each category.

Demographics

As of this writing, 489 semi-structured interviews have been conducted with
active and retired intelligence professionals, intelligence technology research-
ers, academics who teach the intelligence discipline or have published in it, and

consumers of intel-
Distribution of Interviews by Domain ligence products.’
Of the 489 individ-
uals interviewed,

4%
se intelligence 70-percent —- were
Eroresatenae newly hired, active,
Academics . A a
15% or retired _intelli-

Technology Researchers. gence — profession-
als; 15-percent were
academi 11-per-
cent were intelli-
gence technology
researchers; and the
remaining four percent were policy makers or senior consumers of intelligence
products. The graph here shows the distribution of interviews by percentage for
each professional category.

The table below lists each professional category and the corresponding total
number (N) of individuals interviewed. The intelligence professional category
is further divided into three sub-groups. The “novice” sub-group includes new
hires and those with less than two years of experience.'” The “active” sub-

70% (Consumers

Leonard Bickman and Debra Rog, Handbook of Applied Social Research Methods; Meredith
Gall et al., Educational Research; Jonathan Gross, Measuring Culture: A Paradigm for the Anal-
ysis of Social Organization; Ernest House, Evaluating with Validity; Jerome Kirk and Mare
Miller, Reliability and Validity in Qualitative Research, Qualitative Research Methods, Volume 1;
Delbert Miller, Handbook of Research Design and Social Measurement; Michael Patton, Qualita~
tive Evaluation and Research Methods; Peter Rossi and Howard Freeman, Evaluation. A System-
atic Approach.

® Additional interviews are being conducted.

124


PART IV
Notes on Methodology

SE

7



INTRODUCTION

with regard to team interactions. Although, on the face of it, teams of astro-
nauts were composed of peers, a social distinction nevertheless existed
between commander, pilots, and mission specialists.

As with surgery, there was a disincentive for one team member to confront
or criticize another, even in the face of an impending error. Eighty percent of
the current astronauts come from the military, which has very specific rules
regarding confrontations, dissent, and criticism.” In addition to the similarities
in behavior arising from their common backgrounds, the “criticism” taboo
was continually reinforced throughout the astronaut’s career. Virtually any
negative comment on an astronaut’s record was sufficient for him or her to be
assigned to another crew, “washed out” of an upcoming mission and recycled
through the training program, or, worse still, released from the space program
altogether.

Taboos are social markers that prohibit specific behaviors in order to main-
tain and propagate an existing social structure. Generally, they are unwritten
rules not available to outside observers. Insiders, however, almost always per-
ceive them simply as the way things are done, the natural social order of the
organization. To confront taboos is to confront the social structure of a culture
or organization.

I mention the surgical and astronautical studies for a number of reasons.
Each serves as background for the study of intelligence analysts. Astronauts
and surgeons have very high performance standards and low error rates.* Both
studies highlight other complex domains that are interested in improving their
own professional performance. Both studies reveal the need to employ a vari-
ety of research methods to deal with complicated issues, and they suggest that
there are lessons to be learned from other domains. Perhaps the most telling
connection is that, because lives are at stake, surgeons and astronauts experi-
ence tremendous internal and external social pressure to avoid failure. The
same often holds for intelligence analysts.

In addition, surgery and astronautics are highly selective and private disci-
plines. Although their work is not secret, both groups tend to be shielded from
the outside world: surgeons for reasons of professional selection, training, and
the fiscal realities of malpractice liability; astronauts because their community

2 National Aeronautics and Space Administration, Astronaut Fact Book.

2 NASA has launched missions with the shuttle fleet 113 times since 1981 and has experienced
two catastrophic failures. It is probable that both of those were mechanical/engineering failures
and not the result of astronaut error. Surgical reporting methods vary from hospital to hospital,
and it is often difficult to determine the specific causes of morbidity and mortality. One longitudi-
nal study of all surgical procedures in one medical center puts the surgical error rates at that center
between 2.7 percent and 7.5 percent. See Hunter McGuire, Shelton Horsley, David Salter, et al.,
“Measuring and Managing Quality of Surgery: Statistical vs. Incidental Approaches.”



CHAPTER TEN
Survey Methodology

SE

This study included 489 interviews with intelligence professionals, academ-
ics, and researchers throughout the Intelligence Community. It also involved
participation in intelligence training programs, workshops, and focus groups;
direct observation of intelligence analysts performing their duties, and partici-
pant observation in a variety of analytic tasks. My access was not restricted to
specific people, locations, or organizations. I was allowed to observe, inter-
view, and participate in whatever manner I thought would be most beneficial
to the research project.

Unlike other academic studies of the intelligence discipline (case studies or
topic-specific postmortems, for example), this study was process oriented. It
also differed from the work of Sherman Kent, Richards Heuer, and other intel-
ligence professionals concerned with the process of intelligence analysis.'
Rather than having an intelligence professional looking out to the social and
behavioral sciences, this study had a social scientist looking in at the intelli-
gence profession. Although some of the conclusions of this work may be sim-
ilar to previous studies, the change in perspective has also led to some
different findings.

It is important to keep in mind that cultural anthropology is a qualitative
discipline and that, in general, its findings are descriptive and explanatory
rather than inferential or predictive. The use of ethnographic methods to
describe a culture, the environment in which that culture operates, and the
work processes that culture has adopted is designed to generate testable theory
that can be investigated experimentally or quasi-experimentally using other

' Sherman Kent, Strategic Intelligence for American World Policy; Richards J. Heuer, Jr.. Psy-
chology of Intelligence Analysis.

119



A TAXONOMY OF INTELLIGENCE

system to make assumptions and interpret information. His assumptions are
usually implicit rather than explicit and may not be apparent even to him.”2!

These definitions reflect the other end of the spectrum from those con-
cerned with tools and techniques. They suggest that the analytic process is a
construction of the human mind and is significantly different from individual
to individual or group to group. Certainly, Kam goes farthest along this path,
but even he does not suggest that one forgo tools; rather, he says that the pro-
cess of choosing the tool is governed by cognition as well.

Recognizing that the scope of intelligence analysis is so broad that it
includes not only methods but also the cognitive process is a significant step.
Viewing analysis as a cognitive process opens the door to a complex array of
variables. The psychology of the individual analyst must be considered, along
with individual analytic tools. In the broadest sense, this means not merely
understanding the individual psyche but also understanding the variables that
interact with that psyche. In other words, intelligence analysis is the socio-
cognitive process,” occurring within a secret domain, by which a collection of
methods is used to reduce a complex issue to a set of simpler issues.

Developing the Taxonomy

The first step of science is to know one thing from another, This
knowledge consists in their specific distinctions; but in order that it
may be fixed and permanent distinct names must be given to differ-
ent things, and those names must be recorded and remembered.

Carolus Linnaeus

My research was designed to isolate variables that affect the analytic pro-
cess. The resulting taxonomy is meant to establish parameters and to stimulate
dialogue in order to develop refinements. Although a hierarchic list is artificial
and rigid, it is a first step in clarifying areas for future research. The actual
variables are considerably more fluid and interconnected than such a structure
suggests. Once the individual elements are refined through challenges in the
literature, they might be better represented by a link or web diagram.”

To create this intelligence analysis taxonomy, I used Alexander Ervin’s
applied anthropological approach, which employs multiple data collection
methods to triangulate results.” I also drew on Robert White’s mental work-

2 Ephraim Kam, Surprise Attack. The Victim's Perspective, 120
» That is, analysis does not occur in a vacuum. It is socially constructed. See Lev Vygotsky, Mind
and Society.

® See Chapter Four for Judith Meister Johnston's systems analysis approach to describing the flu-
idity of the intelligence process

37





BIBLIOGRAPHY

Weiser, M., and J. Shertz. “Programming Problem Representation in Novice
and Expert Programmers,” Instructional Journal of Man-Machine Studies 14
(1983): 391-96.

Weisgerber, Leo. Vom Weltbild der Deutschen Sprache. Dusseldorf: Padago-
gischer Verlag Schwann, 1953.

White, Robert. Task Analysis Methods: Review and Development of Tech-
niques for Analyzing Mental Workload in Multiple Task Situations. St. Louis,
MO: McDonnell Douglas Corporation, 1971, MDC J5291.

Whorf, Benjamin. Language, Thought, and Reality. New York: John Wiley,
1956.

Wirtz, James J. The Tet Offensive: Intelligence Failure in War. Ithaca, NY:
Cornell University Press, 1991.

Wisher, R.A., M.A. Sabol, and J.A. Ellis. Staying Sharp: Retention of Military
Knowledge and Skills. Alexandria, VA: US Army Research Institute for the
Behavioral and Social Sciences, 1999, ARI Special Report 39. (<http://
www.ari.army.mil>.)

Wohlstetter, Roberta. Pearl Harbor: Warning and Decision. Stanford, CA:
Stanford University Press, 1962.

Wolf, Eric. Pathways of Power. Building an Anthropology of the Modern
World. Berkeley: University of California Press, 2001.

Yin, Robert. Case Study Research: Design and Methods 2nd ed. Thousand
Oaks, CA: Sage, 1994.

Ziemke, Caroline, Philippe Loustaunau, and Amy Alrich. Strategic Personal-
ity and the Effectiveness of Nuclear Deterrence. Washington, DC: Institute
for Defense Analyses, 2000, D-2537.

Ziotnick, Jack. “Bayes’ Theorem for Intelligence Analysis” (1972) in H.
Bradford Westerfield, ed. Inside CIA’s Private World: Declassified Articles
from the Agency's Internal Journal. New Haven, CT: Yaie University Press,
1995.

Zou, Yali, and Enrique Trueba. Ethnic Identity and Power. Albany: State Uni-
versity of New York, 1998.

156



BIBLIOGRAPHY

Godson, Roy, ed. Comparing Foreign Intelligence: The U.S., the USSR, the
U.K. and the Third World. Washington, DC: Pergamon-Brassey, 1988.

Goldberg, Lewis. “Simple Models or Simple Processes? Some Research on
Clinical Judgments,” American Psychologist 23 (1968): 483-96.

. “Man versus Model of Man: A Rationale, Plus Some Evidence, for a
Method of Improving on Clinical Inferences,” Psychological Bulletin 73
(1970): 422-32.

Goldgeier, James M. “Psychology and Security,” Security Studies 6, no. 4
(1997): 137-66.

Gott, Sherrie, PR.S. Kane, and Alan Lesgold. Tutoring for Transfer of Techni-
cal Competence. Brooks AFB, TX: Armstrong Laboratory, Human
Resources Directorate, 1995, AL/HR-TP-1995-0002.

Graesser, A.C., M.A. Gernsbacher, and S. Goldman, eds. Handbook of Dis-
course Processes, Mahwah, NJ: Lawrence Erlbaum Associates, 2003.

, and Natalie Person. “Question-Asking During Tutoring.” American
Educational Research Journal 31 (1994): 104-37.

Greenberg, Jerald, and Robert A. Baron. Behavior in Organizations: The
Human Side of Work. Upper Saddle River, NJ: Prentice-Hall, 2002.

Grimes, Barbara, ed. Ethnologue, 14th ed. Dallas, TX: Summer Institute of
Linguistics, 2000.

Gross, Jonathan. Measuring Culture: A Paradigm for the Analysis of Social
Organization, New York: Columbia University Press, 1985.

Grove, William, and Paul Meehl. “Comparative Efficiency of Informal (Sub-
jective, Impressionistic) and Formal (Mechanical, Algorithmic) Prediction
Procedures: The Clinical-Statistical Controversy,” Psychology, Public Pol-
icy, and Law 2, no. 2 (1996): 293-323.

Gumperz, John, and Dell Hymes. Directions in Sociolinguistics: The Ethnog-
raphy of Communication. Oxford: B. Blackwell, 1986.

., and Stephen Levinson. Rethinking Linguistic Relativity. Cambridge:
Cambridge University Press, 1996.

Hacker, D.J. Metacognition: Definitions and Empirical Foundations [On-line
report]. Memphis, TN: The University of Memphis, 2001. Retrieved 30 May
2001 from the World Wide Web. (http://www.psyc.memphis.edu/trg/
meta.htm.)

Hammond, John, Ralph Keeney, and Howard Raiffa. “The Hidden Traps in
Decision Making.” Harvard Business Review, September—October 1998,
47-58.

Harvey, J., and C. Boettger. “Improving Communication Within a Managerial
Workgroup,” Journal of Applied Behavioral Science 7 (1971): 164-74.

Hayes, J.R. The Complete Problem Solver, Philadelphia: The Franklin Insti-
tute, 1981.

Hays, R. T., et al. “Flight Simulator Training Effectiveness: A Meta-Analy-
sis.” Military Psychology 4, no. 2 (1992): 63-74.

145



CHAPTER FOUR

ers, they add to the accumulation of material that affects the ability of an ana-
lysts to produce.

Product Influences. The accumulation of completed intelligence products,
which is represented as a stock, is not in practice an end-state for analysis. A
customer may respond to a delivered product by levying additional or revised
tasking. In all instances, this information influences the level of need for poli-
cymaker requirements and causes the process to begin again. Each iteration of
the process is different, not because the steps in the process change, but
because those responsible for carrying out the steps have changed as a result
of their participation in the previous run. These changes can include a greater
level of experience with the process, with the customer, with the topic area, or
with the quirks of the organization and its processes. The changes are a mani-
festation of the concept that the system is greater than the sum of its parts.

Findings Based on Systems Analysis

Systems analysis clearly demonstrates the defects of the traditional Intelli-
gence Cycle model. To recapitulate briefly, the traditional model merely repre-
sents a simple list of steps rather than a dynamic closed feedback loop. In
addition, although the steps are meant to be performed by several different
actors, the model does not provide useful information about what each actu-
ally contributes to the cycle, nor does it accurately represent the path a request
takes as it is addressed. Another problem with the traditional model is that
none of its features help identify ways of developing a consistent product. For
example, there is no allowance for a statement of objectives or for any forma-
tive or summative evaluations to check that objectives have been met.

On the other hand, the model that resulted from a systems analysis provides
a more complex view. That model shows cause and effect, and it shows what
other elements have an impact on the development of intelligence products
and how and why elements depend on other elements. These advantages of the
systems model are clearly apparent in considering the role of analysts in pro-
duction, a crucial element of the cycle that the traditional model all but
ignores.

Impact on Production and Analyst's Control. Study of the systems model
shows that the “Analyst’s Ability to Produce” (upper right-hand quarter of the
diagram) is the central factor in the production cycle and the driver of the feed-
back loop. The systems view also makes us aware of a less obvious fact that is
critically important to a discussion of analytic failure.

A look at the entire system makes readily apparent the number of factors of
varying complexity that influence an analyst’s ability to produce: the analyst’s
capabilities; the product evaluation process; the political and cultural values of

54



CHAPTER TWO
Findings

Se ee

Scientific knowledge, like language, is intrinsically the common
property of a group or else nothing at all. To understand it we shall
need to know the special characteristics of the groups that create
and use it.

Thomas Kuhn!

The more we learn about the world, and the deeper our learning, the
more conscious, specific, and articulate will be our knowledge of
what we do not know.

Karl Popper?

The purpose of this research was to identify and describe conditions and
variables that negatively affect intelligence analysis, to develop relevant and
testable theory based on these findings, and to identify areas in which strate-
gies to improve performance may be effective. Although there has recently
been a great deal of concern that intelligence error and failure rates are inordi-
nately high, in all likelihood, these rates are similar to those of other complex
socio-cognitive domains, such as analysis of financial markets. The significant
differences are that other complex domains employ systematic performance

' Philosopher of science Thomas Kuhn described the now-common concept of paradigm shifts in
scientific revolutions. He posited that paradigm shifts are tied to cultural and social construction-
ist models, such as Vygotsky's (See footnote 22 in Chapter Three). Thomas Kuhn, The Structure
of Scientific Revolutions.

2 Karl Popper was one of the 20th century’s pre-eminent philosophers of science. Karl Popper,
Conjectures and Refutations: The Growth of Scientific Knowledge



CHAPTER FOUR

Systems Model of the Intelligence Cycle

Volume of Requests

‘or Intel Products Completed ite
amand on fealyst Prodvete
Ss.
Increase or Decrease in
in Requirements
Complexity of Document, Tumaround Time
Poltical and Cuturht

Type of Document ‘Values of

Regu Organization

World Everts

Collected Data Porslable Osta

CChaghe in Amount EFfecyy’ot Requiriments

of Gotected Data “raning

Evahation of
INTs ren Relevance
Collection Demands Experience
Requirements Formanting Processing
of Avalable Data
—~ ‘Addtional of Revised

Customer fequests. Level of Relevance
me of Product

Relevance to New issues
Original Request Raised

52



CHAPTER ELEVEN

observational data could have been sorted into several categories, I based the
placement of the data on the question that generated the interview response.

Table 1. Q-Sort 1. Data Grouping According to Organizational Function.

National - Technical

Defense

Law Enforcement —
Homeland Security

Central Intelligence

Defense Intelligence

Department of

Reconnaissance Office

Agency Agency Homeland Security
National Security ‘Amy Intelligence Federal Bureau of
Agency Investigation
National Air Force Intelligence Department of Energy

National Geospatial
Intelligence Agency

Navy Intelligence

Depariment of the
Treasury

Department of State
(INR)

Marine Corps
Intelligence

Drug Enforcement
Administration

In several instances throughout the text, the quotes that were used may well
fit in a number of other categories. Once the data were sorted by variable, the
coding and context identifier notes were removed from all data in order to
assure participant anonymity, in keeping with the American Anthropological
Association Code of Ethics, section III, A.*

Table 2. Q-Sort 2. Data Grouping According to Variable Categories.

Time ‘Analytic Organiza- Analytic ‘Analytic
Constraints Methods tional Identity Training
Norms
Products Tradecraft Taboos Reportorial Formal
Interactions Science Biases Academic Informal

The quotes that appear throughout the text are exemplars from each vari-
able category and indicate trends found in the data-set. Although the exemplar
quotes are not universal, nor are they necessarily subject to generalization,
they do represent consistent findings from the interview and observation data.
Utilizing this approach to develop theory is similar to the method in which
grounded theory is employed in sociology, specifically, using grounded data
to generate theory rather than using some a priori technique. The significant
advantage to this approach is that the theory is directly tied to data, providing
it additional validity. Another advantage is that the individuals who allowed
me to interview and observe them are given some voice in the final product by
way of direct quotes, which also provides some qualitative context.

> American Anthropological Association, Code of Ethics of the American Anthropological Asso-
ciation.

128



CHAPTER FOUR

Testing the Intelligence Cycle Through Systems
Modeling and Simulation

ae

Judith Meister Johnston!
Rob Johnston

Throughout the Intelligence Community, the process of analysis is represented
conventionally by a model known as the Intelligence Cycle (See next page).
Unfortunately, the model omits elements and fails to capture the process accu-
rately, which makes understanding the challenges and responsibilities of intelli-
gence analysis much more difficult. It also complicates the tasks of recognizing
where errors can occur and determining methods for change based on accurate
predictions of behavior. Our analysis of the Intelligence Cycle, employing a sys-
tems approach and a simulation created to represent it, demonstrated these short-
comings.” Because of its wide acceptance and use in training and in discussions
of the analytic process, the traditional representation of the Intelligence Cycle
will be closely considered in this chapter, especially with regard to its impact on
analytic products, its effectiveness, and its vulnerability to error and failure.

The Traditional Intelligence Cycle

The Intelligence Cycle is customarily illustrated as a repeating process con-
sisting of five steps.? Planning and direction encompasses the management of

' Dr. Judith Meister Johnston is an educational psychologist with expertise in human performance
technology and instructional systems design. A Booz Allen Hamilton Associate, she supports
human factors work for the Intelligence Community.

2 Simulation involves the development of a computer-based model that represents the internal
processes of an event or situation and estimates the results of proposed actions.

45



ethenogerasphy\n [F ethnographie, fr. ethno- + -graphie -graphy] (1834) :
the study and systematic recording of human cultures: also: a descriptive work
produced from such research. (Merriam Webster’s Collegiate Dictionary,
Eleventh Edition)



FINDINGS

I’m looking for links and patterns. Once I figure out the pattern, I
can figure out where to look next.

use patterns. If things start happening, out of the ordinary things, I
pay attention to them.

I try to build patterns out of the data. It helps me predict what will
happen next.

I come up with a few scenarios and see what the evidence supports.

I look for data that are diagnostic: some piece of evidence that rules
out certain possibilities.

T try to weigh the evidence to see which scenario it supports.

Although anomaly-detection, pattern-recognition, and weighing data may
appear to be idiosyncratic tradecraft based on individual expertise and cogni-
tive skills, these methods can be formalized and replicated if the operating
parameters, variables, and rules of evidence are made explicit.'° This is to say
that intelligence analysis can be reconstructed in the context of a scientific
method, which is merely an articulated, formal process by which scientists,
collectively and over time, endeavor to put together a reliable, consistent, and
nonarbitrary representation of some phenomena. Broadly, the steps include:

* observation and description of phenomena;

* formulation of hypotheses to explain phenomena;
* testing of hypotheses by independent experts;

+ refutation or confirmation of hypotheses.

These steps do not suggest that any specific scientific methodology results
in what is ultimately the truth, rather that scientific methods are merely formal
processes used to describe phenomena, make predictions, and determine
which hypothesis best explains those phenomena. The principal value of any
type of methodological formalism is that it allows other researchers to test the
validity and reliability of the findings of any other researcher by making
explicit, and therefore replicable, the means by which anyone reaches a spe-
cific conclusion. "!

The idea that intelligence analysis is a collection of scientific methods
encounters some resistance in the Intelligence Community. The interview data
analyzed in this study highlight many subtle—and not so subtle—prejudices

19 A corollary to these methods can be found in the practice of radiologists. See Chapter Five for
more on expertise.



BIBLIOGRAPHY
a

Published Sources

Akin, Omer. Models of Architectural Knowledge. London: Pion, 1980.

Allen, Charles, Comments of the Associate Director of Central Intelligence for
Collection at a Public Seminar on Intelligence at Harvard University, spring
2000. (<http://pirp.harvard.edu/pdf-blurb.asp? id+518>.)

Allison, Graham T. Essence of Decision: Explaining the Cuban Missile Crisis.
Boston: Little, Brown and Company, 1971,

American Anthropological Association. Code of Ethics of the American
Anthropological Association. Arlington, VA, 1998. (http://www.aaanet.org/
committees/ethics/ethcode.htm.)

. Statement on the Confidentiality of Field Notes. Arlington, VA, 2003.
(http://www.aaanet.org/stmts/fieldnotes.htm.)

American College of Radiology. Written communications between staff and
Johnston, 2002.

Arvey, Richard D., and James E. Campion. “The Employment Interview: A
Summary of Recent Research,” Personnel Psychology 35 (1982): 281-322.

Ashcraft, M.H. Fundamentals of Cognition. New York: Addison Wesley
Longman, 1998.

Atkinson, R. C., and H. A. Wilson. Computer-Assisted Instruction: A Book of
Readings, New York: Academic Press, 1969.

Au, Terry K. “Chinese and English Counterfactuals: The Sapir-Whorf
Hypothesis Revisited,” Cognition 15 (1983): 155-87.

Barr, A., M. Beard, and R. C. Atkinson. “A Rationale and Description of a
CAI Program to Teach the BASIC Programming Language.” Instructional
Science 4 (1975): 1-31.

Bayes, Thomas. “An Essay Toward Solving a Problem in the Doctrine of
Chances,” Philosophical Transactions of the Royal Society of London 53:
370-418. (http://plato.stanford.edu/entries/bayes-theorum/#Bib.)

Ben-Israel, Isaac. “Philosophy and Methodology of Intelligence: The Logic of
Estimate Process,” Intelligence and National Security 4, no. 4 (1989): 660-718.

Ben-Zvi, Abraham. “The Study of Surprise Attacks,” British Journal of Inter-
national Studies 5 (1979).

139



A TAXONOMY OF INTELLIGENCE

cases and directed to use various methods to analyze the situations and to gen-
erate final products. During the training exercises, the verbal and physical
behavior of individuals and groups were observed and compared with the tax-
onomic model. I participated in a number of the exercises myself to gain a bet-
ter perspective. This process corroborated most of the recommendations that
had been made by the experts and novices and also yielded additional vari-

ables for two of the categories.

The resulting taxonomy is purely descriptive. It
is not intended to demonstrate the weight or
importance of each variable or category. That is,
the listing is not sufficient to predict the effect of
any one variable on human performance. The
intention of the enumeration is to provide a frame-
work for aggregating existing data and to create a
foundation for future experimentation. Once the
variables have been identified and previous find-
ings have been aggregated, it is reasonable to con-
sider experimental methods that would isolate and
control individual variables and, in time, indicate
sources of error and potential remediation

Systemic Variables

The column of Systemic Variables incorporates
items that affect both an intelligence organization
and the analytic environment. Organizational
variables encompass the structure of the intelli-
gence organization; leadership, management, and
management practices; history and traditions; the
working culture, social practices within the orga-
nization, and work taboos; and organizational
demographics. They also include internal politics,
the hierarchical reporting structure, and material
and human resources. Industrial and organiza-
tional psychology, sociology, and management
studies in business have brought attention to the
importance of organizational behavior and its
effect on individual work habits and practices.
The works of Allison, Berkowitz and Goodman,
Elkins, Ford, Godson, and Richelson, among oth-
ers, examine in general the organizational aspects
of intelligence.”

39

Systemic Variables
Organization
Internal
Structure
Leadership
Cutture
History
Traditions
Social Practice
Taboo
Group Characteristics
Hierarchy
Resources and Incen-
tives
Manpower
Budget
Technology
Assets
R&D
Facilities
Work Groups-Teams
External
Consumer Needs
Time and Imperatives
Consumer Use
Consumer Structure
Consumer Hierarchy
Conumer Reporting
Politics
Internal-Organization
Policy
Tradition
Taboo
Security/Access
External-National
Law
Policy
External-International
Security
Denial
Deception
Policy



DEFINITIONS

interactive, dynamic, and social nature of intelligence analysis. The interview
participants were not asked to define intelligence analysis as such; rather, they
were asked to describe and explain the process they used to perform analysis.
The interview data were then triangulated with the direct and participant
observation data collected during this study.*

Despite the seemingly private and psychological nature of analysis as
defined in the literature, what I found was a great deal of informal, yet pur-
poseful collaboration during which individuals began to make sense of raw
data by negotiating meaning among the historical record, their peers, and their
supervisors. Here, from the interviews, is a typical description of the analytic
process:

When a request comes in from a consumer to answer some question,
the first thing I do is to read up on the analytic line. [1] check the
previous publications and the data. Then, I read through the ques-
tion again and find where there are links to previous products.
When I think I have an answer, I get together with my group and ask
them what they think. We talk about it for a while and come to some
consensus on its meaning and the best way to answer the con-
sumer’s question. I write it up, pass it around here, and send it out
for reviews

The cognitive element of this basic description, “when I think I have an
answer,” is a vague impression of the psychological processes that occur dur-
ing analysis. The elements that are not vague are the historical, organizational,
and social elements of analysis. The analyst checks the previous written prod-
ucts that have been given to consumers in the past. That is, the analyst looks
for the accepted organizational response before generating analytic hypothe-
ses.

The organizational-historical context is critical to understanding the mean-
ing, context, and process of intelligence analysis. There are real organizational
and political consequences associated with changing official analytic findings
and releasing them to consumers. The organizational consequences are associ-
ated with challenging other domain experts (including peers and supervisors).
The potential political consequences arise when consumers begin to question
the veracity and consistency of current or previous intelligence reporting.
Accurate or not, there is a general impression within the analytic community

4 In research, triangulation refers to the application of a combination of two or more theories, data
sources, methods, or investigators to develop a single construct in a study of a single phenome-
non

5 Intelligence analyst’s comment during an ethnographic interview. Such quotes are indented and
italicized in this way throughout the text and will not be further identified; quotes attributable to
others will be identified as such:



CHAPTER TEN

Agency Aggregation According to Interviewee Job-type

National-Technical Defense Law Enforcement-
Homeland Security

Central Intelligence Defense Intelligence Department of
Agency Agency Homeland Security
National Security Army Intelligence Federal Bureau of
Agency Investigation
National Air Force Intelligence | Department of Energy
Reconnaissance Office
National Geospatial Navy Intelligence Department of Treasury
Intelligence Agency
Department of State Marine Corps Drug Enforcement
(INR), Intelligence Administration

The figure below shows the distribution of intelligence professionals inter-
viewed for this study according to each broader functional category. Of the
345 intelligence professionals interviewed, 214 work within the National-
Technical Intelligence category, 76 in the Defense Intelligence category, and
55 in the Law Enforcement—Homeland Security category.

Distribution of Interviews by Functional Category

® National - Technical

B® Defense
22%
lo Law Enforcement -
| Homeland Security

126



CHAPTER FIVE

Analytic Methodologists

Most domains have specialists who study the scientific process or research
methods of their discipline. Instead of specializing in a specific substantive
topic, these experts specialize in mastering the research and analytic methods
of their domain. In the biological and medical fields, these methodological
specialists are epidemiologists. In education and public policy, they are pro-
gram evaluators. In other fields, they are research methodologists or statisti-
cians. Whatever the label, each field recognizes that it requires experts in
methodology who focus on deriving meaning from data, recognizing patterns,
and solving problems within a domain in order to maintain and pass on the
domain’s heuristics. They become in-house consultants—organizing agents—
who work to identify research designs, methods for choosing samples, and
tools for data analysis.

Because they have a different perspective than do the experts in a domain,
methodologists are often called on by substantive experts to advise them on a
variety of process issues. On any given day, an epidemiologist, for example,
may be asked to consult on studies of the effects of alcoholism or the spread of
a virus on a community or to review a double-blind clinical trial of a new
pharmaceutical product. In each case, the epidemiologist is not being asked
about the content of the study; rather, he is being asked to comment on the
research methods and data analysis techniques used.

Although well over 160 analytic methods are available to intelligence ana-
lyst, few methods specific to the domain of intelligence analysis exist. Intel-
ligence analysis has few specialists whose professional training is in the
process of employing and unifying the analytic practices within the field. It is
left to the individual analysts to know how to apply methods, select one
method over another, weigh disparate variables, and synthesize the results—
the same analysts whose expertise is confined to specific substantive areas and
their own domains’ heuristics.

Conclusion

Intelligence agencies continue to experiment with the right composition,
structure, and organization of analytic teams. Yet, although they budget sig-
nificant resources for technological solutions, comparatively little is being

» Exceptions include: S. Feder, “FACTIONS and Policon”; R. Heuer, Psychology of Intelligence
Analysis; R. Hopkins, Warnings of Revolution: A Case Study of El Salvador; J. Lockwood and K.
Lockwood, “The Lockwood Analytical Method for Prediction (LAMP)” J. Pierce, “Some Math-
ematical Methods for Intelligence Analysis”; E. Sapp, “Decision Trees”; J. Zlotnick, “Bayes”
Theorem for Intelligence Analys

nR



BIBLIOGRAPHY

Seashore, S. Group Cohesiveness in the Industrial Workgroup. Ann Arbor:
University of Michigan Press, 1954.

Segall, Marshall. Cross-Cultural Psychology: Human Behavior in Global
Perspective. Monterey, CA: Brooks/Cole, 1979.

Senders, John, and Neville Moray. Human Error: Cause, Prediction and
Reduction. Hillsdale, NJ: Lawrence Earlbaum Associates, 1991.

Shiels, Frederick L. Preventable Disasters: Why Governments Fail. Savage,
MD: Rowman and Littlefield Publishers, 1991.

Shiffrin, R.M. and W. Schneider. “Controlled and Automatic Human Informa-
tion Processing: II. Perceptual Learning,” Psychological Review 84 (1977).

Shlaim, Avi. “Failures in National Intelligence Estimates: The Case of the
Yom Kippur War,” World Politics 28, no. 3 (1976): 348-80.

Shweder, Richard. Thinking Through Cultures. Cambridge, MA: Harvard
University Press, 1991.

Simmel, Georg. The Sociology of Georg Simmel. Translated by K. Wolff.
Glencoe, IL: Free Press, 1950.

Simon, D., and Herbert Simon. “Individual Differences in Solving Physics
Problems” in Robert Siegler, ed. Children’s Thinking: What Develops? Hills-
dale, NJ: Lawrence Erlbaum Associates, 1978.

Slavin, Robert. “Research on Cooperative Learning: Consensus and Contro-
versy,” Educational Leadership 47, no. 4 (1989): 52-55.

. Cooperative Learning. New York: Longman, 1983.

Slovic, P., S. Lichtenstein, and B. Fischoff. “Decision-Making” in R.C. Atkin-
son et al. eds. Steven’s Handbook of Experimental Psychology (2nd ed.), Vol-
ume 2: Learning and Cognition, New York: John Wiley & Sons, 1988.

Solomon, H. Economic Issues in Cost-Effectiveness Analyses of Military Skill
Training. Alexandria, VA: Institute for Defense Analyses, 1986, IDA P-
1897.

Spencer, Herbert. The Study of Sociology. New York: D. Appleton. 1874.

Spencer, Ken. “Modes, Media and Methods: The Search for Educational
Effectiveness,” British Journal of Educational Technology 22, no. | (1991):
12-22.

Spradley, James. Participant Observation. New York: Rinehart and Winston,
1980.

Stanton, Neville, and Sarah Stevenage. “Learning to Predict Human Error:
Issues of Acceptability, Reliability and Validity,” Ergonomics, 41, no. 11
(1998): 1737-56.

Stephenson, William. The Study of Behavior: Q-Technique and its Methodol-
ogy. Chicago: University of Chicago Press, 1953.

Suppes, P., and M. Morningstar. Computer-Assisted Instruction at Stanford,
1966-68: Data, Models, and Evaluation of the Arithmetic Programs. New
York: Academic Press, 1972.

154



CHAPTER TWO

Asked to define their profession, the majority of analysts described the pro-
cess of analysis rather than the actual profession. The question, “What is an
intelligence analyst?” resulted most often in a description of the work day and
of the production cycle of analytic products and very seldom in an explanation
of analytic methodology or a definition of an analyst outside of some specific
context. With very few exceptions, analysts did not describe intelligence anal-
ysis as its own discipline with its own identity, epistemology, and research tra-
dition.

This is not necessarily uncommon. When physicians are asked to describe
their profession, they tend to respond with a specific subdiscipline: “I’m a car-
dio-thoracic surgeon,” for example. When asked for a more general descrip-
tion, however, they tend to respond, “I’m a doctor” or “I’m a physician.” That
is, in selective, insular professional cultures, practitioners are able to define
their role in both a specific and general fashion. Intelligence analysts had diffi-
culty defining their professional identity in a general way and often relied on
specific context to explain what it is that they do and, by extension, who they
are.

The perception of individual analysts regarding their professional identity
was associated most often with their organization’s function or with their own
educational background and not with intelligence analysis as its own unique
discipline.

I work counternarcotics.
I work counterterrorism.
I’ma military analyst.
I'ma leadership analyst.
I’m an economist.

I'ma political scientist.

In addition to these categories, many analysts described their professional
identity in terms of intelligence collection methods or categories.

1 do all-source analysis.
I'ma SIGINT analyst.
I’man IMINT analyst.

I'ma technical analyst.

"” George Mead was an American pragmatist philosopher and social psychologist, who, with John
Dewey, made the University of Chicago the home of pragmatist philosophy and the “Chicago
School” of sociology at the end of the 19th century. George Mead, Mind, Self, and Society.



INTEGRATING METHODOLOGISTS.

case still does not fit with their expertise and reject the data set as an anomaly;
or consult other experts.

An item of information, in and of itself, is not domain specific. Imagine
economic data that reveal that a country is investing in technological infra-
structure, chemical supplies, and research and development. An economist
might decide that the data fit an existing spending pattern and integrate these
facts with prior knowledge about a country’s economy. The same economist
might decide that this is a new pattern that needs to be stored in long-term
memory for some future use, or he might decide that the data are outliers of no
consequence and may be ignored. Finally, the economist might decide that the
data would be meaningful to a chemist or biologist and, therefore, seek to col-
laborate with other specialists, who might reach different conclusions regard-
ing the data than would the economist.

In this example, the economist is required to use his economic expertise in
all but the final option of consulting other experts. In the decision to seek col-
laboration, the economist is expected to know that what appears to be new
economic data may have value to a chemist or biologist, domains with which
he may have no experience. In other words, the economist is expected to know
that an expert in some other field might find meaning in data that appear to be
economic.

Three disparate variables complicate the economist’s decisionmaking:

* Time context. This does not refer to the amount of time necessary to
accomplish a task but rather to the limitations that come from being close
to an event. The economist cannot say a priori that the new data set is the
critical data set for some future event. In “real time,” they are simply data
to be manipulated. It is only in retrospect, or in long-term memory, that
the economist can fit the data into a larger pattern, weigh their value, and
assign them meaning.

Pattern bias. In this particular example, the data have to do with infra-
structure investment, and the expert is an economist. Thus, it makes per-
fect sense to try to manipulate the new data within the context of
economics, recognizing, however, that there may be other, more impor-
tant angles.

Heuristic bias. The economist has spent a career becoming familiar with
and using the guiding principles of economic analysis and, at best, has
only a vague familiarity with other domains and their heuristics. An econ-
omist would not necessarily know that a chemist or biologist could iden-
tify what substance is being produced based on the types of equipment
and supplies that are being purchased.

67





CHAPTER THREE

Structural anthropology addresses many of the enculturation and identity issues
that affect individual behavior. Cognitive scientists are building models that can
be tested in experimental conditions and used for developing new tools and tech-
niques. Sociology and social theory have much to offer in studying social net-
works and communication.

The organization of knowledge in intelligence is not a small task, but I
believe that the effort should be undertaken for the betterment of the profes-
sion. The taxonomy proposed here could serve as a springboard for a number
of innovative projects, for example: development of a research matrix that
identifies what is known and how that information may be of use in intelli-
gence analysis, setting a research agenda in areas of intelligence that have
been insufficiently studied, application of research from other domains to
develop additional training and education programs for analysts, creation of a
database of lessons learned and best practices to build a foundation for an
electronic performance support system, integration of those findings into new
analytic tools and techniques, and development of a networked architecture
for collaborative problem-solving and forecasting. It is my hope that this tax-
onomy will help intelligence practitioners take steps in some of these new
directions.



CHAPTER FIVE

This example does not describe a complex problem; most people would
recognize that the data from this case might be of value to other domains. It is
one isolated case, viewed retrospectively, which could potentially affect two
other domains. But, what if the economist had to deal with 100 data sets per
day? Now, multiply those 100 data sets by the number of domains potentially
interested in any given economic data set. Finally, put all of this in the context
of “real time.” The economic expert is now expected to maintain expertise in
economics, which is a full-time endeavor, while simultaneously acquiring
some level of experience in every other domain. Based on these expectations,
the knowledge requirements for effective collaboration quickly exceed the
capabilities of the individual expert.

The expert is left dealing with all of these data through the lens of his own
expertise. Let’s assume that he uses his domain heuristics to incorporate the
data into an existing pattern, store the data in long-term memory as a new pat-
tern, or reject the data set as an outlier. In each of these options, the data stop
with the economist instead of being shared with an expert in some other
domain. The fact that these data are not shared then becomes a potentially crit-
ical case of analytic error.”

In hindsight, critics will say that the implications were obvious—that the
crisis could have been avoided if the data had been passed to one or another
specific expert. In “real time,” however, an expert often does not know which
particular data set would have value for an expert in another domain.

The Pros and Cons of Teams

One obvious solution to the paradox of expertise is to assemble an interdis-
ciplinary team. Why not simply make all problem areas or country-specific
data available to a team of experts from a variety of domains? This ought, at
least, to reduce the pattern and heuristic biases inherent in relying on only one
domain. Ignoring potential security issues, there are practical problems with
this approach. First, each expert would have to sift through large data sets to
find data specific to his expertise. This would be inordinately time-consuming
and might not even be routinely possible, given the priority accorded gisting
and current reporting.

Second, during the act of scanning large data sets, the expert inevitably
would be looking for data that fit within his area of expertise. Imagine a chem-
ist who comes across data that show that a country is investing in technologi-

22 L. Kirkpatrick, Captains Without Eyes: Intelligence Failures in World War II; F. Shiels, Pre-
ventable Disasters; J. Wirtz, The Tet Offensive: Intelligence Failure in War, R. Wohlstetter, Pearl
Harbor.

68



ORGANIZATIONAL CULTURE

to assume. This is all the more pronounced because, for the most part, the
Intelligence Community organizations lack a civilian counterpart.

Occasionally, the anticipatory socialization of people entering the intelligence
analysis discipline will derive from accounts of current or former practitioners.
More generally, however, a newcomer’s initial impressions stem from the fic-
tional media portrayals, which tend to emphasize the supposed glamour of oper-
ational tasks and pay little attention to the reality of research-based analytic
work. The absence of hard knowledge about intelligence work is attributable, in
part, to the organizational secrecy of the Intelligence Community and, in part, to
the actual socialization process that occurs after one has been accepted for
employment and has passed the required background investigation.

A newcomer’s experience is often contrary to initial expectations. Employ-
ees are discouraged from talking about the specifics of their work outside of
the organization or with those who have not been “cleared.” On an individual
level, this experience translates into professional culture shock and social iso-
lation. Organizationally, an intentionally closed system of this kind has a
number of potential performance-related consequences, among them perpetu-
ation of the existing organizational culture by hiring familial legacies or those
most likely to “fit in,” job dissatisfaction, low morale and consequent reduc-
tion in employee readiness, increased employee turnover, greater likelihood of
“groupthink,” and strong internal resistance to organizational change. '¢

Since the attacks of 11 September, the Intelligence Community has become
more open about its role in government, its day-to-day working environment,
and its employees’ functions and responsibilities. While this openness is an
extension of an ongoing trend toward public outreach—an example is the
CIA’s Officer-in-Residence program established in 1985—the community has
accelerated this trend toward openness in an effort to help the public, and its
representatives, understand the missions and value of the Intelligence Com-
munity. '7

This trend toward openness has improved employee retention by counter-
acting the culture shock of misinformed anticipatory socialization and result-
ant employee turnover. This trend also helps prepare the organization for the
inevitable changes to come by increasing the potential recruitment pool,
expanding the intellectual diversity of its staff, and fostering better relations
with its broader constituency, the American public.

'6 Irving Janis, Groupthink.
"7 See CIA Officer in Residence Program in Web Resources in bibliography.

103



CHAPTER EIGHT

research fellowships, professional workshops, and academic forums will help
to facilitate better employee relations by providing potential employees with a
clearer perspective on what to expect after receiving their badge.



FINDINGS

and ten. In this theory, analytic efficiency and effectiveness are purely func-
tions of system openness and do not take into account analytic methods or per-
sonnel.

This theory will require additional refinement, and it may or may not be
represented by a tradeoff curve like the one proposed here. The theory will
also require numerous controlled quantitative experiments to test its explana-
tory power.

Finding: Time Constraints

The work itself is a 24-hour-a-day job, but it never seems like | have
any time to actually analyze anything when I'm at my desk. I spend
most of my time reading daily traffic, answering e-mail, coordinat-
ing papers with everybody, and writing. Mostly I read and write, but
when the workday is over, I go home and think. It isn’t like I can
turn off my brain. So, I guess I do most of my real analysis on my
own time.

The majority of the analysts interviewed indicated that time was one of
their greatest constraints at work. This comment triangulated with the findings
from direct and participant observation. In addition, analysts indicated that
there has been a communitywide shift toward focusing on short-term issues or
problem solving, thereby addressing the immediate needs of intelligence con-
sumers. This shift in product focus, coupled with a growth in available all-
source raw intelligence, has resulted in a change in the pace of analytic pro-
duction. In order to generate the daily products, analysts have had to change
the way they go about doing their work.

I haven't been doing this very long, but I wish I had been a journal-
ism major instead of poli-sci. The pace is excruciating.

I don't get much sleep. It’s like cramming for finals, except we do it
every day.

Everything I do is reactive. I don’t have time to work my subject.
We’re not pro-active here.

I’m so busy putting out today’s fires, I don’t have any time to think
about what kind of catastrophe is in store for me a month from now.

About 15 years ago, I used to have 60 percent of my time available
for long-term products. Now, it’s between 20 and 25 percent.

I probably have about 30 percent of my time for self-initiated prod-
ucts.





The Author

Dr. Rob Johnston is an ethnographer who specializes in the cultural anthro-
pology of work. He has been a research staff member at the Institute for
Defense Analyses and a Director of Central Intelligence Postdoctoral
Research Fellow at the Central Intelligence Agency’s Center for the Study of
Intelligence, where he is now a member of the staff.

Dr. Johnston is a Fellow of the Royal Anthropological Institute, the Society
for Applied Anthropology, and the Inter-University Seminar on Armed Forces
and Society.

161



Part IV: Notes on Methodology

Chapter Ten: Survey Methodology 119
Methodology .. 120
Demographics 124

Chapter Eleven: Q-Sort Methodology ....

Chapter Twelve: The “File-Drawer” Problem and
Calculation of Effect Size ....

.. 133

Intelligence Tools and Techniques .. 133
Cognitive Processes and Intelligence 134
Tools and Techniques as Cognitive Processes 134
Intelligence Analysis as Individual Cognitive Process
Error .
Language and Cognition .

Appendix: Selected Literature ...

» 139

Bibliography...

Published Sources .
Web resources ...

139

Afterword by Joseph Hayes ...
The Author

161



SURVEY METHODOLOGY

content and, thus, are not attributable to any specific individual. This is not
simply the result of security procedures within the Intelligence Community; it
is also the professional obligation of every member of the American Anthro-
pological Association, as stated in the American Anthropological Association
Code of Ethics.*

The interview technique employed in this study was semi-structured. Sev-
eral specific questions about the participant’s perception of the nature of intel-
ligence, the analytic process, the intelligence production cycle, and
intelligence errors and failures were standard throughout the interviews. Other
questions, specific to the individual’s job responsibilities, were tailored to
each respondent. This method allowed for a more open-ended approach,
which surveys and highly structured interviews do not. The semi-structured
method is more akin to an open conversation (with consistent data collection
constructs and probing questions) than to a formal interview, which helps put
the respondents at ease and makes the entire process seem somewhat less con-
trived.

Access to interview participants was made possible through the Center for
the Study of Intelligence. Individuals at CSI introduced me to their contacts
throughout the Intelligence Community, including active and retired senior
analysts, managers and senior leadership, as well as to academics and
researchers. The various intelligence-training centers put me in touch with
new hires and novice analysts. Each interviewee was asked to make recom-
mendations and provide contact information for others who might be inter-
ested in participating in this research project. In addition, numerous
interviewees were approached without a formal or informal introduction from
a previous participant. Only four of the 489 individuals contacted to date have
declined to participate in this study. This constitutes a participation rate of
greater than 99 percent, which is unusually high for this type of research.
Although a participation rate this high may be an artifact of the sampling
method or of an organizational pressure to participate, it also may indicate a
general desire within the Intelligence Community to support performance
improvement research,

Unlike random sampling, purposive sampling is an attempt to collect data
from specific data sources. In anthropological studies, purposive sampling is
regularly used to address specific issues and to answer specific questions.
Normally, this approach requires finding a “key informant” or someone on the
inside of a specific culture who will become the researcher’s ally and access
agent. In this particular study, the CSI staff acted as access agents to the Intel-
ligence Community at large.

5 American Anthropological Association, Code of Ethics of the American Anthropological Asso-
ciation.



CHAPTER TWELV)
The “File-Drawer” Problem and Calculation of Effect Size

a

The file-drawer problem appears to have two causes: the reluctance of
researchers to report their null results and the reluctance of professional jour-
nal editors to include studies whose results fail to reach statistical signifi-
cance. Such studies remain in the “file-drawers” of the researchers. How
much would these inaccessible studies affect the results of our meta-analysis?
The answer seems to be not much.'

Effect Size - Difference Between Two Means

Mean
Effect Size
4

PL, tsi

Technology

Conventional
Instruction

50th Xth

Gene Glass, and Barry McGaw, “Choice of the Metric for Effect Size in Meta-Analysis; Larry
Hedges, “Estimation of Effect Size from a Series of Independent Experiments”; Larry Hedges and
Ingram Olkin, “Vote-Counting Methods in Research Synthesis.”

129



APPENDIX

Sherman Kent, “Words of Estimative Probability,” Sherman Kent and the
Board of National Estimates: Collected Essays (Washington, DC: Center for
the Study of Intelligence, 1994).

Lisa Krizan, Intelligence Essentials for Everyone (Washington, DC: Joint Mil-
itary Intelligence College, 1999).

Intelligence Analysis as Individual Cognitive Process

Alexander Butterfield, The Accuracy of Intelligence Assessment: Bias, Per-
ception, and Judgment in Analysis and Decision (Newport, RI: Naval War
College, 1993).

Richards J. Heuer, Jr., Psychology of Intelligence Analysis (Washington, DC:
Center for the Study of Intelligence, 1999).

Robert Jervis, Perception and Misperception in International Politics (Prince-
ton: Princeton University Press, 1977).

Ephraim Kam, Surprise Attack: The Victim’s Perspective (Cambridge, MA:
Harvard University Press, 1988).

Geraldine Krotow, The Impact of Cognitive Feedback on the Performance of
Intelligence Analysts (Monterey, CA: Naval Postgraduate School, 1992).

Mark Lowenthal, Intelligence: From Secrets to Policy (Washington, DC: CQ,
2000).

David Moore, Creating Intelligence: Evidence and Inference in the Analysis
Process. Unpublished Masters Thesis (Washington DC: Joint Military Intel-
ligence College, 2002).

J. R. Thompson, R. Hopf-Weichel, and R. Geiselman, The Cognitive Bases of
Intelligence Analysis (Alexandria, VA: Army Research Institute, Research
Report 1362, 1984).

Error

Daniela Busse and Chris Johnson “Using a Cognitive Theoretical Framework
to Support Accident Analysis,” in Proceedings of the Working Group on
Human Error, Safety, and System Development (Seattle, WA: International
Federation for Information Processing, 1998, 36-43).

Peter Fishburn, Utility Theory for Decision Making (New York: John Wiley,
1970).

Paul Fitts and Michael Posner, Human Performance (Belmont, CA: Brooks
Cole, 1967).

John Hammond, Ralph Keeney, and Howard Raiffa, “The Hidden Traps in
Decision Making,” Harvard Business Review, Sept— Oct. 1998: 47-58.

135



CHAPTER TWO

satisfaction and anxiety by giving larger meaning to an individual’s daily
actions. '®

Finding: Analytic Training

When I started, there wasn't much training available. There were a
few advanced courses, but, for the most part, it was on the job.

A professional identity is generally a disciplinary norm, and it regularly
occurs in other domains that are as cognitively demanding as intelligence
analysis, such as medicine, aeronautics, and jurisprudence. These other
domains practice a general system of professional enculturation that
progresses from a basic education program to specialized training."? These
training programs help to differentiate communities of practitioners from the
general public, create specific and unique professional identities, and develop
basic communication and task-specific skills. They also help the profession to
continue to advance through formal research efforts.

This is not the case within the Intelligence Community as a whole. Gener-
ally, the intelligence agencies that do provide basic and advanced training do
so independently of other intelligence organizations.2” A number of intelli-
gence agencies do not provide basic analytic training at all or have only
recently begun to do so, relying instead on on-the-job experiences and infor-
mal mentoring.

We haven't had a culture of training analysts here in the past. It’s
only in the last year or so that we've started to change that.

When I started here, analysts were considered administrative per-
sonnel. We didn’t have a training program. | think they just started
one this year.

My background was technical analysis, and we had a lot of opera-
tional training where I used to work. But now that I’m doing more
strategic analysis, I’ve had to make it up as I go along.

'8 Philip Cushman, Constructing the Self, Constructing America; Anthony Giddens, Modernity
and Self-Identity; John P. Hewitt, Self and Society; Lewis P. Hinchman and Sandra K. Hinchman,
Memory, Identity, Community; Carl Jung, The Undiscovered Self, George Levine, ed., Construc-
tions of the Self.

' Enculturation is the process or mechanism by which a culture is instilled in a human being from
birth until death. In this instance, professional enculturation refers to the acquisition of a profes-
sional identity through specific cultural rituals and practices, as displayed, for example, by practi-
tioners who have graduated from medical school, law school, and basic military training.

% See footnote 7 in the Introduction for several recent cross-agency training initiatives.

28



Findings Based on Systems Analysis ..
Recommendations

Part II: Potential Areas for Improvement

Chapter Five: Integrating Methodologists into Teams of Experts

Becoming an Expert ..
The Power of Expertise .
The Paradox of Expertise
The Burden on Intelligence Analysts ..
The Pros and Cons of Teams
Can Technology Help? .......
Analytic Methodologists
Conclusion

Chapter Six: The Question of Foreign Cultures:
Combating Ethnocentrism in Intelligence Analysis .

Case Study One: Tiananmen Square
Case Study Two: The Red Team
Conclusion and Recommendations

Chapter Seven: Instructional Technology:
Effectiveness and Implications for the Intelligence Community

Background ...
Meta-analysis Demonstrates the Effectiveness of

Instructional Technology ...
Current Research on Higher Cognitive Abilities .
Discussion
Conclusion ....

Chapter Eight: Organizational Culture:
Anticipatory Socialization and Intelligence Analysts .....

Organizational Socialization ...
Anticipatory Socialization ....
Consequences of Culture Mismatch
Anticipatory Socialization in the Intelligence Community
Conclusion and Recommendations .....

Chapter Nine: Recommendations ...

The First Step: Recognizing A Fundamental Problem
Performance Improvement Infrastructure ....
Infrastructure Requirements .
Research Programs .....
The Importance of Access

viii



CHAPTER TWO

described by participants during the interviews and observed during direct and
participant observations throughout the fieldwork.

Analysts were asked to describe the work processes they employed to
answer questions, solve problems, describe and explain phenomena, make
forecasts, and develop intelligence products. The process they described
began with an examination of previous analytic products developed by their
organization in order to establish a baseline from which they could build their
own analysis.

When a request comes in from a consumer to answer some question,
the first thing I do is to read up on the analytic line.

The first thing I do is check the pervious publications, and then I
sort through the current traffic.

I’ve looked at our previous products, and I’ve got a good idea of the
pattern; so, when I sort through the traffic, I know what I’m trying
to find.

I try to keep up with all the products that come out of our area, sof
know where to start my piece.

A literature search is often the first step in any research endeavor. The util-
ity of this practice is not merely to define and understand the current state of
research in the field but also to determine major controversies and divergences
of opinion. Trying to discern controversies and divergence in intelligence
products is often difficult, because some of them—national intelligence esti-
mates (NIE), in particular—are specifically designed to produce a corporate
consensus for an audience of high-level policymakers.

These products can and do include divergent opinions, in the form of foot-
notes, but these tend to indicate inter-, rather than intra-, organizational differ-
ences. Dissenting footnotes are products of the coordination process, the result
of an inability on the part of one or several community organizations to con-
vince the others of a particular point of view. Not surprisingly, the least prob-
able opinion is often the hardest to defend, whereas the most probable opinion
is the easiest to support.

The literature search approach may promote a logical consistency among
analytic products, but it has the unintended consequence of imposing on the
analyst using it a preexisting mental model of the phenomena in question. The
existing analytic products describe, implicitly or explicitly, a set of working
hypotheses that an analyst may wish to reflect in his or her own work. Of
course, these existing hypotheses are rarely tested each time they are incorpo-
rated into new products. What tends to occur is that the analyst looks for cur-
rent data that confirms the existing organizational opinion or the opinion that

22



CHAPTER EIGHT

Interestingly, research suggests that the relationship between satisfaction
and task performance, although positive, is not especially strong.'? Thus,
while job satisfaction may be important to the longevity of any individual
career cycle, it is not a major factor in individual job performance. It does,
however, increase absenteeism, which has a negative effect on overall organi-
zational productivity.

Absenteeism and Turnover. Research indicates that the lower an individ-
ual’s job satisfaction, the more likely he or she is to be absent from work.'4 As
with job satisfaction and task performance, this relationship is modest but also
statistically significant. An employee may even choose to leave an organiza-
tion altogether. This voluntary resignation is measured as employee turnover
and has fiscal consequences for both the individual and the organization.

Fiscal Cost. Employee turnover is a critical cost element. The expense of
recruiting and training new employees, along with lost productivity from
vacant positions and overtime pay for replacement workers, increases operat-
ing costs and also reduces employee organizational output.

A 2002 study by the Employment Policy Foundation found that the esti-
mated turnover cost is $12,506 per year per full-time vacancy for the average
employee with total compensation (wages and benefits) of $50,025.!5 As the
average annual turnover benchmark within the Fortune 500 is 23.8 percent,
one can clearly see how critical it is for organizations to lessen the number of
employees who leave voluntarily. Even unscheduled absences can be expen-
sive—averaging between $247 and $534 per employee, per day, according to
the same study.

Anticipatory Socialization in the Intelligence Community

The secrecy is strange. I thought it would be romantic, but it turns
out that it is just strange.

Iwas sold on the cool factor. It’s still sort of cool, I guess.

Accepting a job with one of the 14 members of the Intelligence Community
differs from other professions in that it is difficult for new employees to have a
clear and precise understanding of the roles and responsibilities they are about

8 The correlation is 0.17 according to Michelle T. laffaldano and Paul M. Muchinsky in their
“Job Satisfaction and Job Performance: A Meta-Analysis.”

“ Lyman W, Porter et al., “Organizational Commitment, Job Satisfaction and Turnover Among
Psychiatric Technicians.”

‘5 This number does not take into account the additional costs within the Intelligence Community
for background and security investigations.

102



PART I

Research Findings

a



INTRODUCTION

is so small and the selection and training processes are so demanding.‘ Intelli-
gence analysts share many of these organizational and professional circum-
stances.

The Intelligence Community is relatively small, highly selective, and
largely shielded from public view. For its practitioners, intelligence work is a
cognitively-demanding and high-risk profession that can lead to public policy
that strengthens the nation or puts it at greater risk. Because the consequences
of failure are so great, intelligence professionals continually feel significant
internal and external pressure to avoid it. One consequence of this pressure is
that there has been a long-standing bureaucratic resistance to putting in place a
systematic program for improving analytical performance. According to
71 percent of the people I interviewed, however, that resistance has dimin-
ished significantly since September 2001.

It is not difficult to understand the historical resistance to implementing
such a performance improvement program. Simply put, a program explicitly
designed to improve human performance implies that human performance
needs improving, an allegation that risks considerable political and institu-
tional resistance. Not only does performance improvement imply that the sys-
tem is not optimal, the necessary scrutiny of practice and performance would
require examining sources and methods in detail throughout the Intelligence
Community. Although this scrutiny would be wholly internal to the commu-
nity, the concept runs counter to a culture of secrecy and compartmentaliza-
tion.

The conflict between secrecy, a necessary condition for intelligence, and
openness, a necessary condition for performance improvement, was a recur-
ring theme I observed during this research. Any organization that requires
secrecy to perform its duties will struggle with and often reject openness, even
at the expense of efficacy. Despite this, and to their credit, a number of small
groups within the Intelligence Community have tasked themselves with creat-
ing formal and informal ties with the nation’s academic, non-profit, and indus-
trial communities. In addition, there has been an appreciable increase in the
use of alternative analyses and open-source materials.

These efforts alone may not be sufficient to alter the historical culture of
secrecy, but they do reinforce the idea that the Intelligence Community itself
has a responsibility to reconsider the relationship between secrecy, openness,
and efficacy. This is especially true as it relates to the community’s perfor-
mance and the occurrence of errors and failure. External oversight and public
debate will not solve these issues; the desire to improve the Intelligence Com-

+ There are currently 109 active US astronauts and 36 management astronauts. See National Aero-
nautics and Space Administration-Johnson Space Center career astronaut biographies.



ACKNOWLEDGEMENTS

There are literally slightly more than 1,000 people to thank for their help in
developing this work. Most of them I cannot name, for one reason or another,
but my thanks go out to all of those who took the time to participate in this
research project. Thank you for your trust. Particular thanks are due the other
researchers who coauthored chapters: Judith Meister Johnston, J. Dexter
Fletcher, and Stephen Konya.

There is a long list of individuals and institutions deserving of my gratitude,
but, at the outset, for making this study possible, I would like to express my
appreciation to Paul Johnson and Woody Kuhns, the chief and deputy chief of
the Central Intelligence Agency’s Center for the Study of Intelligence, and
their staff for their support and to John Phillips and Tom Kennedy of the Intel-
ligence Technology Innovation Center who, along with their staff, administer
the Director of Central Intelligence Postdoctoral Fellowship Program.

I would like to thank Greg Treverton and Joe Hayes for their help through-
out this project and for their willingness to give of their time. Dr. Forrest
Frank, Charles Perrow, and Matthew Johnson deserve recognition for the
material they contributed. Although it was not possible to cite them as refer-
ences for those contributions, these are indicated in the footnotes, and I give
them full credit for their work and efforts.

I would also like to thank Bruce Berkowitz, Mike Warner, Fritz Ermarth,
Gordon Oehler, Jeffrey Cooper, Dave Kaplan, John Morrison, James Wirtz,
Robyn Dawes, Chris Johnson, Marilyn Peterson, Drew Cukor, Dennis
McBride, Paul Chatelier, Stephen Marrin, Randy Good, Brian Hearing, Phil
Williams, Jonathan Clemente, Jim Wilson, Dennis, Kowal, Randy Murch,
Gordie Boezer, Steve Holder, Joel Resnick, Mark Stout, Mike Vlahos, Mike
Rigdon, Jim Silk, Karl Lowe, Kevin O’Connell, Dennis Gormley, Randy
Pherson, Chris Andrew, Daniel Serfaty, Tom Armour, Gary Klein, Brian
Moon, Richard Hackman, Charlie Kisner, Matt McKnight, Joe Rosen, Mike
Yared, Jen Lucas, Dick Heuer, Robert Jervis, Pam Harbourne, Katie Dorr,



CHAPTER NINE

Development and Validation of Analytic Methods. The Intelligence Com-
munity routinely generates ad hoc methodologies to solve specific analytic
problems or crises. However, once the problem has been solved, or the crisis
averted, the new analytic method may or may not become part of the institu-
tional memory. Often these new methods are lost and need to be re-created to
address the next problem or crisis. In addition, these methods are seldom
tested against other competing analytic methods for validity or reliability. It is
difficult for an analyst to know which analytic method to employ in a given
situation or requirement.

There are obvious inefficiencies in the current model. First, there is the loss
of corporate knowledge each time an innovative analytic method is generated
and subsequently abandoned. Second, there is no effectiveness testing center
where analytic methods can be compared for specific cases. Although there
are hundreds of analytic strategies, there is no way to determine which strat-
egy is the most effective for any particular problem set.

The lack of an analytic methodology research agenda leads analysts to
choose methods with which they are most familiar or to choose those dictated
by circumstance, such as deadlines. Moreover, instead of advancing the con-
cept that intelligence analysis is science and needs to be engaged in like any
other scientific discipline, the paucity of effectiveness data supports a deep-
seated community bias that analytic methods are idiosyncratic and, therefore,
akin to craft.

The development of a research agenda for analytic methodology that is
focused on collecting effectiveness and validation data is the first step in mov-
ing intelligence analysis from a tradecraft model to a scientific model. This
may be the most culturally difficult recommendation to implement: there is
cultural resistance to adopting a science-based model of intelligence analysis
that is rooted in the traditions, norms, and values of the Intelligence Commu-
nity. Another difficult step will be to introduce effectiveness data and corre-
sponding analytic methods to the community at large and to incorporate these
in future training programs.

Training Effectiveness. Successful analysis demands group cohesion and
the implementation of consistent, effective analytic methods within the Intelli-
gence Community. The best way to achieve this is through formal basic and
advanced training programs. As noted earlier, several agencies within the
community have invested resources in formal training programs, but these
programs are unique to each agency and are often missing evaluations of stu-
dent performance. Although most formal courses include a written subjective
evaluation of the instructor, as well as the student’s perception of the value of
the course, the evaluation of student performance has yet to be formalized.

14



CHAPTER SEVEN

the procedure’s applicability in unfamiliar situations, modify it as needed, and
use it to develop new approaches and procedures. Early on, Bloom discussed
learning objectives as a hierarchy beginning with knowledge at the most rudi-
mentary level and ascending through comprehension, application, analysis,
and synthesis to evaluation.'7 Bloom’s is not the only such hierarchy to
emerge from research on instructional design, but it seems to be the best
known, and it describes as well as any the various levels of knowledge, skill,
and ability to which learners may aspire.

Current Research on Higher Cognitive Abilities

Analysts have begun to discuss development of the higher cognitive abili-
ties needed to deal with unanticipated and novel challenges.'* Components of
such “cognitive readiness” may include:

* Situation awareness—the ability to comprehend the relevant aspects of a
situation and use this understanding to choose reasonable courses of
action." Practice and feedback in complex, simulated environments have
been shown to improve situation awareness.

Memory—the ability to recall and/or recognize patterns in a situation that
lead to likely solutions. It may be supported by two underlying theoretical
mechanisms: encoding specificity, which stresses the importance of
responding to relevant external and internal perceptual cues, and transfer-
appropriate processing,?' which stresses the actions performed during
encoding and retrieval. Some instructional techniques, such as overlearn-
ing,22 have been shown to enhance long-term retention.”

Transfer—the ability to apply what is learned in one context to a different
context. It can be perceived either as the ability to select and apply proce-
dural knowledge gained in one context to another (“low road” transfer) or
as the ability to apply the principles abstracted from a set of contexts to
another (“high road” transfer).”* Extensive practice, with feedback, will

17 Benjamin. S. Bloom, Taxonomy of Educational Objectives.
"8 J, E, Morrison, and J. D. Fletcher, Cognitive Readiness.

 M. R. Endsley, “Design and Evaluation for Situation Awareness Enhancement.”

2, Tulving and D. M. Thomson, “Encoding Specificity and Retrieval Processes in Episodic

. D. Bransford, and J. J. Franks, “Level of Processing Versus Transfer-Appropri-

ate Processing.”
2 The use of specific problem-solving methods repetitively.

2 R. A. Wisher, M. A. Sabol, and J. A. Ellis Staying Sharp: Retention of Military Knowledge and
Skills.

92



TESTING THE INTELLIGENCE CYCLE

relationships with each other, the flow of the process, and phenomena that
influence the elements and the flow. The model uses four icons to represent
actions and relationships within the system: stocks, flows, converters, and
connectors. The icons and their placement within the systems model show the
relationships of the elements of the analyzed phenomenon.

The Components of the Systems Model

Tcon Purpose

Stock ‘Stocks represent accumulations. These are quantities
that can increase or decrease, such as the amount of
work that needs to be completed, the time available in
which to do it, experience one might bring to a task.

Flows represent activities. They control the filling or
O-—O8 draining of stocks, causing conditions to change.
Flow
Converters change inputs into outputs. They usually
O represent the variables that initiate change. In the
Converter example, a converter might represent a sudden and

drastic world event.

Stock | Connectors link elements to other elements,
representing assumptions about what depends on
what.

Flow

Converter

The systems model of the Intelligence Cycle provides insights into the process
of analysis as well as other factors that can influence the successful and timely
completion of an intelligence task. It also provides a way to understand the
impact of change in any area of the Intelligence Cycle on other elements, either
through reflection or by applying mathematical values to the influences and rela-
tionships and running simulations of the model.

Demand. As in the traditional Intelligence Cycle model, the systems model
begins with requirements for information that generally come from policy-
makers. These requirements are represented by a stock (found in the upper
left-hand quarter of the diagram) because they can increase or decrease based
on the level of need for information (a flow). The change in level of need is
influenced by national and world events, as well as by new questions or
requests for clarification of items in previously delivered products. Each
request does not contribute equally to the amount of work, which is influenced
by the types of documents or products requested, the complexity of the prod-

St



CHAPTER ONE

that consumers of intelligence products require a static “final say” on a given
topic in order to generate policy. This sort of organizational-historical context,
coupled with the impression that consumers must have a final verdict, tends to
create and reinforce a risk-averse culture.

Once the organizational context for answering any given question is under-
stood, the analyst begins to consider raw data specific to answering the new
question. In so doing, the analyst runs the risk of confirmation biases. That is,
instead of generating new hypotheses based solely on raw data and then
weighing the evidence to confirm or refute those hypotheses, the analyst
begins looking for evidence to confirm the existing hypothesis, which came
from previous intelligence products or was inferred during interactions with
colleagues. The process is reinforced socially as the analyst discusses a new
finding with group members and superiors, often the very people who collab-
orated in producing the previous intelligence products. Similarly, those who
review the product may have been the reviewers who passed on the analyst’s
previous efforts.

This is not to say that the existing intelligence products are necessarily inac-
curate. In fact, they are very often accurate. This is merely meant to point out
that risk aversion, organizational-historical context, and socialization are all
part of the analytic process. One cannot separate the cognitive aspects of intel-
ligence analysis from its cultural context.

Definition 3: Intelligence errors are factual inaccuracies in analy-
sis resulting from poor or missing data; intelligence failure is sys-
temic organizational surprise resulting from incorrect, missing,
discarded, or inadequate hypotheses.

During interviews, participants were asked to explain their understanding of
the terms intelligence error and intelligence failure. There was little consensus
regarding the definitions of error and failure within the Intelligence Commu-
nity or within the larger interview sample. Here are some sample responses:

I don’t know what they mean.
There are no such things. There's only policy failure.

You report what you know, and, if you don't know something, then it
isn't error or failure. It’s just missing information.

Failure is forecasting the wrong thing.
Failure is reporting the wrong thing.

Error is forecasting the wrong thing.



SURVEY METHODOLOGY

group includes all those currently working in the Intelligence Community
with more than two years of experience. The “retired” sub-group includes
those who have spent more than fifteen years in the intelligence profession
and have since gone on to either full retirement or other organizations outside
of the Intelligence Community.

Of the 345 intelligence professionals interviewed, 20 percent were novices,
65 percent were active, and 15 percent were retired. The active and retired
sub-groups include senior managers.

Interview Categories and Numbers

Category N
Intelligence Professionals 345
Novice (60)
Active (233)
Retired (52)
‘Academics B
Technology Researchers 33
Consumers 18
Total Interviewed 489

In order to assure anonymity for the participants, I have created broader job-
related functional categories and associated the number of individuals inter-
viewed with the broader categories rather than linking them to specific organiza-
tions within the Intelligence Community. This is in contrast to aggregating the
agencies according to each agency’s specific mission, process, or product.
Although not an official member of the Intelligence Community, the Drug
Enforcement Administration is included because of its intelligence function and
resources. The table on the next page shows how I aggregated the agencies into
National-Technical, Defense, and Law Enforcement-Homeland Security catego-
ries according to the professional functions of interview participants.

© The use of two years as a divide between novice and active is derived from the total amount of
experience it is possible to gain in that time. See the discussion of expertise in Chapter Five

125



as “CNN plus secrets.” Johnston doesn’t put it quite that way, but the Intelli-
gence Community does more reporting than in-depth analysis.

None of the analytic agencies knows much about the analytic techniques of
the others. In all, there tends to be much more emphasis on writing and com-
munication skills than on analytic methods. Training is driven more by the
druthers of individual analysts than by any strategic view of the agencies and
what they need. Most training is on-the-job.

Johnston identifies the needs for analysis of at least three different types of
consumers—cops, spies, and soldiers. The needs of those consumers produce
at least three distinct types of intelligence—investigative or operational, stra-
tegic, and tactical.

The research suggests the need for serious study of analytic methods across
all three, guided by professional methodologists. Analysts should have many
more opportunities to do fieldwork abroad. They should also move much
more often across the agency “stovepipes” they now inhabit. These move-
ments would give them a richer sense for how other agencies do analysis.

Together, the analytic agencies should aim to create “communities of prac-
tice,” with mentoring, analytic practice groups, and various kinds of on-line
resources, including forums on methods and problem solving. These commu-
nities would be linked to a central repository of lessons learned, based on
after-action post-mortems and more formal reviews of strategic intelligence
products. These reviews should derive lessons for individuals and for teams
and should look at roots of errors and failures. Oral and written histories
would serve as other sources of wherewithal for lessons. These communities
could also begin to reshape organizations, by rethinking organizational
designs, developing more formal socialization programs, testing group config-
urations for effectiveness, and doing the same for management and leadership
practices.

The agenda Johnston suggests is a daunting one, but it finds echoes in the
work of small, innovative groups across the Intelligence Community—groups
more tolerated than sponsored by agency leaders. With the challenge work-
force demographics poses for the Community—the “gray-green” age distribu-
tion, which means that large numbers of new analysts will lack mentors as old
hands retire—also comes the opportunity to refashion methods and organiza-
tions for doing intelligence analysis. When the finger-pointing in Washington
subsides, and the time for serious change arrives, there will be no better place
to start than with Rob Johnston’s fine book.



SURVEY METHODOLOGY

might merely serve to emphasize the small size of Intelligence Community. In
any case, this approach to sampling may help to ameliorate the sampling bias
inherent in qualitative research.

In addition to semi-structured interviews, both direct and participant obser-
vation data collection methods were employed. The direct observation method
involved watching Intelligence Community analysts perform their tasks in
both actual and training environments, recording the physical and verbal inter-
actions they had with one another, and observing the steps used to create intel-
ligence products. Direct observation occurred over the course of two years by
observing 325 individual analysts and teams of analysts performing their spe-
cific tasks. The data collected from observing the 325 analysts were not
included in the semi-structured interview data because I did not use the formal
semi-structured interview process to structure those interactions. These obser-
vational data were recorded separately in field notes and used for triangulating
the findings from the interviews.

The participant observation method is employed to give the researcher a
“first-person” understanding of the context and nuances associated with a task
and the culture in which that task occurs. Although the researcher possesses
only an approximation of the knowledge and understanding of the actual prac-
titioners of the task and their culture, this “first-person” perspective can lead
the researcher to new insights and new hypotheses.

During this study, the participant observation was conducted during ana-
lytic production cycles, scenario development, and red cell exercises. This
included monitoring my own analytic strategies, the analytic strategies of oth-
ers as diagramed or verbalized, the physical and verbal social interactions
among the participants, the environment in which the tasks occurred, and the
steps used to create a final intelligence product. These data, along with notes
on social dynamics, taboos, and social power, were recorded in field notes and
created a separate data source for triangulation.

With modern anthropology, these data normally would be captured on film,
audiotape, or in some digital format. Due to the security requirements of the
Intelligence Community, however, the data were captured only in the written
form of field notes. As is the case with the field notes, the identity of the inter-
view participants will not be disclosed. This is in keeping with both the secu-
rity practices of the Intelligence Community and the professional standards
described in the American Anthropological Association Statement on the
Confidentiality of Field Notes.”

The data from the interviews were analyzed using a method called interpre-
tational analysis. This approach included segmenting the interview data into

7 American Anthropological Association, Statement on the Confidentiality of Field Notes

123



FINDINGS

foreknowledge that this researcher has a number of personal and professional
biases.*

Finding: Secrecy Versus Efficacy

Secrecy and efficacy conflict. Secrecy interferes with analytic effectiveness
by limiting access to information and sources that may be necessary for accu-
rate or predictive analysis. In turn, openness interferes with security by
degrading the value of information resources and by revealing specific
sources and methods.

Perfect secrecy would ultimately be unproductive, because it would
restrict information to one mind or to a very small group of minds. Limiting
available resources in this way would produce organizational failure in
competition with resources available to a large and diverse group of adver-
saries. Perfect openness would also lead to organizational failure, because,
with full access to all information, there would never be an instance of
advantage for any one group over any other group. In addition, perfect
openness would result in adversaries being aware they are under observa-
tion and could lead them to alter their behavior to deceive the observer if
they so desired.*

Between these two extremes, there is some notional point where secrecy
and openness converge to create an optimal performance tradeoff. My percep-
tion is that, within the Intelligence Community, more organizational emphasis
is placed on secrecy than on effectiveness. It is important, in my view, that
there be a voice in favor of openness to counterbalance the many voices
whose sole or primary responsibility is the advocacy and maintenance of

+ Throughout the project, my data collection method consisted of written field notes. Anthrapolo-
gists traditionally include specific detail from participant input or direct observation, Uswally, this is
in the form of precise descriptions of the actual behavior of participants and transcripts of their ver-
bal interactions. It is also standard practice in field work to capture these data, and the data from the
interviews and focus groups, on audio- or videotape. These practices were not followed in this par-
ticular case for two reasons: first, the nature of my work was not to document actual practices and
procedures; rather, it was to derive categories of variables and individual variables in order to create
a taxonomy, and to use the prototype taxonomy to structure the interactions; second, the nature of
intelligence work and the environment in which it occurs, as well as its professional practitioners,
require that certain data be restricted.

5 This has been demonstrated in the psychological literature and is referred to as the Hawthorne
Effect. Derived from research that began with an experimental program at Western Electric’s
Hawthome Works conducted between 1927 and 1930, the Hawthorne Theory, broadly inter-
preted, states that the behavior of subjects changes when they are aware of being observed. See
Fritz J. Roethlisberger and William J. Dickson, Management and the worker; Elton Mayo, The
Human Problems of an Industrial Civilization.



BIBLIOGRAPHY

Kahneman, Daniel, and Amos Tversky. “Prospect Theory: An Analysis of
Decision Under Risk,” Econometrica 47, no. 2 (1979): 263-91.

., Paul Slovic, and Amos Tversky. Judgment Under Uncertainty: Heu-
ristics and Biases. Cambridge: Cambridge University Press, 1982.

Kam, Ephraim. Surprise Attack. The Victim’s Perspective. Cambridge: Har-
vard University Press, 1988.

Kaufman, David L., Jr. Introduction to Systems Thinking. Minneapolis: Future
Systems, 1978.

Kaufman, Roger. “A Holistic Planning Model: A Systems Approach for
Improving Organizational Effectiveness and Impact,” Performance and
Instruction Journal, October 1983: 3-10.

Kent, Sherman. Strategic Intelligence for American World Policy. Princeton:
Princeton University Press, 1966.

. “Words of Estimative Probability,” Sherman Kent and the Board of
National Estimates: Collected Essays. Washington, DC: Center for the Study
of Intelligence, 1994.

Kirk, Jerome, and Marc Miller. Reliability and Validity in Qualitative
Research. Qualitative Research Methods, Volume 1, Newbury Park, CA:
Sage, 1986.

Kirkpatrick, Lyman B., Jr. Captains Without Eyes: Intelligence Failures in
World War II. London: MacMillan Company, 1969.

Klahr, D., and H.A. Simon. “What Have Psychologists (and Others) Discov-
ered About the Process of Scientific Discovery?” Current Directions in Psy-
chological Science 10.

Klein, Gary, Roberta Calderwood, and A. Clinton-Cirocco. Rapid Decision
Making on the Fire Ground. Yellow Springs, OH: Klein Associates, 1985,
KA-TR-84-41-7 (prepared under contract MDA903-85-G-0099 for the US
Army Research Institute, Alexandria, VA).

Knight, Frank H. Risk, Uncertainty and Profit. Boston: Houghton Mifflin,
1921.

Knorr, Klaus. “Foreign Intelligence and the Social Sciences.” (Research
monograph, Center of International Studies, Princeton University, 1964.)

Krizan, Lisa. Intelligence Essentials for Everyone. Washington, DC: Joint
Military Intelligence College, 1999.

Krotow, Geraldine. The Impact of Cognitive Feedback on the Performance of
Intelligence Analysts. Monterey, CA: Naval Postgraduate School, 1992, AD-
A252.

Kuhn, Thomas. The Structure of Scientific Revolutions. Chicago: The Univer-
sity of Chicago Press, 1970.

Kulik, Chen-Lin, and James Kulik. “Effectiveness of Computer-Based Educa-
tion in Colleges,” AEDS Journal, Winter/Spring 1986: 81-108.

148



CHAPTER SIX

In contrast, 25 percent responded that democracy was guided by the center
(the party and the cadres), 19.5 percent that democracy meant that the govern-
ment would solicit people’s opinions (the party would ask people what they
thought), and 11 percent that democracy meant the government would make
decisions for the people based on the people’s interests but not including the
people’s direct vote. These three responses were more in line with then-current
party doctrine and tended to be from individuals over 36 years of age living in
both urban and rural settings. This was the same demographic that experienced
the Cultural Revolution.

The election of representatives and the division and limitation of those rep-
resentatives’ power—what I would have considered to be two key aspects of
democracy—were chosen by 10 percent of the sample, only slightly larger
than the 6.3 percent of Chinese respondents who reported that they didn’t
know what the word “democracy” meant. My own perception of democracy
fit with a young, urban, elite, college educated population, not with the major-
ity of Chinese citizens.

There was a very small sample of citizens in Tiananmen Square demanding
what looked and sounded like my American version of democracy. Yet, how-
ever much the students’ message resonated in the West, it did not do so in
China. My expectations notwithstanding, there was a cognitive disconnect
between students and average citizens, which, along with the visceral semiot-
ics of the Cultural Revolution, kept the two apart.’ It was not just the message
that had kept people in their homes during the PLA siege on Tiananmen; it
was also the messengers.

The label “ethnocentrism” might be accurate, but it does not diagnose the
root of the problem. I did not use a variety of tools or techniques to question
my underlying assumptions and, therefore, I failed to make an accurate fore-
cast. There were obvious statistical and analytic flaws. The former was princi-
pally a sampling error, both frame and selection bias (the students at
Tiananmen did not represent the general population in Beijing or China at
large). More significant than simple technical or statistical flaws, however, my
frame of reference and my assumptions about meanings, context, and values
(or culture) misled me.

The assumptions I made about the Tiananmen protests were products of my
own enculturation, and I am not convinced that anything short of the experi-
ence of analytic failure would have been sufficient for me to examine the pro-
cess underpinning my reasoning. 1 never would have reexamined my mental
mode without experiencing failure. Failure is an event that is easily remem-

7 Semiosis is the production of cultural signifiers or signs and the cultural or contextual meaning
of those signs. This includes all modes of visual and auditory production.

80



CHAPTER TEN

Relying on such a “social network” sampling method for collecting inter-
view data does pose potential statistical biases.° The likelihood that each new
interviewee was referred to me because of a friendly relationship with a previ-
ous interviewee may mean that those references are “like minded” and not
necessarily representative of the population of intelligence professionals. In
order to counteract that bias, efforts were also made to enlist individuals with-
out any social network-based introduction. The “cold” contacts were informed
of the nature of the research project, its sponsorship, and its goals, given refer-
ence information for verification, and then invited to participate. The “cold”
contact interviewees were also asked to make recommendations and provide
contact information for others who might be interested in participating in the
study.

This strategy was used in an attempt to reduce the affects of sampling bias
by generating parallel social network samples. The figure below is a visual
representation of a parallel social-network sampling model. The central, or
first-order, node on the left is a “cold” contact or unknown individual who rec-
commends several second-order contacts, each represented as a node within the
left box. The second-order “cold” contacts then make additional recommenda-
tions for third-order contacts, and so on. The central (first-order) node on the
right is a “hot” contact or a known individual who recommends several sec-
ond-order contacts, each represented as a node within the right box. The sec:
ond-order “hot” contacts
then make recommenda-
tions for third-order con-
tacts, and so on.

Social Network Mapping

In many instances, the
contacts from both social
network samples over-
lapped or converged on
specific individuals, as
represented by the over-
lapped fourth-order nodes
in the central column.
There are several possible
explanations for this con-
——— BS vergence. It may indicate

that there are a number of
respected “thought leaders” in the Intelligence Community whom each con-
tact believed I should interview for this project, or the convergence of nodes

© Social network sampling is also known as “snowball” sampling in sociology
and psychology.



INTEGRATING METHODOLOGISTS

done to advance methodological science. Methodological improvements are
left primarily to the individual domains, a practice that risks falling into the
same paradoxical trap that currently exists. What is needed is an intelligence-
centric approach to methodology that will include the methods and procedures
of many domains and the development of heuristics and techniques unique to
intelligence. In short, intelligence analysis needs its own analytic heuristics
that are designed, developed, and tested by professional analytic methodolo-
gists.

The desired outcome would be a combined approach that includes formal
thematic teams with structured organizational principles, technological sys-
tems designed with significant input from domain experts, and a cadre of ana-
lytic methodologists. These methodologists would act as in-house consultants
for analytic teams, generate new methods specific to intelligence analysis,
modify and improve existing methods of analysis, and promote the profes-
sionalization of the discipline of intelligence. Although, at first, developing a
cadre of analytic methodologists would require using specialists from a vari-
ety of other domains and professional associations, in time, the discipline
would mature into its own subdiscipline with its own measures of validity and
reliability.

73



INTRODUCTION

In August 2001, I accepted a Director of Central Intelligence postdoctoral
research fellowship with the Center for the Study of Intelligence (CSI) at the
Central Intelligence Agency. The purpose of the fellowship, which was to
begin in September and last for two years, was to identify and describe condi-
tions and variables that negatively affect intelligence analysis. During that
time, I was to investigate analytic culture, methodology, error, and failure
within the Intelligence Community using an applied anthropological method-
ology that would include interviews (thus far, there have been 489), direct and
participant observation, and focus groups.

I began work on this project four days after the attack of 11 September, and
its profound effect on the professionals in the Intelligence Community was
clearly apparent. As a whole, the people I interviewed and observed were
patriotic without pageantry or fanfare, intelligent, hard working, proud of their
profession, and angry. They were angry about the attack and that the militant
Islamic insurgency about which they had been warning policymakers for
years had murdered close to 3,000 people in the United States itself. There
was also a sense of guilt that the attack had happened on their watch and that
they had not been able to stop it.

Having occurred under the dark shadow of that attack, this study has no
comparable baseline against which its results could be tested, and it is difficult
to identify biases that might exist in these data as a result of 11 September. In
some ways, post-9/11 data may be questionable. For example, angry people
may have an ax to grind or an agenda to push and may not give the most reli-
able interviews. Yet, in other ways, post-9/11 data may be more accurate.
When people become angry enough, they tend to blurt out the truth—or, at
least, their perception of the truth. The people I encountered were, in my judg-

xiii



COMBATING ETHNOCENTRISM

At the height of the Cultural Revolution, any dissent was sufficient to bring
accusations of counterrevolutionary sympathies and to qualify one for “re-
education,” which could mean public denunciation, job loss, incarceration,
forced labor, relocation, and even murder, torture, and rape. The traditional
values of respect and honor were replaced with violence and terror, and the
historical social unit of the family had been disrupted and replaced with the
cult of Mao.

For those who had lived through the Cultural Revolution, the student chal-
lenge to the government in Tiananmen in 1989 was also a challenge to social
order and stability. The people we interviewed remembered, correctly or not,
that the faction of the Communist Party then in power and the PLA had
stopped the Red Guard and the Cultural Revolution, arrested its highest rank-
ing proponents and beneficiaries, the Gang of Four, and eventually restored
order to the nation. The point of view of the people we interviewed was that
the PLA, despite the low social status of soldiers, had stopped the chaos.
Although they did not approve of killing students, the threat of another cul-
tural revolution, democratic or otherwise, was more disturbing to them than
the bloody climax in the square. Social order was the higher virtue.

Tiananmen Square: Discussion

We cut nature up, organize it into concepts, and ascribe significan-
ces as we do, largely because we are parties to an agreement to
organize it in this way—an agreement that holds throughout our
speech community and is codified in the patterns of our language.

Benjamin Whorf5

In 1987, a Chinese academic, Min Qi, performed the first national survey of
Chinese political culture.6 Respondents were asked, among other things, to
select statements that best described their understanding of democracy. Of the
1,373 respondents, 6.6 percent responded that democracy meant that people
could elect their political leaders and 3.4 percent that power was limited and
divided. These replies tended to be from individuals under 25 years of age, in
college, and living in urban centers.

5 Benjamin Whorf, along with fellow anthropologist Edward Sapir, developed the linguistic rela-
tivity hypothesis, asserting that different speech communities had different patterns of thought
Although challenged by linguisv/philosopher Noam Chomsky and others with the Universal
Grammar hypothesis, linguistic relativity still has a significant amount of empirical research sup-
port. Benjamin Whorf, Language, Thought, and Reality.

§Min Qi, Zhongguo Zhengzhi Wenhua [Chinese Political Culture]. Translation courtesy of a
friend of the author who prefers to remain anonymous.

79



CHAPTER SIX

Quite the contrary, each of these is a “virtual” America, an approximation of
life in the United States on some foreign soil, and it is the time spent away
from these institutions that is important.

The red team experience reinforced lessons I learned from my own analytic
failures and biases. Watching the struggle between the man enculturated in the
region of interest and the academic experts was a frustrating experience. It
was clear that the experts would not, or could not, hear what he was saying
and that neither he nor I knew how to get the other experts to listen. I doubt
this communication failure was the result of stubbornness or arrogance on
anyone’s part. It seemed rather that the experts’ thinking naturally defaulted to
their own cultural reference points, which interfered with his attempts to com-
municate his cultural knowledge.

Specific cultural knowledge is a skill and the foundation for forecasting the
behavior and decisionmaking of foreign actors. Acquiring cultural knowledge
should be taken as seriously as learning any other facet of one’s analytic capa-
bilities. Moreover, it is incumbent on analysts to educate their own leadership
and policymakers about the value and utility of cultural knowledge for intelli-
gence analysis.

Conclusion and Recommendations

Ethnocentrism is a normal condition, and it results in analytic bias. The ana-
lytic community and intelligence researchers need to develop tools and tech-
niques to combat analytic ethnocentrism. I believe that using cultural diversity
as a strategy to combat ethnocentrism has much to recommend it.

Security concerns may make it very difficult, if not impossible, to hire peo-
ple who are genuinely representative of a given culture. As an alternative to
focusing on hiring practices, I recommend a formal cultural training program
to facilitate acculturation. The program would include language acquisition
and a classroom segment centered on specific cultures, but it would go beyond
these by having the students go to countries of interest and interact with the

8 Some social action groups have appropriated the words “cultural diversity” from Levi-Strauss
and the French school of structural anthropology as a rallying cry to advance an agenda of equal
access to resources and power. That is, the concept has been politicized, and, invoking the words
“cultural diversity” in a public forum ensures that people will have some emotional reaction. This
is not my intention. The use of the words in this work is meant strictly in the technical sense, spe-
cifically, that is, to refer to individuals whose enculturation occurs among different cultures or
individuals who have experienced acculturation. Acculturation is not specific to any one group,
all people can and do experience acculturation to one degree or another through cultural contact
and cultural diffusion, defined as the spreading of a cultural trait (e.g. material object, idea, or
behavior pattern) from one society to another without wholesale dislocation or migration. More-
over, acculturation can be accomplished purposefully through training and fieldwork.

84



CHAPTER EIGHT

often be hired on the spot. The cultural message in this case is that such
employees are easily let in to and out of the organization.

Another, particularly pertinent example is intelligence work, which requires
that recruits undergo employment screenings unlike those found in most civilian
jobs. Potential CIA analysts must submit to a thorough background investiga-
tion, a polygraph examination, and financial and credit reviews. Further, a bat-
tery of psychological and medical exams must be passed prior to a formal
employment offer. The timeframe for the background check eliminates the pos-
sibility of a rapid hiring decision. Even more important are the nonverbal mes-
sages sent to the recruit that this is a position of secrecy and high importance.

Several sources of information contribute to beliefs about any organization.
Friends or relatives who are already part of the organization might share their
experiences with the person considering employment. Information might also
be acquired from other sources, such as professional journals, magazines,
newspaper articles, television, governmental and private Web sites, public
statements or testimony, and annual reports. While these sources of informa-
tion about an organization are far from perfect (all may contain positive and
negative hyperbole), they are still useful from the point of view of forming
preliminary ideas about what it might be like to work for that organization.

Because competition for highly qualified employees is fierce, successful
recruitment usually involves a skillful combination of salesmanship and diplo-
macy. Recruiters tend to describe their organizations in glowing terms, gloss-
ing over internal problems and external threats, while emphasizing positive
features. The result is that potential employees often receive unrealistically
positive impressions of conditions prevailing in a specific organization. When
they arrive on the job and find that their expectations are not met, they experi-
ence disappointment, dissatisfaction, and even resentment that they have been
misled. In fact, research findings indicate that the less employees’ job expecta-
tions are met, the less satisfied and committed they are and the more likely
they are to think about quitting or actually to do so.”

These negative reactions are sometimes termed entry shock, referring to the
confusion and disorientation experienced by many newcomers to an organiza-
tion. In order to avoid entry shock, it is important for organizations to provide
job candidates with accurate information about the organization. Research
supports the notion that people exposed to realistic job previews later report
higher satisfaction and show lower turnover than those who receive glowing,
but often misleading, information about their companies.'® Moreover, having

* John P. Wanous et al., “The Effects of Met Expectations on Newcomer Attitudes and Behavior:
A Review and Meta-analysis.”

"© Bruce M. Meglino et al., “Effects of Ralistic Job Previews: A Comparison Using an Enhance-
ment and a Reduction Preview.”

100



TESTING THE INTELLIGENCE CYCLE

Systematic Analysis

Many disciplines (for example, business process, organizational manage-
ment, human performance technology, program evaluation, systems engineer-
ing, and instructional systems design) employ specific methods to analyze the
effectiveness of products, programs, or policy implementation. Although they
are often given different, domain-specific names and may involve varying lev-
els of detail, these analytic methods involve the identification of inputs, pro-
cesses, and outputs. Once these elements are identified, the evaluation process
maps the relationships of the inputs, their implementation in processes, and
their impact on intended—as opposed to actual—outputs.5 The reasoning
underlying this approach is that an effective product, result, or action is one
that matches its objectives and that these objectives are reached by processes
that logically lead from the objectives to results. Along the way, existing prac-
tices and barriers to reaching goals effectively can be identified. Finally, inter-
ventions, which can range in complexity from simple job aids to a complete
restructuring of the process, can be proposed and implemented and their
impacts assessed.°

This method of analysis has been employed successfully to evaluate
processes that have characteristics similar to the Intelligence Cycle, and
we use it here to examine the effectiveness of the Intelligence Cycle and
its utility in representing the creation of sound analytic products while
avoiding failure or error.

Findings Based on Systematic Analysis

The Intelligence Cycle is represented visually to provide an easy-to-grasp and
easy-to-remember representation of a complex process. Although this type of rep-
resentation may make the flow of information and the interrelationships of steps
easy to identify, it does not indicate who or what may affect the completion of a
step or the resources needed to begin the next step. In its concise form, then, the
visual representation of the Intelligence Cycle is reduced to a map of information
handling. Without explicit descriptions of the steps in the process or the benefit of
prior knowledge, it can raise questions of accuracy and completeness and can
occasion misconceptions, particularly concerning the roles and responsibilities of
intelligence analysts.

5 Mare J. Rosenberg, “Performance technology: Working the system.”
6 Roger Kaufman, “A Holistic Planning Model: A Systems Approach for Improving Organiza-
tional Effectiveness and Impact.”

47



TESTING THE INTELLIGENCE CYCLE

ucts, and the turnaround time imposed. All of these factors determine the level
of demand placed on the analyst.

Production. This section focuses on the process of producing intelligence
products. The elements described are tied, directly or indirectly, to the flow
that represents changes in the analyst's ability to produce. In turn, these
changes cause products to be completed and requests of policymakers to be
fulfilled. It is important to note that this portion of the model deals with factors
that influence the act of analysis and does not attempt to address methods of
analysis.

Factors that influence the ability of analysts to produce are numerous and
complex, as shown. First and foremost are the capabilities an analyst brings to
the task. This is represented by a stock—usually an increasing one—that
derives from an analyst’s education, training, and experience.

Another influence is the number and frequency of evaluations and revisions
imposed on a work in progress. That a draft of the product must be reviewed
and edited by a number of others places variable constraints on the time avail-
able for creating the original draft. This factor increases in significance when
the product requested has a short deadline.

Political and cultural values of the organization also have an influence, usu-
ally constraining. Strictly following traditional heuristics and methods and
meeting organizational or management expectations may influence both an
analyst’s ability to produce and the quality of the output. The weight of these
influences will vary depending on the experience of the analyst.

Another factor that influences the analyst’s ability to produce is the amount
of relevant, usable data (a stock) available. The term “relevant, usable data”
describes all collected intelligence that is relevant to meeting the request and
that exists in a format that can be used to develop the product. To become
usable, the data must go through steps that are influenced by a variety of other
people, organizations, systems, and technologies. This process is represented
by the stock and flow chain that appears across the middle of diagram.

Data are collected from a variety of sources, represented by the INTs con-
verter.!° These data add to the stock of collected data. The ways in which
accumulated collected data are converted to the stock of available data are
influenced by internal research demands and specific collection requirements
imposed by analysts, policymakers, and others. Once the data are processed
and put into an agreed format for use by intelligence producers and consum-

‘© INT is an abbreviation for intelligence, usually contained in acronyms for the various types of
intelligence collected by the Intelligence Community, for example, HUMINT (human intelli-
gence) and SIGINT (signals intelligence).

53



INTRODUCTION

gence analysts would use analytic methods and tools from their own domain
in order to analyze and solve intelligence problems, When interdisciplinary
problems have arisen, the organizational assumption has been that a variety of
analytic methods would be employed, resulting in a “best fit” synthesis.

This individualistic approach to analysis has resulted in a great variety of
analytic methods—I identified at least 160 in my research for this paper—but
it has not led to the development of a standardized analytic doctrine. That is,
there is no body of research across the Intelligence Community asserting that
method X is the most effective method for solving case one and that method Y
is the most effective method for solving case two.”

‘The utility of a standardized analytic doctrine is that it enables an organiza-
tion to determine performance requirements, a standard level of institutional
expertise, and individual performance metrics for the evaluation and develop-
ment of new analytic methodologies.* Ultimately, without such an analytic
baseline, one cannot assess the effectiveness of any new or proposed analytic
method, tool, technology, reorganization, or intervention. Without standard-
ized analytic doctrine, analysts are left to the rather slow and tedious process
of trial and error throughout their careers.

Generally, in research literature, one finds a taxonomy, or matrix, of the
variables that affect the object under study. Taxonomies help to standardize
definitions and inform future research by establishing a research “road map.”
They point out areas of interest and research priorities and help researchers
place their own research programs in context. In my search of the intelligence
literature, I found no taxonomy of the variables that affect intelligence analy-
sis.

Following the literature review, I undertook to develop working definitions
and a taxonomy in order to systematize the research process. Readers will find
the working definitions in the first chapter. The second chapter highlights the
the broader findings and implications of this ethnographic study. Because the
first two chapters contain many quotes from my interviews and workshops,
they illustrate the tone and nature of the post-9/I1 environment in which I
worked.

The taxonomy that grew out of this work was first described in an article for
the CSI journal, Studies in Intelligence, and is presented here as Chapter

? There is no single Intelligence Community basic analytic training program. There is, however,
community use of advanced analytic courses at both the CIA University and the Joint Military
Intelligence College. The Generic Intelligence Training Initiative is a recent attempt to standard-
ize certain law enforcement intelligence analysis training programs through a basic law enforce-
‘ment analyst training curriculum. The program has been developed by the Training Advisory
Council, under the Counterdrug Intelligence Coordinating Group and the Justice Training Center.
* See the appendix.

xviii



BIBLIOGRAPHY

Chu, Godwin, and Wilbur Schramm. Learning from Television: What the
Research Says. Stanford, CA: Stanford University, Institute for Communica-
tion Research, 1968.

Clauser, Jerome K., and Sandra M. Weir. Intelligence Research Methodology:
An Introduction to Techniques and Procedures for Conducting Research in
Defense Intelligence. Washington, DC: US Defense Intelligence School,
1976.

Cline, Marjorie, Carla Christiansen, and Judith Fontaine. Scholar’s Guide to
Intelligence Literature: Bibliography of the Russell J. Bowen Collection.
Frederick, MD: University Publications of America, 1983.

Coch, Lester, and John French. “Overcoming Resistance to Change,” Dorwin
Cartwright and Alvin Zander, eds. Group Dynamics: Research and Theory.
New York: Harper & Row, 1960.

Cohen, Peter, James Kulik, and Chen-Lin Kulik. “Educational Outcomes of
Tutoring: A Meta-Analysis of Findings,” American Educational Research
Journal 19 (1982): 237-48.

Cushman, Philip. Constructing the Self, Constructing America, Reading, MA:
Addison-Wesley, 1995.

Davis, Jack. “Combating Mindset,” Studies in Intelligence 35 (Winter 1991):
13-18.

Dawes, Robyn. Written communications between Dawes and Johnston, 2002.

. “A Case Study of Graduate Admissions: Application of Three Princi-

ples of Human Decision Making,” American Psychologist 26 (1971): 180-

88.

, David Faust, and Paul Meehl. “Clinical Versus Actuarial Judgment,”
Science 243 (1989): 1668-74.

Dearth, Douglas, and R. Thomas Goodden, eds. Strategic Intelligence: Theory
and Application, 2nd ed. Washington, DC: Joint Military Intelligence Col-
lege, 1995.

Denzin, Norman, and Yvonna Lincoln, eds. Handbook of Qualitative
Research. Thousand Oaks, CA: Sage, 1994.

Deutsch, Morton. “The Effects of Cooperation and Competition Upon Group
Process” in Dorwin Cartwright and Alvin Zander, eds. Group Dynamics:
Research and Theory. New York: Harper & Row, 1960.

Duffy, T. M., and D. H. Jonassen, eds. Constructivism and the Technology of
Instruction: A Conversation. Hillsdale, NJ: Erlbaum, 1992.

Eco, Umberto. A Theory of Semiotics. Bloomington, IN: Indiana University
Press, 1979.

Egan, D., and B. Schwartz. “Chunking in Recall of Symbolic Drawings,”
Memory and Cognition 7 (1979): 149-58.

Elkins, Dan. An Intelligence Resource Manager's Guide. Washington, DC:
Joint Military Intelligence College, 1997.

142



COMBATING ETHNOCENTRISM

bered; it affects the ego and drives one to investigate errors and to adapt or
change behavior based on those investigations. Failure is a learning event and
results in a teachable moment.*

There seems to be little reason to perform a postmortem when events unfold
as predicted. The natural assumption is that the mechanisms of analysis were
valid, because the results of the analysis were accurate. The obvious danger is
that this assumption discounts the possibility that one may be accurate purely
by accident. Moreover, by focusing only on failure, one risks sampling bias by
only choosing cases in which there was error. The risk of ignoring success is
that potential lessons may go undiscovered. An alternative to relying on fail-
ure to challenge one’s assumptions is to create a standard practice of review-
ing each case regardless of outcome, principally through the use of a formal
After Action Review (AAR).

Case Study Two: The Red Team

Recently, I was asked to serve on a newly formed red team within the
Department of Defense. I agreed to participate, despite a number of serious
concerns having to do both with the nature and structure of red teams in gen-
eral and with my own experience with ethnocentrism and its effects on analy-
sis. These concerns are applicable not only to red teams, but also to any
analyst put in the position of trying to “think like them.”®

This particular red team was part of a constructive/conceptual war game in
which there were 11 participants, seven of whom had doctorates. Of the seven
doctorates, three were psychologists, one was a historian, one was an econo-
mist, one was a political scientist, and one was an anthropologist. The other
four participants had extensive military backgrounds. There were no physical
scientists or engineers. Nine of the 11 participants were white males, one was
a male born in the region of interest, and one was a white female. All were
middle class. Seven of the 11 were raised in nominally Christian homes and
three in nominally Jewish homes. (I say nominally because it was not possible
to determine their level of religious commitment during this exercise.)

I mention the demographics of the group because it was not representative
of the adversary we were intended to simulate. Although the group had

® Charles Perrow, Normal Accidents.

° Military red teams are meant to simulate the actions of an adversary in some type of war game or
crisis simulation, usually with the goal of generating scenarios for training and readiness or for
logistics and planning. These war games may be live, e.g., force-on-force simulations like those of
the US Army Combat Training Centers; virtual, as in flight simulators; or constructive, either dig-
ital theater-level simulations or purely conceptual games centered on strategic, tactical, or opera-
tional issues.

81



CHAPTER ONE
Definitions

a

Because I conducted human performance-related fieldwork before I came
to this project, I carried into it a certain amount of experiential bias, or “cogni-
tive baggage.” The research findings from those other studies could bias my
perspective and research approach within the Intelligence Community. For
example, surgeons and astronauts do not need to deal with intentionally
deceptive data. Patients are not trying to “hide” their illnesses from surgeons,
and spacecraft are not thinking adversaries intent on denying astronauts criti-
cal pieces of information. This one difference may mean that intelligence
analysis is much more cognitively challenging than the other two cases and
that the requisite psychomotor skills are significantly less important. In an
effort to counteract the biases of experience, I will attempt to be explicit about
my own definitions in this work.

Working Definitions

The three main definitions used in this work do not necessarily represent
definitions derived from the whole of the intelligence literature. Although
some of the definitions used in this work are based on the Q-sort survey of the
intelligence literature described later, some are based on the 489 interviews,
focus groups, and two years of direct and participant observations collected
during this project.



Taxonomy of Intelligence Analysis Variables

/Organization
Internal
Structure
Leadership
Culture
History
Traditions
Social Practice
Taboo
Group Characteris-
tics
Hierarchy
Resources & Incentives
Manpower
Budget
Technology
Assets
RaD
Facilities
Work Groups-Teams
Extenal
Consumer Needs
Time and Imperatives
Consumer Use
Consumer Structure
Consumer Hierarchy
Conumer Reporting
Politics
Internal-Organization

Security/Access
External-National
Law
Policy
External-international
Security
Denial
Deception
Policy

User Requirements
/Operations
Information Acquisition
Collection Methods
Overt
Covert
Information Reliability
Reproducible
Consistent
Information Validity
Historical
Single Source
Dual Source
Triangulation
Information Archive
Storage
Access
Correlation
Retrieval
Analytical Methodology
‘Approach
Intuitive
Structured
‘Semi-structured
Information Processing
Historical Information
‘Current Information
Decision Strategies
Estimative
Prodictive
Reporting
Verbal Methods
Written Methods

n
Weltanschauung
(worldview)
Ailiation
Familial
Cultural
Ethnic
Religious
Social
Linguistic
Political
Psychology
Bias
Personality Profile
Security Trust
Cognitive Processing
Learning Style
Information Acquisi-
tion
Information Process-
ing
Expertise
Problem-solving
Decisionmaking
Cognitive Load
Speed/Accuracy
Stress Effects
Education
Domain
Location
Mentor
Training
Organizational
Domain
Procedural
Readiness
Resources
Facilities

Formal
Inter-organization
Hierarchical
Inter-division
Inter-group
Intra-organization
Hierarchical
Intra-division
Intra-group
Individual
Hierarchical
Inter-division
Intra-group
Informal
Inter-organization
Hierarchical
Inter-division
Inter-group
Intra-organization
Hierarchical
Intra-division
Intra-group
Individual
Hierarchical
Inter-group
Intra-group
Technology
Networked Analysis
Collaboration

32



COMBATING ETHNOCENTRISM

Student leaders began a hunger strike to force a dialogue between the students
and the government. All signs seemed clearly to point to a popular movement
for democracy, for which there was a groundswell of support.

The Chinese government seemed hesitant or unsure. The People’s Libera-
tion Army (PLA) was sent to surround the square, but citizens blocked their
advance and tried to persuade the troops to be neutral. A curfew order was not
obeyed; martial law was declared and ignored. Another PLA move on Tianan-
men Square was repelled. It appeared that the students had forced a stalemate
and that their demands would be heard.

At that point, my assumption was that the government was weakened and
would be forced to respond to the protesters’ demands, at least to some degree.
1 anticipated a dialogue and concessions on both sides. Although I imagined
the government was capable of resorting to violence, I assumed that it would
not. It seemed inconceivable that the citizens of Beijing—10-12 million peo-
ple—would not intervene on behalf of the students. That many people could
have overwhelmed the PLA had they chosen to do so. I also assumed that the
soldiers of the PLA would be reluctant to fire on their own people, partly
because the majority of both groups were from the same, dominant ethnic
group of China, the Han, and, in part, because the soldiers represented a lower
rung of Chinese society then did the students. The notion of soldiers killing
students would be an affront to the sensibilities of the Han, or so | thought. I
was wrong.

In the end, when the PLA carried out its orders to clear the square with
force and end the protest, support for the protesters turned out to be relatively
slight. The Chinese “middle class” never came to the students’ aid; the great
majority of the Beijing populace simply watched the events unfold. Moreover,
it turned out that the labor groups participating in the demonstration were
actually protesting against corporate corruption and the lack of job stability
brought about by market reforms and not in support of the students’ demands
for a loosening of restrictions on expression. What I perceived to be a ground-
swell of popular support for the students had been exaggerated and wishful
thinking on my part.

My failure to anticipate the way events would actual unfold in Tiananmen
Square was tied to ethnocentric thinking and a lack of accurate and contextual
information. Students in the United States are encouraged to be politically
active, and their protests are often seen merely as minor inconveniences that
need to be endured. In China, however, the protesting students were seen as a
direct challenge to political authority and, much more so than in the United
States, their actions were viewed as an outright conflict between the future
elite and the current leadership. The protest itself was viewed as a violation of
a taboo, upsetting the cultural order and the stability of society.

1



FINDINGS

Finding: Focus on Current Production

The present daily production cycle and the focus on current intelligence
also affect group interactions and the analytic process.

Group Interactions:

It doesn’t matter if I'm writing a piece myself or if I’m coordinating
a piece with some group. We don't sit around and test hypotheses,
because we’re too busy writing. We’ve got serious deadlines here.

If, by group analysis, you mean the senior expert in the room tells
everybody what he thinks, and then we generally agree so that we
can get back to our own deadlines, then, sure, there’s a group pro-
cess.

We used to have groups that did current reporting and different
groups that did longer term products. We still have some of that, but
it is very limited. I couldn't say what happened exactly, but we're all
doing current production now.

The Analytic Process:
People seem to have confused writing with analyzing. They figure
that if you just go through the mechanics of writing something, then
you must have analyzed it. I don’t know about everybody else, but it
doesn't work that way for me. I need time to think through the prob-
lem.

Our products have become so specific, so tactical even, that our
thinking has become tactical. We're losing our strategic edge,
because we’re so focused on today’s issues.

Alternative analysis is a nice concept, but I don’t have the time to do
it. I’ve got to keep up with the daily traffic.

TL use several analytic techniques that are relatively fast. Scenario
development, red teams, competing hypotheses, they’re ail too time
consuming.

We've got Bayesian tools, simulations, all kinds of advanced meth-
ods, but when am I supposed to do any of that? It takes all my time
to keep up with the daily reporting as it is.

I don't have time to worry about formal analytic methods. I’ve got
my own system. It’s more intuitive and a lot faster.



BIBLIOGRAPHY

Lucy, John. Grammatical Categories and Cognition. Cambridge: Cambridge
University Press, 1992.

. Language Diversity and Thought. Cambridge: Cambridge University
Press, 1992.

MacEachin, Douglas J., et al. The Tradecraft of Analysis: Challenge and
Change in the CIA. Washington, DC: Consortium for the Study of Intelli-
gence, 1994.

Marrin, Stephen. CIA’s Kent School. A Step in the Right Direction. Paper pre-
sented at the International Studies Association Annual Convention, New
Orleans, LA, 2002. (http://www.isanet.org/noarchive/marrin.html.)

. “Improving CIA Analysis by Overcoming Institutional Obstacles” in
Russell G. Swenson, ed. Bringing Intelligence About: Practitioners Reflect
on Best Practices. Washington, DC: Joint Military Intelligence College,
2003.

Mathams, Robert. “The Intelligence Analyst's Notebook” in Douglas Dearth
and R. Thomas Goodden, eds. Strategic Intelligence: Theory and Applica-
tion, 2nd ed. Washington, DC: Joint Military Intelligence College, 1995.

Mauss, Marcel. The Gift: The Form and Reason for Exchange in Archaic
Societies. Translated by W. D. Halls. New York: W. W. Norton, 2000.

Mayo, Elton. The Human Problems of an Industrial Civilization. New York:
MacMillan, 1933.

McDowell, Don. Strategic Intelligence: A Handbook for Practitioners, Man-
agers and Users. Canberra, Australia: Istana Enterprises, 1998.

McEvoy, Glenn M., and Wayne F. Cascio. “Strategies for Reducing Employee
Turnover: A Meta-Analysis,” Journal of Applied Psychology 70, no. 2
(1985): 342-53.

McGuire, Hunter, et al. “Measuring and Managing Quality of Surgery: Statis-
tical vs. Incidental Approaches,” Archives of Surgery 127 (1992): 733-37.
McKeithen, Katherine, et al. “Knowledge Organization and Skill Differences

in Computer Programmers,” Cognitive Psychology 13 (1981): 307-25.

Mead, George. Mind, Self, and Society. Charles Morris, ed. Chicago: Univer-
sity of Chicago Press, 1967.

Meglino, Bruce M., et al. “Effects of Realistic Job Previews: A Comparison
Using an Enhancement and a Reduction Preview.” Journal of Applied Psy-
chology 71 (1988): 259-66.

Meister, David. Behavioral Analysis and Measurement Methods. New York:
Wiley, 1985.

. Human Factors: Theory and Practice. New York: Wiley, 1971.

Meister, Judith. “Individual Perceptions of Team Learning Experiences Using
Video-Based or Virtual Reality Environments,” Dissertation Abstracts Inter-
national, UMI No. 9965200, 2000.

Miller, Delbert. Handbook of Research Design and Social Measurement.
Newbury Park, CA: Sage, 1991.

150



RECOMMENDATIONS

Without evaluating preintervention, or precourse, performance and following
that with a postintervention evaluation, it is difficult to determine the effect that
any training intervention will have on employee performance. In addition to
formal measurements based on course objectives, it is important to collect per-
formance data from managers and supervisors to evaluate the retention of train-
ing and the impact that training has had on actual day-to-day performance.

Developing performance metrics will inform and advance the training inter-
ventions currently employed in the Intelligence Community and will deter-
mine the gap between ideal performance and actual performance. As such,
the system is an iterative process of setting performance standards, measuring
actual performance, designing training interventions to improve performance,
and evaluating the effects of those interventions on actual performance. The
data derived from these interventions and measurements will then contribute
to the growth of the knowledge repository and strengthen the ties created
through the communities of practice.

Organizational Culture and Effectiveness. Identifying existing organiza-
tional norms and taboos is the first step to creating an internal dialogue about
the future of an organization and its place in a competitive environment. Cul-
ture drives the operations of an organization, determines the people who are
hired, enculturates new employees, establishes standards of behavior and sys-
tems of rewards, shapes an organization’s products, and determines the social
capital that any organization may possess. In short, culture defines an organi-
zation’s identity to itself and to others.

Understanding the culture of the Intelligence Community and analyzing the
effects of any performance intervention on that culture contributes to the eval-
uation of intervention effectiveness. Effective performance interventions will
have a positive effect on the organization’s culture and become themselves
measurement instruments.

Developing cultural markers to track organizational change and perfor-
mance improvement requires baseline ethnographic data and the identification
of key cultural indicators. Once identified, cultural indicators such as lan-
guage use, norms, and taboos would be measured at regular intervals and
would serve as grounded data to determine levels of change within the organi-
zation. This would permit interventions to be modified before they became
ritualized within the Intelligence Community.

The Importance of Access

The improvement of human performance often requires an organization to
change its culture, and organizational leaders seldom possess sufficient power
to mandate cultural change by edict. At best, management can introduce

115



INTRODUCTION

Three. In addition to the normal journal review process, I circulated a draft of
the taxonomy among 55 academics and intelligence professionals and incor-
porated their suggestions in a revised version that went to press. This is not to
assert that the taxonomy is final; the utility of any taxonomy is that it can be
revised and expanded as new research findings become available. The chapter
by Dr. Judith Meister Johnston that follows offers an alternative model—more
complex and possibly more accurate than the traditional intelligence cycle—
for looking at the dynamics of the intelligence process, in effect the interrela-
tionships of many elements of the taxonomy

The following chapters, prepared by me and other able colleagues, were
developed around other variables in the taxonomy and offer suggestions for
improvement in those specific areas. One of them—Chapter Five, on integrat-
ing methodologists and substantive experts in research teams—also appeared
in Studies in Intelligence. Chapter Nine contains several broad recommenda-
tions, including suggestions for further research.

To the extent possible, I tried to avoid using professional jargon. Even so,
the reader will still find a number of specific technical terms, and, in those
cases, I have included their disciplinary definitions as footnotes.

A Work in Progress

In some respects, it may seem strange or unusual to have an anthropologist
perform this type of work rather than an industrial/organizational psychologist
or some other specialist in professional performance improvement or business
processes. The common perception of cultural anthropology is one of field-
work among indigenous peoples. Much has changed during the past 40 years,
however, Today, there are many practitioners and professional associations
devoted to the application of anthropology and its field methods to practical
problem-solving in modern or postindustrial society.?

It is difficult for any modern anthropological study to escape the legacy of
Margaret Mead. She looms as large over 20th century anthropology as does
Sherman Kent over the intelligence profession. Although Franz Boas is argu-
ably the father of American anthropology and was Margaret Mead’s mentor,
hers is the name everyone recognizes and connects to ethnography. !” Chances
are, if one has read anthropological texts, one has read Mead.

° The Society for Applied Anthropology and the National Association for the Practice of Anthro-
pology section of the American Anthropological Association are the wo principal anthropologi-
cal groups. Another group is the Inter-University Seminar on Armed Forces and Society, a
professional organization representing 700 social science fellows, including practicing anthropol-
ogists, applying their research methods to issues in the military.

xix



A TAXONOMY OF INTELLIGENCE

has on the analytic process. This is an area that could benefit from additional
study.

Conclusion

There is rarely any doubt that the unconscious reasons for practic-
ing a custom or sharing a belief are remote from the reasons given
to justify them.

—Claude Levi-Strauss*!

As it is now practiced, intelligence analysis is art, tradecraft, and science.
There are specific tools and techniques to help perform the tasks, but, in the
end, it is left to individuals to use their best judgment in making decisions.
This is not to say that science is not a part of intelligence analysis. Science is
born of organized knowledge, and organizing knowledge requires effort and
time. The work on this taxonomy is intended to help that process by sparking
discussion, identifying areas where research exists and ought to be incorpo-
rated into the organizational knowledge of intelligence, and identifying areas
where not enough research has been performed.

There are a number of parallels in the field of medicine, which, like intelli-
gence, is art, tradecraft, and science. To solve problems, practitioners are
trusted to use their best judgment by drawing on their expertise. What is
important to remember is that there are numerous basic sciences driving med-
ical practice. Biology, chemistry, physics, and all of the subspecialties blend
together to create the medical sciences, the foundation on which modern med-
icine rests. The practice of medicine has been revolutionized by the sciences
that underpin its workings.

Intelligence analysis has not experienced that revolution. Unlike medicine,
the basic sciences that underpin intelligence are the human sciences, which are
considerably more multivariate and more difficult to control. Because of these
factors, it is a more complex task to measure “progress” in the human sci-
ences. Even so, there are numerous domains from which intelligence may bor-
row. Organizational behavior is better understood today than ever before.
Problem-solving and decisionmaking have been researched since the 1920s.

# Wohlstetter.

41 Claude Levi-Strauss wrote Structural Anthropology in 1958, setting the stage for structuralism
to emerge as an analytic interpretive method. Broadly, structuralism seeks to explore the inter-
relationships (the “structures”) through which meaning is produced within a culture. This mean-
ing, according to structural theory, is produced and reproduced through various practices, phe-
nomena, and activities that serve as systems of “signification.” A structuralist studies activities as
diverse as food preparation and serving rituals, religious rites, games, literary and non-literary
texts, and forms of entertainment to discover the ways in which cultural significance develops.

43



CHAPTER NINE

analyses requires significant research expertise, ample time, and broad organi-
zational access.

In time, analytic performance research will become a highly specialized
domain and will require continuous organizational access not normally avail-
able to outsiders. It will become necessary for the Intelligence Community to
establish internal or cooperative research centers in order to acquire the
research expertise necessary to analyze and effect performance improve-
ment. There are numerous community outreach efforts on which these centers
can be built. Those efforts need to be expanded, however, and those programs
need to include domains beyond the traditional relationship between the Intel-
ligence Community and political or geographic area experts.*

Institutional Memory. The results of this research program indicate that
there is a loss of corporate knowledge in the Intelligence Community due to
employee attrition and the lack of a central knowledge repository for captur-
ing “lessons learned.” A number of industries and government organizations,
including the Departments of Defense and Energy and the National Aeronau-
tics and Space Administration, already maintain centers for lessons learned as
an information hub for its employees.*

These centers act as information repositories for successful and unsuccess-
ful operations and interventions. Their purpose is to reduce the amount of
organizational redundancy and levels of error and failure by tracking, analyz-
ing, and reporting on after-action reviews and analytic outcome data The
other primary function of these repositories is to establish networks for com-
munities of practice within and among organizations.

Networked communities of practice allow professionals to interact, exchange
methodological information, post and respond to individual case studies, and
develop ad hoc teams of experts for specific problem solving tasks. With simple
search tools, basic database software, and a simple network visualization inter-
face, any analyst in the Intelligence Community would be able to identify any
other expert whose domain specialty was needed to answer a specific question
or solve a specific problem. Another advantage of this model is the develop-
ment of formal and informal mentoring within the network. Any novice would
be able to find an expert within the Intelligence Community and establish a rela-
tionship that would be beneficial to both. With appropriate incentives, experts
would be encouraged to contribute to the network and make availabie their time
and expertise for the purpose of mentoring.

+ An example of researching the validity and reliability of metrics can be found in the Bros Men-
tal Measurement Yearbook at the Buros Institute of Mental Measurements Web site.

4See the US Army Center for Army Lessons Leamed (CALL) Web site, which has links to
numerous other repositories.

5 See Chapter Six for a more detailed explanation of the After Action Review process.

112



CHAPTER SEVEN

advance beyond the constraints of the prescribed, prebranched, programmed
learning and ad hoc principles commonly used to design technology-based
instruction. The long-term vision is that training, education, and performance
improvement will take the form of human-computer conversations.

There has been progress toward this end. This conversational capability has
been realized in systems that can discuss issues with students using a formal
language, such as computer programming or propositional calculus.** More
recent research suggests that significantly improved natural-language dia-
logue capabilities can be achieved by instructional technology.*? Such an
interactive, generative capability is needed if we are to deal successfully with
the extent, variety, and mutability of human cognition. Much can now be
accomplished by instructional technology, but much more can be expected.

Conclusion

The research discussed above suggests that instructional technology can:
* reduce costs of instruction;

* increase the accessibility of instruction;

* increase instructional effectiveness for analysts;

* reduce the time analysts need to learn facts, concepts, and procedures;
* track progress and ensure that all learners achieve instructional targets;

* provide opportunities for helping analysts to compress experience and
achieve the higher cognitive levels of mastery demanded by their craft.

In addition, the findings suggest a rule of “thirds.” This rule posits that the
present state-of-the-art in instructional technologies can reduce the cost of
instruction by about a third and either increase achievement by about a third or
decrease time to reach instructional objectives by a third. Eventually, instruc-
tional technology should provide a conversation between the analyst and the
technology that will tailor instruction in real time and on demand to the particular
knowledge, skills, abilities, interests, goals, and needs of each individual. This
capability, now available in rudimentary forms, can be expected to improve and
develop with time. Even in its current state of development, however, instruc-
tional technology deserves serious attention within the Intelligence Community.

% For example, BIP and EXCHECK, respectively. For the first, see A. Barr, M. Beard, and R. C.
Atkinson, “A rationale and description of a CAI Program to teach the BASIC Programming Lan-
guage”; for the second, see P. Suppes and M. Morningstar.

8 A. C. Graesser, M. A. Gernsbacher, and S. Goliman, Handbook of Discourse Processes.

96



CHAPTER EIGHT

is the process of learning the ropes; training; and becoming formally and
informally acquainted with what is actually of value within the organization.*
It is also the time when one learns the organization’s norms and taboos and the
extent of its social capital.* In sum, formal and informal socialization are types
of control mechanism for maintaining the norms, or status quo, within any
organization.‘

Organizational Socialization

According to Daniel Feldman, organizational socialization is “the process
through which individuals are transformed from outsiders to participating,
effective members of an organization.” As shown in Figure 1, Feldman
divides this process into three stages: getting in (or anticipatory socialization),
breaking in (or accommodation), and settling in (often referred to as role man-
agement). During the getting-in stage, potential employees try to acquire
information about an organization from available sources, such as Web sites,
professional journals, and corporate annual reports. The breaking-in stage
includes orientation and learning organizational as well as job-related proce-

Feldman’s three stages of organizational socialization.

dures. The settling-in stage concludes when an individual attains full member
status in the organization.

While each of the three stages of socialization is important, the focus of this
chapter is on the first, or anticipatory, stage. There are several reasons for this
Clearly, the expectations people develop about an organization they are join-
ing are important to a new recruit’s eventual satisfaction, retention, and per-
formance. Moreover, because it can control several aspects of the recruitment
process, this stage is often the easiest for an organization to change. This
chapter will take both a descriptive and prescriptive approach to easing the
ialization of new employees.

’ William G. Tierney and Robert A. Rhoads, Faculty Socialization as Cultural Process.
4 See footnote 7 in Chapter Two.
5 John P. Wanous, Organizational Ent

© Daniel C. Feldman, “The Multiple s

socialization of Organization Members.”

98



Dori Akerman, and, most particularly, my editors: Mike Schneider, Andy
Vaart, and Barbara Pace. Special thanks are also due Adm. Dennis Blair, USN
(Ret.), and Gen. Larry Welch, USAF (Ret.), and their staff.

There were 50 peer reviewers who made sure I did not go too far afield in
my research and analysis. Again, there are mitigating reasons why I cannot
thank them by name. Suffice it to say, their work and their time were invalu-
able, and I appreciate their efforts.

Because I cannot name specific individuals, I would like to thank the organiza-
tions of the Intelligence Community that gave me access to perform this research
and made available research participants: Air Force Intelligence; Army Intelli-
gence; Central Intelligence Agency; Defense Intelligence Agency; Department of
Energy; Department of Homeland Security; Bureau of Intelligence and Research,
Department of State; Department of the Treasury; Federal Bureau of Investiga-
tion; Marine Corps Intelligence; National Geospatial Intelligence Agency;
National Reconnaissance Office; National Security Agency; Navy Intelligence.

Thanks are also due the following: Institute for Defense Analyses; the Sher-
man Kent Center and the Global Futures Partnership at the CIA University; the
CIA’s Publications Review Board; Office of Public Affairs, National Archives;
Joint Military Intelligence College; Advanced Research and Development Activ-
ity, Defense Advanced Research Projects Agency (DARPA); International Asso-
ciation of Law Enforcement Intelligence Analysts; Drug Enforcement
Administration and the DEA Academy; FBI Academy; National Mititary Intelli-
gence Association; Association of Former Intelligence Officers; MITRE; RAND;
Analytic Services, Inc. (ANSER); Potomac Institute; Center for Strategic and
International Studies, Woodrow Wilson International Center; Booz Allen Hamil-
ton; Naval Postgraduate School; Columbia University; Dartmouth College; Uni-
versity of Pittsburgh; Georgetown University; Carnegie Mellon University;
Cambridge University; Johns Hopkins University and the Advanced Physics
Laboratory; George Mason University; Harvard University, Yale University;
American Anthropological Association; Society for the Anthropology of Work;
Society for Applied Anthropology; National Association for the Practice of
Anthropology; Inter-University Seminar on Armed Forces and Society; Royal
Anthropological Institute, and the national laboratories.

I express my sincere apologies if I have failed to include any individuals or
organizations to which thanks are due. Moreover, any errors of commission or
omission are my own. God knows, with this much help, there is no one to
blame but myself. Mostly, though, I would like to thank my long-suffering
wife, to whom this book is dedicated. Thanks, Jude.



CHAPTER SIX

The Question of Foreign Cultures: Combating
Ethnocentrism in Intelligence Analysis

SE

The intelligence literature often cautions intelligence professionals to be
wary of mirror imaging.! Although the term is a misnomer (a mirror image is
a reverse image), the concept is that individuals perceive foreigners—both
friends and adversaries of the United States—as thinking the same way as
Americans.” Individuals do, in fact, have a natural tendency to assume that
others think and perceive the world in the same way they do. This type of pro-
jective identification, or ethnocentrism, is the consequence of a combination
of cognitive and cultural biases resulting from a lifetime of enculturation, cul-
turally bound heuristics, and missing, or inadequate, information.?

Ethnocentrism is a phenomenon that operates on a conscious level, but it is
difficult to recognize in oneself and equally difficult to counteract. In part, this
is because, in cases of ethnocentric thinking, an individual does not recognize
that important information is missing or, more important, that his worldview
and problem-solving heuristics interfere with the process of recognizing infor-
mation that conflicts or refutes his assumptions.

Take, for example, the proposition that others do not think like Americans.
It seems only intuitive that other tribes, ethnic groups, nationalities, and states

} Alexander Butterfield, The Accuracy of Intelligence Assessment; Richards J. Heuer, Jt., Psychol-
ogy of Intelligence Analysis; Lisa Krizan, Intelligence Essentials for Everyone; J. R. Thompson,
R. Hopf-Weichel, and R. Geiselman, The Cognitive Bases of Intelligence Analysis.

In this work, I use the broader term “ethnocentrism” to refer to the concept represented by mir-
ror imaging and projective identification .

+ In anthropology, ethnocentrism is the tendency to judge the customs of other societies by the
standards of one’s own culture. This includes projecting one’s own cognition and norms onto oth-
ers.

rs)



CHAPTER TWO

that analysis is not a science. That is, it is an art or craft in which one can
attain skill but not a formal discipline with tested and validated methodology.

What we do is more art and experience than anything else.
Science is too formal. We can't actually run experiments here.
How would you actually test a hypothesis in intelligence?
Science is what you do in a lab.

We're not scientists; we're analysts. We don't generate the data.
We don't worry too much about theory; we worry about the facts.

In my discipline, I might be a scientist, but, in intelligence, I am a
practitioner.

T use science for my area, but I don’t think intelligence analysis is
science.

As long as intelligence analysis continues to be tradecraft, it will remain a
mystery. The quality of any tradecraft depends on the innate cognitive capa-
bilities of the individual and the good fortune one has in finding a mentor who
has discovered, through many years of trial and error, unique methods that
seem to be effective. This process of trial and error is, in general, similar to
any scientific process, except that the lessons learned in tradecraft, unlike
those of other disciplines, often occur without being captured, tested, or vali-
dated.

In an oral tradition, individual tradecraft methods are passed on by means
of apprenticeship. The consequence for any culture tied to an oral tradition is
the loss of important knowledge that occurs with the loss of practitioners. In
organizations, the retirement of experts and innovators leads to the loss of that
expertise and innovation, unless there is some formal written and educational
system to keep that knowledge alive. !2

The data collected through both interviews and observation indicated that
there were, in fact, general methods that could be formalized and that this pro-
cess would then lead to the development of intelligence analysis as a scientific
discipline. The principal difficulty lies not in developing the methods them-

"Rather than engage in the longstanding and ongoing debate in the academic community about
what is and what is not science or a scientific method, suffice it to say that any scientific method
needs to be explicit, replicable, and refutable. The literature surrounding this debate is volumi-
nous. The philosophy of science, logic, language, and epistemology has taken this debate in a
number of directions. There is, however, a general theme that replication is a key ingredient to
any scientific method,

"2 See section on Endangered Languages in Barbara Grimes, ed.. Ethnologue. 14th ed.



BIBLIOGRAPHY

Berkowitz, Bruce D., and Allan E. Goodman. Best Truth: Intelligence in the
Information Age. New Haven, CT: Yale University Press, 2000.

. Strategic Intelligence for American National Security. Princeton, NJ:
Princeton University Press, 1989.

Bernard, Russell. Research Methods in Anthropology: Qualitative and Quan-
titative Approaches. Thousand Oaks, CA: Sage, 1994,

Betts, Richard K. “Policy-Makers and Intelligence Analysts: Love, Hate or
Indifference,” Intelligence and National Security 3, no, 1 January 1988):
184-89.

. Surprise Attack. Washington, DC: The Brookings Institution, 1982.

Bickman, Leonard, and Debra Rog. Handbook of Applied Social Research
Methods. Thousand Oaks, CA: Sage, 1997.

Bloom, Alfred. The Linguistic Shaping of Thought: A Study in the Impact of
Language on Thinking in China and the West. Hillsdale, NJ: Lawrence
Erlbaum Associates, 1981.

Bloom, Benjamin S., ed. Taxonomy of Educational Objectives: The Classifi-
cation of Educational Goals: Handbook I, Cognitive Domain. New York:
Longmans, Green, 1956.

. “The 2 Sigma Problem: The Search for Methods of Group Instruction
as Effective as One-to-One Tutoring,” Educational Researcher, 13 (1984):
416,

Boas, Franz. “An Anthropologist’s Credo,” The Nation 147 (1938): 201-4.

Bogdan, Robert. Participant Observation in Organizational Settings. Syra-
cuse, NY: Syracuse University Press, 1972.

Boroditsky, Lera. “Does Language Shape Thought? Mandarin and English
Speakers’ Conceptions of Time,” Cognitive Psychology 43 (2001): 1-22.

Bourdieu, Pierre. “The Forms of Capital” in John Richardson, ed. Handbook
of Theory and Research for the Sociology of Education. New York: Green-
wood Press, 1986.

Brei, William. Getting Intelligence Right: The Power of Logical Procedure.
Washington, DC: Joint Military Intelligence College, 1996.

Brinberg, David, and Joseph McGrath. Validity and the Research Process.
Beverly Hills, CA: Sage, 1985.

Brown, J.S., R.R. Burton, and J. DeKleer. “Pedagogical, Natural Language,
and Knowledge Engineering in SOPHIE I, II, and III” in D. Sleeman and J.S.
Brown, eds. Intelligent Tutoring Systems, New York: Academic Press, 1982.

Busse, Daniela, and Chris Johnson. “Using a Cognitive Theoretical Frame-
work to Support Accident Analysis,” Proceedings of the Working Group on
Human Error, Safety, and System Development. Seattle, WA: International
Federation for Information Processing, 1998, 36-43.

Butterfield, Alexander. The Accuracy of Intelligence Assessment: Bias, Per-
ception, and Judgment in Analysis and Decision. Newport, RI: Naval War
College, 1993, AD-A266.

140



CHAPTER NINE

Recommendations

a

The First Step: Recognizing A Fundamental Problem

It is far too early in the research process to determine if any one organiza-
tional model for the Intelligence Community is more or less effective than any
other, but I believe there is a fundamental structural question that needs to be
addressed at the outset. This is, in my view, that current reporting competes
for time and resources with indications and warning (I&W) intelligence. This
emphasis is unlikely to change, for several reasons. First, current intelligence
reporting results in significant “face-time” for the Intelligence Community
with policy makers, who, in turn, provide the resources that fund and support
community activities. This is a significant contributor to the social capital that
the Intelligence Community commands.

The second reason is that in-depth research of the kind that contributes to
I&W intelligence is a long-term investment whose payoff is often an abstrac-
tion. Not infrequently, successful warnings are taken for granted. Those that
fail, however, may well involve the community in public recriminations that
cost the Intelligence Community significant social capital. In this sense, the
Intelligence Community’s focus on current reporting is understandable. The
problem is that producing current intelligence tends to become an all-consum-
ing activity. The majority of analysts who participated in this study said that
their time was spent on current reporting. Unfortunately, this does little to
improve I&W intelligence, which requires long-term research, in-depth exper-
tise, adoption of scientific methods, and continuous performance improve-
ment. The return for the Intelligence Community, in terms of social capital,
may be quite limited and even, as noted above, negative. Thus, the analytic
area most in need of long-term investment often gets the least.

107



CHAPTER SIX

numerous domain matter experts, very few had first-hand knowledge of the
region of interest. Only one participant was from the area, had spent formative
years there, spoke the languages, and experienced the culture firsthand. As
this group was assembled to simulate the behavior and decisionmaking of a
foreign adversary, this aspect was more important than it would have been for
a substantive team developing threat assessments around a specific topic or
target. Consequently, the scenarios developed by the red team often reflected
an adversary whose behavior and decisionmaking resembled those of edu-
cated, white, middle class Americans.

The one member of the red team who had been born in and spent formative
years in the region of interest regularly stopped the scenario development pro-
cess by saying, “They wouldn’t do that” or “They don’t think that way.” On
several occasions, he objected, “This scenario is way too complex” or “They
wouldn’t use that tactic; it requires too much direct communication.” His
objections were not usually based on military considerations; rather, they were
based on the cultural norms and mores of the adversary. He talked of kinship
relationships as a specific type of social network in the region and of the value
of kinship for understanding the adversary’s intentions. In short, he brought an
ethnographic perspective to the exercise.

Having no personal or professional experience with this region or its cul-
tures, I thought it appropriate to defer to his first-person experience. Ulti-
mately, however, it proved difficult to convince the group that this man’s
cultural knowledge was, in fact, an area of specialized knowledge that needed
to be factored into each scenario, This difficulty was born out of another type
of ethnocentric bias.

Inviting an anthropologist to a red team exercise presupposes that the red
team takes seriously the notion that cultural differences matter and that those
cultural factors ought to be made explicit in the analytic process. The problem
in this case was that the anthropologist was not an area expert for this region
and its cultures, and the one area expert who was there lacked the academic
credentials to be taken seriously by the other members of the group. Had I
been able to assert the same concepts that the other individual asserted, it
would have had a certain academic, or scientific, imprimatur because of my
training and experience. Because he lacked these credentials, many of the
other individual’s insights were lost, and the analytic product suffered as a
result.

The Red Team: Discussion

lam reluctant to fault the organizers for the ethnocentric bias in the demo-
graphic composition of the red team. It is very difficult to assemble a truly

82



FINDINGS

seems most probable and, consequently, is easiest to support. As this strategy
is also the most time-efficient technique, it reduces the time constraints associ-
ated with the daily production cycle.

This tendency to search for confirmatory data is not necessarily a conscious
choice; rather, it is the result of accepting an existing set of hypotheses, devel-
oping a mental model based on previous corporate products, and then trying to
augment that model with current data in order to support the existing hypothe-
ses. Although motivational and heuristic factors and a tendency toward
“groupthink” might contribute to confirmatory behavior in intelligence analy-
sis, my observations and interviews during this study suggest that the predom-
inant influence is selectivity bias in order to maintain a corporate judgment.

The maintenance of a corporate judgment is a pervasive and often-unstated
norm in the Intelligence Community, and the taboo against changing the cor-
porate product line contributes to confirmation biases. Once any intelligence
agency has given its official opinion to policymakers, there exists a taboo
about reversing or significantly changing the official or corporate position to
avoid the loss of status, trust, or respect. Often, policymakers perceive a
change in judgment as though the original opinion was wrong, and, although
unstated, there are significant internal and external social pressures and conse-
quences associated with being perceived as incorrect.

An analyst can change an opinion based on new information or by revisit-
ing old information with a new hypothesis; in so doing, however, he or she
perceives a loss of trust and respect among those with whom the original judg-
ment was shared. Along with this perceived loss of trust, the analyst senses a
Joss of social capital, or power, within his or her group.'°

It is even more difficult for an intelligence agency to change its official
position once it has made its judgments known to those outside of the organi-
zation. There is a sense that changing the official product line will be seen out-
side of its context—the acquisition of new information, for instance—and that
it will be perceived by the policymakers as an example of incompetence or, at
least, of poor performance on the part of the intelligence agency.

This perception then carries with it the threat of a loss in status, funding,
and access to policymakers, all of which would have a detrimental effect on
the ability of the intelligence agency to perform its functions. In short, it
serves the interest of the intelligence agency to be perceived as decisive

°S Reciprocity in this case has to do with information, judgment, and trust. The classic anthropo-
logical text on social reciprocity and trust within and between groups is Marcel Mauss’s The Gift.
Originally published in 1950 and based in part on the work of his uncle and mentor, Emile
Durkheim, Mauss's work (Essai sur le Don in its French version) lays the foundation for bis con-
tention that reciprocity is the key to understanding the modern concept of social capital.

23