Occasional Paper Number Seven

JOINT MILITARY

INT

E

LLIGENCE COLLEGE

1

9

6

2

DEFENSE INTELLIGENCE AGENCY

UNITED STATES OF AMERICA
0

The Joint Military Intelligence College supports and encou

research on intelligence issues that distills lessons and impr

support to policy-level and operational consumers

This series of Occasional Papers presents the work of faculty, students and others whose 

intelligence issues is supported or otherwise encouraged by the Joint Military Intellige

through its Office of Applied Research. Occasional papers are distributed to Department 

schools and to the Intelligence Community, and additional copies of unclassified papers a

through the National Technical Information Service <www.ntis.gov>; copies of selected 

also available through the U.S. Government Printing Office <www.gpo.gov>.

Proposed manuscripts for these papers are submitted for consideration to the Applied Re

torial Board. Papers undergo review by senior officials in Defense, Intelligence and occas

ian academic or business communities. 
i

Occasional Paper Number Seven

INTELLIGENCE ANALYSIS IN THEATER JO

INTELLIGENCE CENTERS: AN EXPERIMEN

APPLYING STRUCTURED METHODS

By MSgt Robert D. Folker, Jr. (USAF)

JOINT MILITARY INTELLIGENCE COLLEGE

WASHINGTON, DC

January 2000

The views expressed in this paper are those of the author

and do not reflect the official policy or position of the

Department of Defense or the U.S. Government

This paper has been approved for unrestricted public release by the 

Office of the Secretary of Defense (Public Affairs)
iii

CONTENTS

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

Preamble . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

The Research Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

The Art or Science Debate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

The Experiment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

Considerations and Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

The Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

Implications and Recommendations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

Final Impression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

About the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
v

JOINT INTELLIGENCE CENTERS: AN 

EXPERIMENT IN APPLYING 

STRUCTURED METHODS

Preface

In clear, articulate, unmistakable language, Master Sergeant Folker’s lea

sets forth the key opposing arguments in the long-standing controversy over 

structuring in intelligence analysis. The controversy is largely one-sided, beca

nents of intuitive analysis see no purpose in debating the issue, as they are com

isfied with the status quo. It is only the structuring enthusiasts who see a need 

change in the way analysis is conducted. Because, as Folker points out, suppo

status quo include not only most analysts but most commanders as well, the 

enthusiasts have never made progress in reforming the other side. And they w

any progress until the superiority of structured analysis over intuitive analysis 

which Folker has taken a first giant step in doing.

And it is precisely on this point that Folker challenges the Intelligence Com

indeed the entire U.S. government and the private sector as well—to design a

reliable tests to demonstrate which analytic approach is superior: structured o

Given the wide range of available structuring techniques, each one should b

competition with intuition.

There are many myths prevalent in the analytic community, public and p

cerning structured analysis: It is time-consuming and overly complicated; it eli

indispensable role of intuition; it involves numbers and arcane formulae that re

sis mechanical and devoid of creative solutions; and so on. Tests like Folker’s 

dispose of these myths and demonstrate the immense value and power of struct

His thesis, moreover, should encourage students to undertake research e

that challenge long-held beliefs and test new theories of military interest. JM

are especially well placed to conduct thoughtful, creative research because th

this scholarly environment a rich mixture of background and viewpoints, a

they remain vitally involved in the business of wringing meaning from a grea

intelligence information.

Morgan D. Jones

Former intelligence analyst, supervisor and trainer, Central Intelligence A
vii

JOINT INTELLIGENCE CENTERS: AN 

EXPERIMENT IN APPLYING 

STRUCTURED METHODS

Foreword

A growing number of JMIC students have begun to incorporate replicab

design into their theses. This distillation of Master Sergeant Folker’s thesis 

fruitful this approach can be. By taking advantage of on-site research funds ava

the College, he managed, in brief visits to four Unified Command Joint Intelli

ters, to carry out a controlled experiment to measure the impact of analyst fami

and use of one structured analytic technique — hypothesis testing. His findings

orated by follow-on studies, could have a substantial impact on Intelligence C

analytical practices, and even some influence on how senior policy officials re

lytical products. Managers of analytical personnel might also note that struct

ods would appear, from this study of non-specialized joint intelligence center 

be useful to specialized analysts who may be required to move beyond their ar

expertise to cover new or unfamiliar assignments.

Of equal importance with his findings is MSgt Folker’s documentation of a

research procedures. His subsequent recommendations for follow-on researc

the close relationship between doing research and contributing to the corpora

environment. The Editor welcomes other students to contribute the story of t

discoveries about the intelligence profession to this series of Occasional Paper

This paper benefited from reviews by a panel of JMIC faculty, the Director o

of the Defense Intelligence Agency’s Directorate of Intelligence, and by 

Jones, founder of Analytic Prowess, L.L.C. of Montclair, Virginia.

Russell G. Swenson, Editor and Director, Office of Applied Research, 

AFswerg@dia.osis.gov. 
1

JOINT INTELLIGENCE CENTERS: 

AN EXPERIMENT IN APPLYING 

STRUCTURED METHODS

Exploiting structured methodologies compels the intelligence a

lyze the intelligence problem at hand and distinguishes analytic

from office administration. 

PREAMBLE

A debate exists between those who prefer to think of intelligence ana

form and those who would highlight the value of a structured, scientific ap

assessment of intelligence problems. Analytical approaches to intelligence 

be divided into two overlapping categories—qualitative and quantitativ

analysis attempts to answer questions or solve problems that are not easily 

into quantifiable variables. Qualitative analysis is therefore most often use

military, and warning intelligence. Quantitative analysis, which uses vari

more easily measured, generally attempts to solve scientific or technica

problems. Of the two, qualitative intelligence analysis, if performed caref

provides the information most useful to national policymakers.1

A small number of analysts occasionally apply structured methods 

intelligence analysis, but the traditional approach to solve qualitativ

problems is non-structured.2 Surprisingly, no research known to thi

adequately examined whether exploiting structured methodologies 

qualitative intelligence analysis. 

Intelligence analysts who do use non-structured methods for qualitative 

that structured methods too narrowly define a problem and ignore factors 

measured. They assert that their intuitive approach produces superior res

part, analysts who employ structured methods for qualitative analysis intu

that structured methods ensure sounder, more comprehensive, and accu

However, no empirical evidence is available to support either belief.

1 Richard K. Betts, “Intelligence for Policymaking,” Washington Quarterly 3, 

1980): 119. 2 James D. Hammond, So You Want to Be an Intelligence Analyst? Research Pap

DC: Defense Intelligence College, October 1983), 10. 3 Capt. William S. Brei (USAF), Getting Intelligence Right: The Power of Log

Occasional Paper Number Two (Washington, DC: JMIC, 1996), 1-2.
2

specialized analysts at four Unified Command joint intelligence centers to tes

of applying a structured method to the qualitative analysis of an intelligenc

Only tentative conclusions can be drawn from the findings in this limited expe

the results, if corroborated, have some profound implications for Intelligence C

analysts and their managers. 

In brief, the author found that analysts who apply a structured method—

testing, in this case—to an intelligence problem, outperform those who rely on 

as-art,” or the intuitive approach. The modest character of this experiment, 

finding, clearly calls for follow-on research, but this study at least shows that 

value may be added to information by investing some pointed time and effort 

rather than expecting such value to arise as a by-product of “normal” office acti

THE RESEARCH PROBLEM

Because the Intelligence Community has used technology to increase it

collect information, intelligence analysts face an ever-increasing stream of i

data.4

 But the Community has not made similar improvements in its analytical 

and intelligence analysts are not exploiting all of the analytical methodologies 

them.5 Perhaps in part because intelligence analysts lack familiarity with 

analytical tools, they are being overwhelmed by the amount of informatio

expected to analyze, contributing to “analysis paralysis.”6 Congress is well aw

phenomenon, and has raised the issue publicly.7

A Research Question

A variety of structured methodologies is available to help the intelligence an

qualitative problems,8 but most analysts display an inability or unwillingnes

them.9

 Most people instinctively prefer intuitive, non-structured approa

structured methodologies. Moreover, identifying all relevant qualitative varia

4 Stephen J. Andriole, “Indications, Warning, & Bureaucracies,” Military Intelligence 

Bulletin 10, no. 3 (July-September 1984): 8. 5 David M. Keithly, “Leading Intelligence in the 21st Century: Past as Prologu

Intelligence Journal 7, no. 1 (Spring 1998): 80. 6 Alvin Toffler and Heidi Toffler, War and Anti-War: Survival at the Dawn of the 2

(New York: Little, Brown and Company, 1993), 158. 7 Vernon Loeb and Walter Pincus, “New Spy Satellites at Risk Because Funding I

Pentagon Told,” Washington Post, 12 November 1999, A7. 8 Robert M. Clark, Intelligence Analysis: Estimation and Prediction (Baltimore

Literary Press, Inc., 1996), 11. 9 Stephen J. Andriole, senior intelligence researcher and former director of the 

Technology Office of the Advanced Research Projects Agency, email interview 

15 December 1998.
3

prefer a subjective approach. Can intelligence analysts, specifically tho

analysis in the “softer” sciences such as political intelligence, improve the q

analysis by better exploiting structured methodologies? Comparing the ana

of structured methods with the analytical results derived from traditional n

practices used by intelligence analysts should provide valuable insights i

qualitative intelligence analysis. The need for such an experiment has 

identified by a number of specialists.11

Improvement in qualitative intelligence analysis has been a goal of long s

Intelligence Community, but most attempts to improve the process have ig

and focused on restructuring intelligence organization and providing additio

non-analytical subjects. The U.S. Commission on the Roles and Capabilities 

States Intelligence Community determined that too few resources are de

source analysis, and even fewer resources are obligated to developing an

analytical expertise.12 Currently, only six percent of the intelligence budget 

the U.S. Congress is spent on analysis.13 The House of Representative’s Per

Committee on Intelligence concluded that “[a]nalytical pitfalls have been 

post mortems and other studies of past warning situations, and limited orga

training efforts have been undertaken in response.”14

A Record of Analytic Failures

Although organizational restructuring may be of some benefit to th

Community, the root cause of many critical intelligence failures has b

failure. “Failure to predict the North Korean invasion in 1950 was a failure 

higher echelons. ... No analyses accompanied this raw data.”15 The 

10 Morgan D. Jones, The Thinker’s Toolkit: 14 Powerful Techniques for Problem 

York: Random House, Inc., 1998), 8. 11 Andriole, “Indications, Warning, & Bureaucracies,” 12; Stanley A. Feder, “F

Policon: New Ways to Analyze Politics,” in Inside CIA’s Private World: Decla

from the Agency’s Internal Journal 1955-1992, ed. H. Bradford Westerfield (N

University Press, 1995), 274-275; and Thomas H. Murray, former analyst and 

Central Intelligence Agency and Senior Vice President of Sequoia Associates, In

author, 18 January 1999. 12 U.S. Commission on the Roles and Capabilities of the United States Intelligen

“Chapter 8: Improving Intelligence Analysis,” in Preparing for the 21st Century: 

U.S. Intelligence (Washington, DC: GPO, 1 March 1996) URL: http://www.

su_docs/dpos/epubs/int/int012.html. 13 Philip Seib, “Intelligence Gathering Remains a Vital Function,” 1 March 1999, D

News, URL: <http://www.dia.ic.gov/admin/EARLYBIRD/990302/s199990302g

accessed 2 March 1999. 14 Andriole, “Indications, Warning, & Bureaucracies,” 8. 15 James P. Finley, “Nobody Likes to be Surprised: Intelligence Failures,” Milita

Bulletin 20, no. 1 (January-March 1994): 18-19.
4

attack. The analysis failure occurred in where, when, and why. Joseph Ny

“analytical disarray in 1978 that prevented the drafting of any estimate about 

the Shah, and the 1989 prediction that Saddam Hussein would not make trou

next three years.”17 Members of the Senate Select Committee on Intelligen

voiced their concern about the Intelligence Community’s inability to warn of 

nuclear tests, calling it the “greatest failure for more than a decade.”18 Other

that by improving intelligence analysis the Intelligence Community can bet

and protect U.S. embassies from terrorist threats.19

Some intelligence analysts believe that the ratio of intelligence successes to 

actually quite good. Arthur S. Hulnick, a notable veteran in the intelligence pr

one such believer. He stated in a conference paper delivered at the 1987 Annu

of the American Political Science Association that

the products of the [Intelligence Community] reflect continued growth i

expertise of the analysts, a broader and deeper range of subject materials

an increasingly sophisticated data base from which to work. Coupled 

advances in presentational methodologies, the intelligence product is 

useful, timely and relevant than ever before.20

Despite his optimism for intelligence analysis, Hulnick agrees that th

process itself is worthy of research.”21

Because of the Intelligence Community’s failures to foresee a number of s

important foreign developments, intelligence consumers have tended to dis

analysis of intelligence information. “Whether analysis is good or 

policymakers will care less about it than they do about collection. They may 

it, seeing it as naïve speculation by junior bureaucrats that wastes their tim

Hilsman, former Assistant Secretary of State for Intelligence Research an

Secretary of State for Far Eastern Affairs, argues that the benefits of funding a 

of field agents is not worth the cost or risk. Covert action tempts presidents i

the hard work of analysts and diplomats on an “easy fix,” damages America’s 

and is antithetical to a democracy. He concludes that the Intelligence Commu

16 Finley, “Nobody Likes to be Surprised: Intelligence Failures,” 20. 17 Joseph S. Nye Jr., “Peering into the Future,” Foreign Affairs 73, no. 4 (July/August 1

18 “Don’t Blame the CIA,” Economist, 23 May 1998, 26. 19 John Kifner, “Raids by U.S. Agents and Tirana Police Reportedly Thwart Attack o

New York Times, 21 August 1998, A13. 20 Arthur S. Hulnick, “Managing Intelligence Analysis: Strategies for Playing the 

International Journal of Intelligence and Counterintelligence 2, no. 3 (Fall 1988): 34

21 Hulnick, 329. 22 Richard K. Betts, “Intelligence Warning: Old Problems, New Agendas,” Paramete

(Spring 1998): 33.
5

overcome this credibility problem, it must improve its ability to produce ac

and useful intelligence.

Furthermore, the military’s use of precision, information-intensive we

pressure on the intelligence analyst to deliver useful intelligence at faster ra

of modern warfare, fast and precise weapons demand fast and precise int

intelligence analyst who is unable to conduct analysis quickly can 

information-bottleneck in this environment. It may be arguable that the use 

analytical techniques helps to speed up the analytical process. This assertio

tested, and is beyond the scope of the present research.

Terms of Reference

Quantitative intelligence analysis separates the relevant variables of 

credible numerical measurement. Qualitative intelligence analysis breaks do

ideas that are difficult to quantify into smaller components for better un

Structured methodologies are various techniques used singly or in co

separate and logically organize the constituent elements of a problem to enh

and decisionmaking.25 Some structured methodologies, such as organized 

techniques, complement the analyst’s intuition and facilitate creativity. 

Non-structured methodology is intuitive. Intuition is a feeling or instinc

use demonstrative reasoning processes and cannot be adequately explained 

with the available evidence. For the purposes of this study the term “im

referring to qualitative intelligence analysis, means to increase, either singu

combination, without decreasing, either singularly or in any combination, 

specificity, or timeliness of the result.

This study assumes that exploiting structured methodologies fo

intelligence analysis does not invariably subordinate one’s intuition, 

experience. The analyst must still make a conscious choice as to whether h

conclusion based on exploiting a structured method or reject it in favor o

judgment. Moreover, intuition plays an inescapable role in analysis. How

intuition can usually be relied upon to provide effective solutions to simpl

cannot cope with the highly complex problems that routinely face intelligen

Many factors affect the quality of intelligence analysis. This study is lim

how structured methodologies may influence the results of qualitativ

analysis. Due to time and cost considerations, it was necessary to restrict th

23 Roger Hilsman, “Does the CIA Still Have a Role?” Foreign Affairs 74, no. 5 (Sep

1995): 104-116. 24 Clark, 30. 25 Jones, xi-xvi.
6

the Unified Commands. Specifically, experiments were conducted with i

analysts from the U.S. Special Operations Command (SOCOM), the U

Command (CENTCOM), the U.S. Southern Command (SOUTHCOM), an

Joint Forces Command (JFCOM), formerly Atlantic Command (ACOM). B

these limitations, the evidence presented here should be considered only prov

conclusive.

THE ART OR SCIENCE DEBATE

A long-standing debate exists within the Intelligence Community about wh

should be invested in structured methodologies to improve qualitative i

analysis. At the heart of this controversy is the question of whether intelligen

should be accepted as an art (depending largely on subjective, intuitive judg

science (depending largely on structured, systematic analytic methods). Res

question is necessary to provide direction and determine an efficient an

approach to improve analysis. If qualitative intelligence analysis is an art, the

improve it should focus on measuring the accuracy of one’s intuition, sele

analysts with the best track record, and educating them to become experts in a 

If, on the other hand, qualitative intelligence analysis is a science, then analyst

trained to select the appropriate method for a given problem from a variety o

methodologies and exploit it to guide them through the analytical process.

Qualitative Analysis as an Art

Proponents of this argument contend that qualitative intelligence analysis de

infinite number of variables that are impossible to operationalize because they 

adequately quantified or fully collected. A foreign nation that attempts to conceal 

information on many of the same variables that an analyst seeks will further com

analytical function. Because in many cases the variables are so complex, cou

incomplete, attempting to analyze them using scientific methods is pseud

Therefore, any attempt to make predictions based on quantifying these variables is 

Other arguments attack the rational-actor assumption made by mos

approaches to qualitative intelligence analysis. “Science can be of little help wh

with the often irrational and unpredictable human mind.”28 But these critics ac

that the traditional approach to qualitative analysis, being an art, is vulnerabl

— “humans are still fallible in matters of analysis and response.”29

26 Richard K. Betts, “Surprise, Scholasticism, and Strategy: A Review of Ariel Levite’s 

and Strategic Surprises (New York: Columbia University Press, 1987),” Internati

Quarterly 33, no. 3 (September 1989): 338. 27 John L. Peterson, “Forecasting: It’s Not Possible,” Defense Intelligence Journal 3

1994): 37-38. 28 Finley, “Nobody Likes to be Surprised,” 40.
7

most successful analysts take an inductive approach, in which powe

recognition are enhanced and intuition is elevated.”30 They insist that this 

lead to more creative and original analysis. They argue that qualitativ

analysis is an art because it is an intuitive process based on instinct, e

experience.31 Gary Klein, a cognitive psychologist and pioneer in patter

makes some convincing arguments to support this claim. He challenges the 

using scientific methods for qualitative intelligence analysis outside the cla

“It is time to admit that the theories and ideals of decisionmaking we h

the past 25 years are inadequate and misleading, having produced unused 

ineffective decision training programs and inappropriate doctrine.”32 He con

The culprit is an ideal of analytical decisionmaking which asserts 

must always generate options systematically, identify criteria for ev

these options, assign weights to the evaluation criteria, rate each op

each criterion and tabulate the scores to find the best option.... The t

term is multiattribute utility analysis.33

He concludes that: 

[t]hese strategies sound good, but in practice they are often disapp

They do not work under time pressure because they take too long. Ev

there is enough time, they require much work and lack flexibility for h

rapidly changing field conditions.34

It is important to note that the majority of Klein’s research has been c

leaders in the armed forces and emergency service providers such as fir

specifically members of the Intelligence Community. There has always b

cultures between operators (commanders and other decisionmakers whos

accomplishing the mission at hand) and intelligence analysts. As intellig

have attempted to bridge the gap between the two cultures, they have tende

same intuitive decisionmaking strategies commonly used by operators.35

29 James P. Finley, “Intelligence Failure Matrix,” Military Intelligence Professional 

1 (January-March 1994): 14. 30 Tom Czerwinski, ed. Coping with the Bounds: Speculations in Nonlinearity in 

(Washington: National Defense University, 1998), 139. 31 Czerwinski, 139. 32 Czerwinski, 140. 33 Czerwinski, 140. 34 Czerwinski, 141. 35 The relationship between operators and intelligence analysts has also been the s

controversy. For a good summary see Jack Davis’ article “The Kent-Kendall D

Studies in Intelligence 35, no. 2 (Summer 1991): 37-49.
8

the value of structured analysis is that intelligence failures may less acc

attributed to problems with analysis than to “the decisionmakers who co

products of intelligence services.”36 Given that intelligence consumers c

perform their own analysis of intelligence information, they may care less ab

analysis than about collection issues, for example.37 Nonetheless, if analys

make it possible for policymakers to focus “on the methodologies of 

intelligence producers, they would be more sensitive to the biases and leaps of 

analyses passed to them,”38 even if the improvement in quality of analysis wer

Opponents of scientific methodologies further claim that, because qualitative i

analysis is largely an intuitive exercise, analysts should be encouraged to spe

be educated into becoming experts in their field. These opponents question the 

scientific methods and object to others who would use such methods to chal

expert opinion. To improve analysis, they argue, the analyst must improve h

ability by gaining “area knowledge” through frequent contact with the sub

study. In the face of these challenges, however, and prior to further debati

degree the quality of analysis may be improved by using structured method

research question remains unanswered: Will the use of structured method

qualitative intelligence analysis?

A Rebuttal. At first glance it seems that opponents of the scientific ap

criticizing the results of scientific methods. Yet they offer no empirical data to s

approach produces more accurate results, the scientific or the intuitive. The 

denounce the amount of time it takes to scientifically analyze a problem. Ho

should recognize that there are many scientific methodologies availab

decisionmaking, and selecting and using one need only be as complicated 

consuming as the analyst wishes.39

Unquestionably, the complexity of some scientific methodologies prevents 

being regularly exploited by most intelligence analysts. A graduate of the Joi

Intelligence College (JMIC) made the following observation in his Masters the

A major issue common to all the methods is customer acceptance. Due t

highly mathematical nature of [Bayesian Decision Analysis], many user

feel uneasy trusting the resulting assessments. This will only be over

through proper training of the analysts using [Bayesian Analysis] 

repeated exposure to Bayes on the part of decisionmakers.40

36 Richard K. Betts, “Analysis, War, and Decision: Why Intelligence Failures Are Inevit

Politics 31, no. 1 (October 1978): 61. 37 Betts, “Intelligence Warning: Old Problems, New Agendas,” 26-35. 38 Betts, “Analysis, War, and Decision: Why Intelligence Failures Are Inevitable,” 83. 39 Lieutenant Colonel W. Frank Ball, USMC (Ret.), and Morgan D. Jones, “Impro

Commanders’ Intuitive Decisionmaking Skills,” Marine Corps Gazette (January 199

40 Captain David Lawrence Graves, USAF, Bayesian Analysis Methods for Threat 

MSSI Thesis (Washington: Defense Intelligence College, July 1993), second page of 
9

indictment of other, more simple and appropriate methodologies.

It is apparent that what these critics are saying is that no one in the hea

actionable intelligence uses the scientific approach, so why bother? This 

important questions: 

(1) Which analysts, if any, are exploiting scientific methodologies wh

qualitative intelligence analysis? 

(2) Which approach (the intuitive or the scientific) produces the more ac

and useful result? 

Other critics concede that utilizing scientific approaches to solv

intelligence problems may improve the product of analysis, but they a

benefits are only minor. They insist that allocating resources to other 

reorganizing the intelligence bureaucracy, increasing manpower, an

communication will better improve the intelligence product. One cannot de

of factors affects the quality of intelligence analysis, but improvements to 

not have to be pursued through a linear approach, nor should effort be wa

determine which approach will yield the most significant improvemen

sufficient evidence that a reform promises improvement, it should be pursue

Qualitative Intelligence Analysis as a Science

A few analysts see qualitative intelligence analysis as a science. Since 19

Intelligence Agency (CIA) has experimented with quantitative methods in 

apply innovations in the behavioral sciences to political intelligence anal

time neither the Departments of State nor Defense had attempted to integra

methodologies into their political intelligence production.42 Much of 

incorporate quantitative methodologies into the realm of qualitative intellig

was rejected by analysts because the scientific methods were thought to be 

focused and not relevant to the questions the intelligence analysts were add

analysts tended “to be skeptical of any form of simplification inherent in t

of probabilistic models.”43

In spite of the skepticism, advocates of using a more scientific approach 

intelligence analysis assert that science is a necessary tool to use whe

qualitative analysis. They argue that, although it is impossible to consider 

when conducting analysis, one can identify key variables and weigh the

And although much may be unknown, identifying what is known and 

41 Richards J. Heuer, Jr., “Adapting Academic Methods and Models to Governm

Quantitative Approaches to Political Intelligence: The CIA Experience, ed. Richa

(Boulder: Westview Press, Inc., 1978), 1. 42 Heuer, 2. 43 Heuer, 6.
10

scientific approach to intelligence analysis notes, the identification and use 

strategic insights can yield intelligence of more value than merely contin

fathom, through the “art of intelligence,” transient ideographic actors or 

“[Y]ou can disregard [military or political leaders’] madness and genius 

than societal, economic, or political trends (which are scientifically id

analysis.”44

 Proponents of qualitative analysis as a science argue that scientific 

analysts determine the relevancy of information and form conclusions, 

analysts do not perform well on their own. Hall makes the point that:

in the absence of concrete guidance or structure[,] analysts generally 

well in forming conclusions and in discerning relevancy.... They so

have difficulty in figuring out relationships among seemingly unco

pieces of information and in forming fragments of information into 

ent whole.45

If qualitative intelligence analysis is art, there is also a concern that the 

in love with his art and be reluctant to change it even in the face of new 

more scientific and objective approach encourages the analyst to be an hon

not an advocate. 

Science in Other Types of Qualitative Analysis. Scientific methods h

effectively to assist qualitative analysis in areas outside intelligence. An 

Economist discusses the use of mathematical formulas to solve “norm

problems that most managers argued could not be solved through 

approaches.46 These managers argued that in their decisionmaking process 

an intuitive art rather than a rational science. However, science and the use 

methodologies in this case proved to be a more systematic and profitable 

instinctive trial-and-error. Although many of the managers remained skep

methods would continue to work, these methods do provide another 

decisionmaking that should not be ruled out without evidence from compar

Although still not a standard practice in the private sector, some c

increasingly exploiting quantitative methods and technology to help them 

decisions, increase profits, and compete in the information revolution.4

quantitative methodologies to qualitative analysis facilitates management of 

44 Captain David T. Resch, USA, “Predictive Analysis: The Gap Between 

Practitioners,” Military Intelligence Professional Bulletin 21, no. 2 (April-June 19

45 Lieutenant Colonel Wayne M. Hall, USA, “Intelligence Analysis in the 21st Ce

Intelligence Professional Bulletin 18, no. 1 (January-March 1992): 9. 46 “Art Hammer: The Numbers Man,” Economist, 8 August 1998, 56. 47 Rod Newing, “Consumer Information is the Fuel of a New Industrial Revolu

Times, 3 February 1999, B7.
11

same approach to improve qualitative intelligence analysis.

Inside and Outside of the Classroom. Many proponents of exploi

methodologies to aid intelligence analysis assume that these methods are 

in real-world analysis. Hulnick presents the typical view:

Advances in analytic methodology seem to have grown remarkabl

past decade. While a discussion of explicit methods goes beyond the 

this article — and the knowledge of the author — a wide variety of 

attacking a problem exists. Managers of intelligence must assume t

lysts try different techniques, sometimes by giving the problem to 

groups for analysis by different disciplines, sometimes by seeking an 

expert to provide competing analysis.48

 The former chief of the Analytic Support Group in the CIA’s D

Intelligence agrees with Hulnick. “An analyst must have a repertoir

techniques to apply in problem solving. There now exists a rich set of analy

such as pattern analysis, trend prediction, literature assessment, and statistic

The president of Evidence-Based Research, Inc., also expressed the opin

intelligence analysts use some type of systematic approach when conduct

intelligence analysis.50

This assumption that intelligence analysts use these various scientific 

based not on empirical evidence, but on the belief that experience alone doe

expert, and that an expert must have a tool for structuring knowledge to effi

problem. Yet the numerous interviews conducted in support of this 

intelligence analysts from all different agencies and backgrounds, positiv

this assumption. Of the 40 intelligence analysts interviewed in direct suppor

only one maintained that he regularly used a structured methodology (link 

could adequately explain the method and provide current examples o

Although the other analysts who attempted to explain their analytical 

described how they were tasked and how they collected information and p

dissemination, they consistently and strikingly made no mention of 

interviews serve only to confirm Klein’s more detailed study: “After stud

experienced decisionmakers and 450 decisions, we concluded that 

approach to decisionmaking is typical.”52

48 Hulnick, 331-332. 49 Clark, 11. 50 Richard E. Hayes, president of Evidence-Based Research, Inc., interview by author, 9 

51 Tim DiPace, U.S. Atlantic Command Anti-Terrorism/Force Protection Office

author, 19 March 1999. 52 Czerwinski, 144.
12

Evaluating the analytical effort in the PURPLE SUNSET wargame is dif

because there was so little to measure. Despite assertions by some that ‘the 

lyst is at the center of the intelligence process,’ the... students virtually ign

analysis. ... As a result, there was little time or effort dedicated to thoug

analysis and dissemination.53

This observation was confirmed during the interviews with the JIC a

repeated complaint was the analyst’s lack of time to devote to thoughtful i

analysis. In a separate interview at CIA, it was revealed that in spite of most i

analysts having a broad definition of what analysis involves, they spend lit

effort conducting analysis.54

As noted earlier, Betts asserts that many policymakers see little or n

intelligence analysis because it is based on the views of less experienced person

may be the case because so many intelligence analysts are reluctant to explo

methodologies, and most intelligence analysts are junior to the decisionm

support. Decisionmakers typically form conclusions based on intuition develop

years of experience. Therefore, the intelligence analyst who provides an assess

on inexperience and his own (lesser developed) intuition should not expect 

seriously. To be credible the analyst must use objective scientific method

support his conclusions.

While the research for this study sought to collect definitive evidence of fr

widespread use of scientific methodologies within the Intelligence Communit

anecdotal information could be obtained showing that such methodologies a

used by intelligence analysts in real-world situations, despite the ready ava

these methods. The National Warning Staff, for example, uses only one q

analytical methodology, a modified-Bayesian technique.56

A lack of expertise in exploiting scientific methodologies appears to be at 

the Intelligence Community’s failure to use them. One cannot expect intelligen

to exploit these methodologies unless they are trained how to select the 

analytical method and apply it to a real-world analytical problem. Analysts 

have made use of computer programs, such as FACTIONS and Policon, b

quantitative methodology, to assist in political analysis and warning. In one a

while both intuitive-based and scientific-based political intelligence fore

accurate about 90 percent of the time, intelligence analysts using Policon pro

specific and less ambiguous intelligence.57 Although this example provid

53 Stewart C. Eales, Playing with Intelligence: Officer Application of Intelligence in t

SUNSET Wargame, MSSI Thesis (Washington, DC: JMIC, August 1997), 121-122. 54 An analytic methodologist in CIA’s Directorate of Intelligence, interview by auth

1999. 55 Betts, “Intelligence Warning: Old Problems, New Agendas,” 33. 56 Warning analyst at the National Warning Staff, interview by author, 05 November 19

57 Feder, 274-275.
13

Community share, compare, and exploit these methodologies? 

Qualitative Intelligence Analysis as Both Art and Science

The fallacy in the art or science debate may be the “either/or” p

qualitative intelligence analysis is not exclusively an art nor a science, then 

considered a combination of both intuitive and scientific methods: 

In our world is an infinite set of problems which have no logically co

answer; there are some problems which any framework alone cannot 

Yet strategists must not live by inspiration alone. Inspiration unsupp

rigorous analysis becomes adventurism [sic]. Thus intuitive gifts 

paired with an effective theoretical framework.58

Availability of Structured Methodologies

In order to expand the applicability of quantitative methodologies 

problem solving, relatively uncomplicated methods have been devel

intelligence analysts structure their analysis. These structured methods, 

applied to a broad range of problems and do not focus exclusively on 

solutions, provide a scientific and demonstrable approach to analysis that ca

intelligence analyst’s objectivity. Structured methodologies do not try t

subjective insight of the intelligence analyst. Instead the intent is to 

framework to illustrate and capitalize on intuition, experience, and subjectiv

Some publications do identify which methods are best suited to solve p

of analytical problems,59 but not all intelligence analysts receive trai

methodologies. Morgan Jones, a former analyst for the CIA, has recen

Thinker’s Toolkit: 14 Powerful Techniques for Problem Solving. This boo

structured methods to aid analysis and decisionmaking. Although it is w

business and personal use, the structured methods he discusses are among t

the fraction of Intelligence Community analysts who enroll at the J

Intelligence College or other intelligence schools. 

Numerous other books and software are dedicated to teaching an analys

one specific structured methodology. The Rand Corporation, for example, 

providing a specific methodology for predicting ethnic conflict. The auth

model divided into three stages meant to help an analyst identify the 

58 Steven R. Mann, “Chaos Theory and Strategic Thought,” Parameters 22, no. 3 (Au

59 For example, David A. Schum, Evidence and Inference for the Intelligence Ana

and II (Lanham, MD: University Press of America, 1987); Defense Intelligence 

Methodology Catalog: an Aid to Intelligence Analysts and Forecasters (Washin

1983) and Jerome K. Clauser and Sandra Weir, Intelligence Research Meth

College, PA: HRB-Singer, Inc., 1975).
14

probable strife will turn into real conflict. Many technology companie

software to assist the user in problem solving by guiding him through a 

methodology and product demonstrations via the Internet.61

Exploitation of Structured Methodologies

A structured methodology provides a demonstrable means to reach a conclu

if it can be proven that, in a given circumstance, both intuitive and scientific 

provide the same degree of accuracy, structured methods have significant 

value in that they can be easily taught to other analysts as a way to structure a

their analysis. It is difficult, if not impossible, to teach an intelligence anal

conduct accurate intuitive analysis. Intuition comes with experience. What is c

even though a mixture of science and intuition is probably needed to produ

qualitative intelligence analysis, structured methodologies are severely neglect

the rare cases where a specific methodology is regularly used, there is too mu

on only one method and not a mix of various methods. No one method is app

every problem. What is needed is a set of analytical tools that can be applied i

or in combination as appropriate.

Why Are Structured Methodologies Not Used?

Structured thinking is radically at variance with the way in which the human 

the habit of working.62 Most people are used to solving problems intuitively b

error. Breaking this habit and establishing a new habit of thinking is an extrem

task and probably the primary reason why attempts to reform intelligence an

failed in the past, and why intelligence budgets for analytical methodology hav

extremely small when compared to other intelligence functions.63

Under the accelerating pressures of time, intelligence analysts feel that 

analytical approaches are too cumbersome.64 They also sense that with the inc

of structured methods comes increased accountability. An excessively 

intelligence analyst is less confident in his own ability and will not unnecessa

himself to criticism. Reliance on intuition and the resistance to incorporat

approaches complicate the task of getting analysts to exploit structured metho

assist qualitative intelligence analysis.

60 Ashley J. Tellis, Thomas S. Szayna, and James A. Winnefeld, Anticipating Ethnic Co

Monica: The Rand Corporation 1997), 9. 61 Marek J. Druzdzel, “Decision Analysis Software,” 5 September 1998, Decisi

Laboratory, URL: <http://www.lis.pitt.edu/~dsl/da-software.html>, accessed 17 Feb

62 Jones, The Thinker’s Toolkit, 8. 63 Andriole, “Indications, Warning, & Bureaucracies,” 10-11. 64 Eales, 121-124.
15

conduct qualitative intelligence analysis because they are not convinced 

improve their analysis. If empirical evidence shows that such an improvem

place, then analysts may consider it worth the investment of time, effort, an

on their part to regularly exploit structured methodologies. However, if s

evidence shows that exploiting structured methodologies will not signific

qualitative intelligence analysis, then analysts can devote their time 

enhancing their intuitive skills. 

The results from the series of experiments conducted for this study may 

empirical data needed by intelligence analysts to make a decision on wheth

begin exploiting structured methodologies. 

THE EXPERIMENT

The Control Group Versus the Experimental Group

Arguments both for and against the use of structured methodologie

qualitative intelligence analysis may have merit. Therefore, empirical evide

to determine if exploiting structured methodologies will improve qualitativ

analysis. A direct comparison of analytical results derived from using a stru

with results derived solely from an intuitive approach is necessary f

experiment. The author designed an experiment to make such a comparison

The purpose of the experiment was to compare the analytical conclusi

individual analysts in two different groups: a control group and an experi

Individual analysts in the control group would use the traditional intuitiv

analyze two different qualitative intelligence problems. Analysts in the 

group would exploit a specific structured methodology to aid them in the

these same two problems. Analysts’ answers would be scored as eith

incorrect and compared statistically to determine whether the experimen

significantly better than the control group. 

Competing Hypotheses and Level of Significance

The experiment would test the truth of both the null hypothesis 

alternative, or research, hypothesis (H1). The level of significance was set at 

experiment was conducted.65

65 A one-tailed test of statistical significance is appropriate to determine whether t

and research hypotheses, given their directional nature. Two-tailed tests are 

determine if factors such as rank, experience, education or branch of service ma

the results of the experiment, as no assumption was made as to how these factor

qualitative intelligence analysis. The author is grateful to Francis J. Hughes, JM

invaluable assistance in identifying statistical methods appropriate to this study.
16

H1: Exploiting structured methodologies will improve qualitative intelligen

Determining Sample Size and Choosing a Statistical Test

Due to the real-world commitments of the intelligence analysts at each 

impossible to predetermine how many analysts would be sampled during the e

Not knowing the sample size or expected frequencies of each cell beforeha

impossible to select an appropriate statistical test to analyze the information u

was collected. On the day of the experiment each JIC Commander deci

intelligence analysts would be available to participate. The decision of w

participate was made based upon that day’s schedule as driven by operational re

The decision was not made based upon an attempt to alter the results of the expe

The available analysts from each JIC were randomly assigned to either the 

experimental group. Both the control group and the experimental group had 

each. The total number of analysts sampled (N) during the experiment was 26. 

small sample size and associated low cell frequencies, Fisher’s Exact Probabili

used to determine statistical significance for the hypotheses and for the influ

controlled factors (rank, experience, education and branch of service).66

Collection Procedures

Analysts in both control and experimental groups completed a one-page qu

to provide the demographic information needed to crosscheck the validity of 

gathered from the experiment. Each analyst was asked to identify any 

familiarity they had with using various structured analytical methodologie

frequently, if ever, they used structured methodologies during real-world ana

the analysts in the control group were dismissed and asked to return at a late

analysts in the experimental group were given an hour of training on a specific 

methodology—hypothesis testing. The training was standardized at each JIC 

from the same lesson plan and workbook.67 The analysts in the experimental 

asked not to discuss their training with the analysts in the control group un

experiment.

When the analysts assigned to the control group returned, the two gr

segregated. Each analyst in each group was then given a map, both intelligence 

66 Marija J. Norusis, SPSS 8.0 Guide to Data Analysis (Upper Saddle River, NJ: P

1998), 315. Fisher’s Test produces a probability for a one-tailed test. Several theo

how one can use Fisher’s Test to produce a probability for a two-tailed test. The m

method is to simply double the one-tailed test result, and not to exceed a prob

Doubling the probability for one-tailed results is the method the author used. Other 

more complicated mathematical formulas, but yield similar probabilities. 67 Morgan D. Jones authored the Hypothesis Testing Lesson Plan and workbook, whic

by permission. For further information on his hypothesis testing methods, see Ch

Jones’ The Thinker’s Toolkit.
17

was asked to analyze the first scenario, then to complete the answer sheet 

within one hour. Once this task was completed, they were asked to analy

scenario and complete its answer sheet as individuals within 30 minutes. At 

allotted time they were asked to turn in their completed answer sheets and a

they had made. Both groups were given the same amount of time to analyze 

and to complete the answer sheets, so neither group had an advantage of 

other. All notes made by the analysts were collected to determine what app

in the control group used and to ensure that analysts in the experimental g

applied the hypothesis testing method. All analysts were interviewed after c

experiment to gain further insights into how they approached analyzin

problems presented both in the experiment and in their official duties. 

Only the first question on the answer sheet for each intelligence problem 

was scored as either correct or incorrect. For the first intelligence scenar

answer was to expect an attack at Port Mia. For the second scenario the c

was that the intentions of the adversary government were peaceful.

Map for Intelligence Scenarios 1 and 2.
18

INTELLIGENCE SCENARIO 1

You provide strategic intelligence to the military and political leaders of your co

Federal Republic of Ysla (FRY). The Sovereign Autocracy of Penin (SAP) and F

war. SAP is conducting air reconnaissance, special operations, and air bombin

against FRY to prepare the way for a military attack. You have determined that d

unique geography of your country and the limited amphibious capabilities of 

military, only two places in FRY are vulnerable to an amphibious attack by 

military. SAP naval forces can conduct operations out to 250 nautical miles

defend FRY you need to determine how and where the SAP military invasion w

Using the above assumptions and the scenario map, conduct an analysis of the 

intelligence information (it is listed chronologically). Keep in mind that the FRY 

Leadership is concerned the SAP military will launch a diversionary attack aga

Mia while the main attack is against Port Eyer. They want your analysis t

information on whether the SAP military is planning to use deception, dive

neither in conjunction with their attack.

Based on order of battle from multiple sources you know the majority of SA

amphibious forces are docked in port at Banes. Multiple sources indicate that 

relocated most of its ground forces to Haba and reassigned them to the 

modernized Third SAP Corps. COMINT reporting indicates that a new SA

General was assigned as Commander of the Third SAP Corps. On January 2

sources report that General Sage, the SAP Supreme Military Commander, le

encourage the nearby countries to remain neutral during the conflict; his travel 

and return date are unknown. The SAP Government publicly announced a b

visits to the SAP coast. 007, one of FRY’s top HUMINT agents working in Ca

reported the SAP main attack will occur near Port Eyer and may be prec

diversionary attacks elsewhere in FRY. 086, another FRY spy, working ne

reported that SAP would land a small force at Port Mia before the main attac

Eyer. Multiple sources indicate the SAP military has conducted amphibious 

with submerged obstacles. COMINT reporting indicated major moves of SA

military forces to the northwest part of SAP. COMINT reports also indicated 

movement of forces from Capitol City to the northeast. SAP bombing raids aga

Eyer number twice that against Port Mia. The SAP bombing of Ports Eyer and 

restricted the flow of traffic in and out of both areas. The majority of 

reconnaissance and special operations have been against Port Eyer. On Febru

FRY air reconnaissance showed no changes in ships at Haba. On February 2

air reconnaissance showed no change in shipping traffic in Banes. On March 

air reconnaissance of both Haba and Banes showed increased shipping arou

only. On March 30th, FRY air reconnaissance showed 20 new landing craft d

Haba; air reconnaissance at Banes on the same day was inconclusive due to w
19

QUESTIONS FOR INTELLIGENCE SCENARIO 1

1. Where and how will the SAP’s military attack FRY?

2. Support your conclusion. (Please attach any other notes you made.)

3. What other possible courses of action did you consider?

4. What analytical method did you use? How did you analyze this scenar

5. Was this scenario similar to another current or historical event?

6. What is your assessment of the validity of this experiment?

7. Please provide your name and rank. Your answers and identity w

anonymous!
20

INTELLIGENCE SCENARIO 2

Due in part to your exceptional analysis, the Federal Republic of Ysla (FRY) su

defended itself from a major invasion by the Sovereign Autocracy of Penin (SAP). O

of last year, the war ended in a stalemate after the U.S. intervened, and both FRY 

governments signed an armistice. U.S. troops established a presence in FRY to d

from attacking again. The SAP Government is concerned that the FRY military may 

attack on SAP. The SAP Government wants the U.S. to withdraw its forces from F

SAP signs any peace treaty. Tensions between the FRY and SAP remained high t

the year. However, at the beginning of the New Year, the leader of SAP, Presiden

appeared to be making peaceful overtures to the FRY Government. Your super

asked you to analyze the evidence below and determine whether the SAP Govern

new and peaceful intentions.

On January 15th, the SAP News Agency, the official news agency of the SAP Go

described an interview that President Castrol had with the editorial committee of a C

newspaper, The Island Times on January 10th. President Castrol proposed: “a pea

ment...may be concluded between SAP and FRY and the military forces of both 

may be cut drastically under the condition where the U.S. imperialist troops are w

from FRY.”

In a subsequent meeting with U.S. Senator Harley (Democrat) in mid-January, Presi

trol made it clear that a peace agreement between SAP and FRY could be conclud

U.S. forces were withdrawn.

U.S. intercepts since mid-February of secure, high-level, SAP military command co

tions have revealed no indications of unusual military activities or preparations for h

Over the next seven months the SAP News Agency was consistent in reporting 

peace proposals, only occasionally reverting to a “hard line” stance and then attrib

stance to unnamed spokesmen.

Last year, shortly after the armistice was signed, Red Cross delegates from both 

SAP began meeting on a regular basis to share information to help locate and id

diers from both sides who were missing in action during the war. Initially, the S

Agency coverage of the Red Cross meetings was very negative. After January of 

however, the tone of the coverage changed. The reporting became essentially fa

objective.

In February of this year, the SAP Government began a national blood donation 

claiming shortages, although the reason for shortages was not disclosed nor had th

any major disaster causing shortages to occur. Later the same month, President Ca

another “unofficial” interview to a journalist from the Washington Times. In the interv

ident Castrol again iterated that a peace treaty could be signed between SAP and F

U.S. troops withdraw from FRY.
21

On March 15th, SAP announced the execution of five SAP nationals who were m

FRY spy ring during the war.

The commander-in-chief of SAP armed forces retired on March 29th and was re

deputy commander-in-chief.

The number and tone of SAP propaganda broadcasts changed noticeably in th

number of anti-FRY radio broadcasts declined from about 25 per day to about 1

per day. Also, the usual abusive and slanderous words used in the broadcast

down considerably. 

In April, the two governments signed an agreement allowing each other’s mili

unrestricted movement without pre-travel approval.

Monitoring of SAP News Agency wire service releases containing anti-FRY 

declined from a high of 27 percent in January to 10 percent in April.

Over the past six months SAP military training has maintained its normal cycle.

Training flights of SAP fighter aircraft and bombers have been significantly curta

past four months. In that same period the FRY Government has not detected an

the SAP Government to infiltrate agents or armed guerrillas

The unusually large number of SAP reservists called to active duty during th

discharged this month.

SAP ground-force medical units conducted a large-scale, out-of-training-cy

yesterday.

U.S. photoreconnaissance reveals that dozens of landing craft used in the SAP 

ations against FRY during the war have not been repaired.

INTELLIGENCE SCENARIO 2
22

QUESTIONS FOR INTELLIGENCE SCENARIO 2

1. What are the intentions of the SAP Government? 

2. Support your conclusion. (Please attach any other notes you made.)

3. List all the possibilities you considered.

4. What analytical method did you use? How did you analyze this scenario?

5. Was this scenario similar to another current/historical event?

6. What is your assessment of the validity of this experiment?

7. Please provide your name and rank. Your answers and identity will 

anonymous!
23

HYPOTHESIS TESTING ILLUSTRATION: INTELLIGENCE SCE

Possible Hypotheses are: 

H1: Main attack at Port Eyer, no major deception or diversion.

H2: Main attack at Port Eyer, with deception pointing to Port Mia.

H3: Main attack at Port Eyer, with diversionary attack at Port Mia.

H4: Main attack at Port Mia, no major deception or diversion.

H5: Main attack at Port Mia, with deception pointing to Port Eyer.

H6: Main attack at Port Mia, with diversionary attack at Port Eyer.

Note: Letters characterize the attributed relationship between evidence a

consistent=C; inconsistent=I; ambiguous=?; evidence that is either consistent or am

hypotheses is “lined out” to show that it was considered, then disregarded, in favor 

“directional” evidence.

EVIDENCE H1 H2 H3 H

1. Multiple Sources (MS), majority of SAP Naval forces at Banes. I C C 

2. MS, majority of ground forces assigned to modernized Third 

Corps at Haba

C I C 

3. COMINT, new Marine general commanding Third SAP Corps C I I 

4. 28 Jan, General Sage visits neutral countries; return date 

unknown

? ? ? 

5. SAP bans visits to coasts C C C 

6. 007, main attack at Port Eyer with possible diversion I IC

7. 086, small force attacks Port Mia then main attack at Port Eyer I IC

8. SAP conducts amphibious exercises with submerged obstacles C C C 

9. COMINT, Marines move to northwest SAP C I I 

10. COMINT, major movement of forces from Capitol City to 

northeast SAP

I CC

11. SAP bombs Port Eyer 2X as much as Port Mia C I C 

12. SAP bombing restricts traffic in/out of both ports C C C 

13. Majority of SAP air recon and SOF directed against Port Eyer C I C 

14. 19 Feb and 25 Feb, no change in ships at Haba or shipping at Banes I C I 

15. 8 Mar, increase in shipping at Banes I C I 

16. 30 Mar, 20 new landing craft at Haba; Recon at Banes is inconclusive due to weatherCIC

Hypothesis Testing Matrix Example for Intelligence Scenario 1.
24

Possible Hypotheses are: 

H1: Peace

H2: Business-as-Usual

H3: Deception.

Hypothesis Testing Matrix Example for Intelligence Scenario 268

EVIDENCE H1 H

1. Jan-Apr, SAP News Agency Anti-FRY statements decline to 10% C 

2. Jan-Apr, SAP fighter/bomber training sorties decline C 

3. Jan-Apr, FRY detects no SAP infiltration attempts C 

4. Jan-Jun, SAP military training cycle is normal C C 

5. Jan 15, Castrol proposes peace with FRY C 

6. Mid-Jan, Castrol says peace before U.S. troops leave C 

7. Feb, SAP News Agency begins positive reporting of ICRC C 

8. Early Feb, SAP Government blood drive I C 

9. Mid-Feb, no unusual SAP C2 activity C C 

10. Late-Feb, Castrol says peace treaty before U.S. leaves C 

11. Feb-Aug, SAP News Agency consistently reports peace proposal C 

12. Mar 15, SAP executes members of FRY spy-ring I C 

13. Mar 29, SAP CINC retires and replaced by his Deputy C C 

14. April, FRY and SAP sign agreement allowing unrestricted travel C 

15. April, SAP discharges reservists called up during the war C 

16. April, SAP conducts a large, out-of-cycle medical exercise I C 

17. April, SAP landing craft remain unrepaired C 

18. Spring, SAP propaganda broadcasts decrease in number and tone C 

68 Although assigning a C, I, or ? to an individual line of evidence is a subjective judgmen

displaying the evidence and hypotheses in a matrix the analyst can better judge each piece of 

aggregate all the evidence to determine which hypothesis is most likely true. Subjective ju

intuition still play an inescapable role in analysis. Individual analysts will seldom assign C, I, or 

way, but by visually displaying it, other analysts can question their logic, readily identify wh

evidence they disagree on, and more quickly form a consensus.
25

Tables 1 and 2 summarize the responses given by analysts in the co

experimental groups to each intelligence scenario. Tables 3-10 show how 

answered each intelligence scenario based on rank, years of experie

education, and branch of service.

A. Comparison of Responses between Control and Experimenta

B. Comparison of Responses between Civilian/Officer and En

C. Comparison of Responses Based on Years of Experien

Control Group Experimen

# of Incorrect Answers 11 10

# of Correct Answers 2 3

Table 1. Response by Group to Intelligence Scenario 1

Control Group Experimen

# of Incorrect Answers 4 0

# of Correct Answers 9 13

Table 2. Response by Group to Intelligence Scenario 2

Civilian/Officer Enlis

# of Incorrect Answers 9 12

# of Correct Answers 3 2

Table 3. Response by Rank to Intelligence Scenario 1

Civilian/Officer Enlis

# of Incorrect Answers 1 3

# of Correct Answers 11 1

Table 4. Response by Rank to Intelligence Scenario 2

Less than 10 years 10 years 

# of Incorrect Answers 12 9

# of Correct Answers 2 3

Table 5. Response by Years of Experience to Intelligence Scenar
26

D. Comparison of Responses Based on Level of Education:

E. Comparison of Responses between Military Services:

C. Comparison of Responses Based on Years of Experience: (Co

Less than 10 years 10 years or 

# of Incorrect Answers 2 2

# of Correct Answers 12 10

Table 6. Response by Years of Experience to Intelligence Scenario 2

<Bachelor Degree Bachelor Degree or 

# of Incorrect Answers 12 9

# of Correct Answers 2 3

Table 7. Response by Level of Education to Intelligence Scenario 1

<Bachelor Degree Bachelor Degree or 

# of Incorrect Answers 2 2

# of Correct Answers 12 10

Table 8. Response by Level of Education to Intelligence Scenario 2

Navy/Marine Army/Air For

# of Incorrect Answers 9 12

# of Correct Answers 1 4

Table 9. Response by Branch of Service to Intelligence Scenario 1

Navy/Marine Army/Air For

# of Incorrect Answers 0 4

# of Correct Answers 10 12

Table 10. Response by Branch of Service to Intelligence Scenario 2
27

Fisher’s Exact Probability Test was used to statistically analyze the data 

whether the null hypothesis could be rejected. This formula was also used 

test to measure any variances in responses based on the different demograp

these cases Fisher’s Test would measure the probability that the differences 

were caused by some other factor than the use of a structured methodolo

qualitative intelligence analysis. The original tables displaying demograph

control and experimental groups contained several columns, not just t

Fisher’s Test can be used to measure data only in a two-by-two matrix, so it 

to modify the original tables by combining some columns to arrange t

appropriate two-by-two matrix.

CONSIDERATIONS AND LIMITATIONS

Although this experiment did collect and analyze data on several differe

might affect the answers an analyst would provide, it could not identify a

factors. To encourage individual participation an agreement was made that 

individual analysts would not be released. To encourage the JICs to pro

analysts as possible it was agreed that the specific results of each JIC w

available only to that particular JIC. This experiment was designed to 

answers given by the control and experimental groups; it would not compa

JIC. Therefore, the way in which analysts were made available, driven as i

world events, did not alter the validity of the experiment.

Designing the Experiment

Certainly, one can find some fault in both intelligence scenarios us

experiment. Nevertheless, both scenarios were submitted to a rigorous p

revision process to ensure both scenarios were valid and reliable me

intelligence analyst’s qualitative analytical capability during the time al

experiment and appropriate for the structured methodology taught to the 

group. 

Several obstacles hinder the design of an intelligence scenario to measur

of an analyst’s conclusions. A hypothetical scenario begs the question of wh

authoritatively claim that a correct answer exists. Since the event never 

burden is on the creator of the scenario to ensure only one conclusion can 

contrast, using an actual historical event as the basis for an intelligence sce

risk that the analyst will recognize the scenario for what it is and base his c

the known outcome rather than on the evidence presented. The intelligence 

in the experiment were based on actual historical events.

The first scenario was developed in coordination with Tom Murray

analyst and trainer and currently Senior Vice President of Sequoi

Incorporated. The scenario was based on known strategic intelligence 
28

necessary to alter some of the evidence such as the names of individuals, 

geographic locations in the scenario to disguise the fact that this scenario wa

the invasion at Normandy, but the basic evidence remained unchanged. In th

the analysts were asked to determine where the actual invasion would occur.

The second intelligence scenario was based upon a case study presented in I

Research and Methodology, by Jerome Clauser and Sandra Weir. The 

examined a series of incidents in which it appeared the North Korean Gover

making a sincere effort to improve peace on the peninsula in the 1970’s.69 Th

was modified with the assistance of Morgan D. Jones. The subjects wer

determine whether the adversary in the scenario was sincerely pursuing pe

peace as a deception, or conducting business as usual.

Various students at the JMIC tested both intelligence scenarios before they 

in the experiment at the JICs. The students were asked to analyze both sce

complete an answer sheet. The answer sheet asked them to draw some concl

also asked whether they recognized the scenarios as any actual historical 

scenarios were then modified and given to another group of students to test. 

round of testing only three out of 20 students recognized that the first scenario 

to the Normandy invasion; however, only one of the three who recognized th

link provided the correct answer. None of the students could identify the actual 

as a basis for the second scenario. During the actual experiment at the JICs, fiv

subjects identified the Normandy invasion as the basis for the first intelligenc

Four of these five were in the control group and the other was in the experime

Of the five only one in the experimental group and another in the control grou

the correct answer to the first scenario. No one at the JICs identified the actua

event used as a basis for the second scenario. These checks on the historical id

of the scenarios address the potential concern of “analysis as art” proponent

familiarity with a particular region or problem would affect the outcome o

similar experiment. It would appear that this concern, though well-found

overcome by careful selection of scenarios. Scenario construction or borrowi

simple matter, as a real-world “outcome” must be captured to ensure a “correct

known. 

The questions on the answer sheets were open-ended rather than multi

Although using a multiple choice test simplifies the scoring process, it does not 

portray how analysts provide answers outside the classroom. A multiple 

defines the range of options available; in qualitative intelligence analysis, d

narrowing the range of options is a part of the answer in itself. Asking o

questions provided insights into how many possible hypotheses each analyst 

and how specific a response each analyst felt confident in providing.

69 Clauser and Weir, 358-369.
29

Rather than allow the analysts to work at their own pace, a specific perio

allotted to work each problem; however, the analysts were allowed to turn i

sheets anytime before the allotted time expired. Although it would be 

compare the number of correct answers with the amount of time spent on t

would be of little help in determining whether exploiting structured meth

improve qualitative intelligence analysis. Based on observations mad

experiment, it seems that the amount of time spent on analyzing a problem 

with the analyst correctly answering a problem or using a structured method

and experimental groups had analysts who completed both problems qu

analysts who used the entire amount of time allotted.

Most analysts work under a deadline. Allotting a fixed amount of time 

problem put pressure on the analyst (simulating real-world constraints) an

answer sheets were returned in time to be included in this study. It also he

of time constant so the results of the experiment were less likely to be taint

factor. Many of the analysts remarked that they enjoyed the opportunity to c

a specific problem without being interrupted. 

THE FINDINGS

This exploratory research provides empirical evidence suggesting that ex

structured methodology (hypothesis testing in this example) will impro

intelligence analysis. Even though this evidence is for a narrowly defined 

second intelligence scenario) and not conclusive, such evidence did not ex

experiment was conducted. Factors such as rank, experience, education, 

service did not appear to affect the results.

Description of the Findings

As Tables 1 and 2 show, the overall qualitative intelligence analysis o

experimental group was better than that of those in the control group for b

Despite this, the improvement measured in the first scenario did not meet th

statistical significance (see results for Table 1 in Table 11). Nor did the 

computed for Tables 3-10 meet the established threshold of statistical sig

results for Tables 3-10 in Table 11), confirming that the differences in 

probably not due to these factors. However, the improved qualitative a

experimental group over that of the control group in the second scenario w

significant (see results for Table 2 in Table 11). 
30

Expectations

The results were surprising. The author expected a significant impro

qualitative intelligence analysis in the first scenario, whose inherent com

seemed, would give the analysts who used the hypothesis testing method an 

over those who did not employ a structured method. But this advantag

materialize. Most of the analysts using hypothesis testing seemed to have ha

identifying all of the possible hypotheses and determining the consistency of 

of evidence with each hypothesis. 

Because the second scenario was simple and straightforward, the author th

there would be little or no advantage to be gained by structuring the problem 

the evidence. In the event, however, the structured approach proved more effec

In hindsight it was asking too much to expect the analysts to proficiently 

hypothesis testing method to complex problems like the first intelligence sce

just one hour of training. This may explain, in part, why the experimental gro

perform significantly better than the control group on the first scenario. Becau

scenario, with six hypotheses to consider, is more complicated than the secon

with only three hypotheses, one may be tempted to conclude that the hypoth

method is useful for structuring only simple problems. Nevertheless, ba

information gathered during this experiment, the author would guess that the 

testing method can be useful in structuring more intricate problems, provid

analyst has been adequately trained to proficiently apply the method.

 Table No. Description Scenario No. Fisher’s Prob

1 Control and Experiment 1 .5

2 Control and Experiment 2 .048

3 Rank 1 .844

4 Rank 2 .718

5 Experience 1 .844

6 Experience 2 1.0

7 Education 1 .844

8 Education 2 1.0

9 Branch of Service 1 .686

10 Branch of Service 2 .244

Table 11. Probabilities that Results are Due to Chance
31

Quite simply, the experimental group performed significantly better on 

intelligence scenario than did the control group. The improvement appears to 

members of the experimental group using a particular structured methodolog

members of the control group did not use any structured methodology t

analysis. The improvement does not appear to be a consequence of rank, 

education, or branch of service. The improvement measured in qualitative i

analysis during the first intelligence scenario is not significant.

Why Improvement in the Second Scenario But Not the First?

The first scenario may have affected the control group’s analysis of 

scenario. In the first scenario it was obvious that some form of deception was 

the problem was determining which attack was real and which was the dece

second scenario was more clear and direct; however, some analysts may have b

and assumed deception was being used in the second scenario because of t

exposure to the use of deception in the first scenario. It appears that those an

used the hypothesis testing method were better able to remain objective and 

evidence in the second scenario without being influenced by the first scenari

the analysts who based their analysis solely on their intuition were less object

analysis.

Another Approach to Analyzing the Data

During the experiment each analyst was asked one question from both i

scenarios. Their answers were marked as either correct or incorrect. Thus, an a

answered both questions correctly scored 100 percent. If an analyst ans

question correctly and the other incorrectly, he scored 50 percent. If he ans

questions incorrectly, he scored 0 percent. The scores from analysts who used a 

methodology ranged from 50-100 percent. The analysts who did not use a 

methodology ranged in scores from 0-50 percent. While three analy

experimental group answered both questions correctly, no one in the con

answered both questions correctly. By this measure, exploiting a structured m

improved qualitative intelligence analysis.

To Reject H0 or Not?

Considering all the arguments, the information collected, and the an

interpretation of that information, the null hypothesis—that exploiting 

methodologies will not improve qualitative intelligence analysis—can be r

course there are several qualifiers that affect the veracity of the alternate hypoth

exploiting structured methodologies will improve qualitative intelligence an

single most critical qualifier is that analysts must be adequately trained to ensu

able to proficiently apply the appropriate structured methodology. 
32

gy g

complicated and ambiguous that the correct answer will never be revealed. 

as this experiment demonstrated, there are instances where qualitativ

analysis can be improved by exploiting structured methodologies. Only 

different structured methodologies to different types of intelligence prob

find out where and when structured methods are most effective. 

This experiment examined only one structured methodology—hypo

Other experiments that examine different methods, different problem

different analysts from different agencies could be pursued by other invest

meantime, analysts within the Intelligence Community should be encoura

various structured methodologies. 

IMPLICATIONS AND RECOMMENDATIONS

A Conclusion

This study has produced an answer to the question: “Will exploiti

methodologies improve qualitative intelligence analysis?” The approach ta

the question was straightforward. Competing hypotheses were developed 

research question, arguments supporting both hypotheses were explored

experiment was designed to provide empirical evidence, and a statistical te

determine the significance of the evidence produced. For both scenarios the 

used structured methods outperformed the analysts who did not. The 

measured in the second intelligence scenario was statistically significant; 

null hypothesis—that exploiting structured methodologies will not impro

intelligence analysis—was rejected.

The author has determined that exploiting structured methodologies 

qualitative intelligence analysis. There are, however, several qualifiers. The 

is not necessarily automatic or significant for every case. Intelligence ana

adequately trained to select an appropriate structured methodology or c

methodologies germane to the problem and apply the method(s) profici

precise conclusion drawn from the experiment as presented in this study i

structured methodologies can improve qualitative intelligence analysis.” Fu

is needed to definitively answer the original research question.

Other Insights

Based on the numerous interviews the author conducted with intellig

from many different intelligence units and agencies, it is apparent tha

research is needed in this area of intelligence analysis. Nearly every 

different definition of intelligence analysis. Most analysts believe that a

automatically as they gather information and prepare that information for 

It is not a function for which they specifically allocate time. 
33

analysis for the analyst; the analyst still must do his own thinking. But by 

problem the analyst is better able to identify relevant factors and assumptio

and consider different outcomes, weigh different pieces of evidence, and m

based on the available information. While exploiting a structured method

guarantee a correct answer, using a structured methodology ensures th

performed and not overlooked. 

From the discussions with the test subjects after the experiment it beca

the author that analysts in the control group were not as clear in their th

analysts in the experimental group. After reading the scenarios members 

group formed a conclusion, then went back to the scenario to find evidence 

their conclusion and ignored contradictory evidence. When asked to justify 

analysts in the control group often cited some “key” information that gave t

insight. Members of the control group seemed to be looking for the 

information—the “Holy Grail”— that would make sense of everythi

approach flies in the face of the fact that, more often than not, qualitativ

analysis must be accomplished with incomplete information.

Members of the experimental group examined all evidence provided i

prior to making their decision. They felt confident that they were making the 

they could with the amount of information available. They acknowled

decision may not be the right one and added that if more evidence became 

would reevaluate their conclusion taking into account this new information

Even though most analysts had not received training in or used struct

previously, analysts who received training in hypothesis testing expressed th

exploiting such a method would aid and improve their analysis.

Recommendations

The author recommends that various structured methodologies be 

intelligence analysts of every service and agency during both initial an

training. Intelligence analysts should be encouraged to use structured m

when drafting analytical narratives. Analysts should be expected to util

methodologies to visually present their analysis during intelligence br

intelligence units conducting real-world qualitative intelligence analys

encouraged to experiment with different structured methodologies and 

other similar units in regular analytical exercises. The results from expl

structured methodologies from both real-world situations and during exerc

published and shared with other intelligence units, services, and agencies.

Further Research

The results of this experiment need to be confirmed. The author recom

series of similar experiments be conducted, either separately or in any co

test other factors that may affect the outcome of qualitative intelligence ana
34

experiment as the original except reverse the order in which the intelligence sc

given to the analysts. Alternatively the researcher may develop his own sc

present them to different test subjects in a different order and track the results t

order in which the scenarios were presented affected the results of their analys

Intensive Training. Expand the amount of time spent training the experim

from one hour to three or more hours and include more complex problems. U

scenarios as in the original experiment to test this author’s theory that mo

would have allowed the analysts to proficiently apply the hypothesis testing 

more complicated problems such as the first intelligence scenario.

Sophisticated Scenario. Future researchers may wish to conduct a similar 

using a more sophisticated scenario to determine whether design flaws in t

scenarios affected the results of the original experiment.

Before and After. Instead of dividing analysts into control and experimen

have them analyze a scenario intuitively then train the same analysts in an 

structured methodology and have them analyze the same problem using the 

technique. Compare their first response with their second response and m

improvement. A researcher utilizing this approach must develop a test to me

improvement arises from analysts’ exploiting a structured method or simply f

the same or similar problem again.

Different Method. Conduct an experiment using a structured methodology 

hypothesis testing. The researcher must develop a new intelligence scenario 

to the structured methodology being tested.

Team Approach. Instead of measuring the responses of individuals, 

analysts into teams and ask them to develop a consensus and answer the que

team. Have some teams use a structured methodology and have other team

traditional intuitive approach.

Measure Time. Instead of holding time as a constant, ask the analysts t

problems as quickly as possible and record how long it takes each analyst t

problem. Consider both accuracy and timeliness when measuring impro

qualitative intelligence analysis.

Multiple Choice. Instead of asking open-ended questions, provide the ana

multiple choice answer sheet. This will simplify the scoring process an

improvement while ensuring that every analyst considers the same possible ou

Two-Tailed Test. Develop a hypothesis and design an experiment to 

exploiting structured methodologies will cause a change in qualitative intelligen

in either direction. Also, other demographic information (such as gender

personality type) should be collected and analyzed to see if these factors affect t
35

The author’s motivation to conduct research into qualitative intellig

comes from his own experience with the difficulties common to conductin

others in analysis. Somewhat surprisingly, the present research has 

remarkably little analysts within the Intelligence Community appear to 

analytic methods, and less surprisingly, how great is the gap between the

future analytic potential. It is clear to this researcher that a moderate 

analytic training would substantially improve intelligence analysis. 
37

An analytic methodologist in Central Intelligence Agency’s D

Intelligence. Interview by author, 29 April 1999.

Andriole, Stephen J., Ph.D. Handbook of Decision Support Systems

Summit, PA: Professional and Reference Books, 1989.

________. Handbook of Problem Solving: An Analytical Methodology. N

Petrocelli Books, Inc., 1983.

________. “Indications, Warning, & Bureaucracies.” Military Intelligenc

Bulletin 10, no. 3 (July-September 1984): 6-12.

________. Methods for Intelligence Analysis, Production, and 

Unpublished research paper funded by DIA. Contract no. MDA90

December 1983.

________. Senior Intelligence Researcher and former Director of the Cy

Technology Office of the Advanced Research Projects Agency. Email interv

15 December 1998.

“Art Hammer: The Numbers Man.” Economist, 8 August 1998, 56.

Ball, W. Frank, Lieutenant Colonel, USMC (Retired), and Morg

“Improving Marine Commanders’ Intuitive Decisionmaking Skills.” M

Gazette (January 1996): 63-64.

Betts, Richard K., Ph.D. “Analysis, War, and Decision: Why Intelligence 

Inevitable.” World Politics 31, no. 1 (October 1978): 61-89.

________. “Intelligence for Policymaking.” Washington Quarterly 3, no

1980): 118-129.

________. “Intelligence Warning: Old Problems, New Agendas.” Param

(Spring 1998): 26-35.

________. “Policy-makers and Intelligence Analysts: Love, Hate, or Ind

Intelligence and National Security 3, no. 1 (January 1988): 184-189.

________. “Strategic Intelligence Estimates: Let’s Make Them Useful.” 

10, no.4 (Winter 1980): 20-27.

________. “Surprise, Scholasticism, and Strategy: A Review of Ariel Le

Intelligence and Strategic Surprises (New York: Columbia University P

International Studies Quarterly 33, no. 3 (September 1989): 329-343.
38

Procedure. Occasional Paper Number Two. Washington, DC: Joint Military I

College, 1996.

Brown, Thomas A., and Emir H. Shuford. Quantifying Uncertainty Into 

Probabilities for The Reporting of Intelligence. R-1185-ARPA. Santa Monica

Corporation, July 1973.

Clark, Robert M. Intelligence Analysis: Estimation and Prediction. 

American Literary Press, Inc., 1996.

Clauser, Jerome K., and Sandra Weir. Intelligence Research Methodo

College, PA: HRB-Singer, Inc., 1975.

Coping with the Bounds: Speculations on Nonlinearity in Military Affair

Czerwinski. Washington, DC: National Defense University, 1998.

Critical Technologies for National Defense. Ed. J.S. Przemieniecki. Washi

American Institute of Aeronautics and Astronautics, 1991.

Davis, Jack. “The Kent-Kendall Debate of 1949.” Studies in Intelligence 

(Summer 1991): 37-49.

De Francesco, Henry F. Quantitative Methods for Substantive Analysts. Lo

Melville Publishing Co., 1975.

Defense Intelligence Agency. Methodology Catalog: An Aid to Intelligenc

and Forecasters. DDE-2200-227-83. Washington, DC: DIA, December 1983. 

DiPace, Tim. U.S. Atlantic Command Anti-Terrorism/Force Protectio

Interview by author, 19 March 1999.

“Don’t Blame the CIA.” Economist, 23 May 1998, 26.

Druzdzel, Marek J. “Decision Analysis Software,” 5 September 1998

Systems Laboratory. URL: <http://www.lis.pitt.edu/~dsl/da-software.html>. A

February 1999.

Eales, Stewart Christopher. Playing with Intelligence: Officer App

Intelligence in the PURPLE SUNSET Wargame. MSSI Thesis. Washington, 

Military Intelligence College, August 1997.

Finley, James P. “Intelligence Failure Matrix.” Military Intelligence P

Bulletin 20, no. 1 (January-March 1994): 14.

________. “Nobody Likes to be Surprised: Intelligence Failures.” Military I

Professional Bulletin 20, no. 1 (January-March 1994): 15+.
39

Intelligence Estimating. New York: University Press of America, 1993.

Graves, David Lawrence, Captain, USAF. Bayesian Analysis Metho

Prediction. MSSI Thesis. Washington, DC: Defense Intelligence College, 1

Grunwald, Michael, and others. “CIA Halted Plot to Bomb U.S. Embass

Washington Post, 25 September 1998, A27+.

Hall, Wayne M., Lieutenant Colonel, USA. “Intelligence Analysis in the 

Military Intelligence Professional Bulletin 18, no. 1 (January-March 1992): 

Hammond, James D. So You Want to be an Intelligence Analyst? Re

Washington, DC: Defense Intelligence College, October 1983.

Hayes, Richard E. President of Evidence-Based Research, Inc. Intervi

9 February 1999.

Henry, Ryan, and C. Edward Peartree. “Military Theory and Informa

Parameters 28, no. 3 (Autumn 1998): 121-135.

Hilsman, Roger. “Does the CIA Still Have a Role?” Foreign Affa

(September/October 1995): 104-116.

Hulnick, Arthur S. “Managing Intelligence Analysis: Strategies for Pl

Game.” International Journal of Intelligence and Counterintelligence 2, no. 

321-342.

“Hypothesis Testing: A Technique for Structuring Analysis.” Outline, Le

Workbook. N.p., n.d. Provided on 15 March 1999 By Morgan D. Jones.

Inside CIA’s Private World: Declassified Articles from the Agency’s In

1955-1992. Ed. H. Bradford Westerfield. New Haven: Yale University Press

Jones, Morgan D. Former analyst for the Central Intelligence Agency a

Analytic Prowess, L.L.C. Interview by author, 15 March 1999.

________. The Thinker’s Toolkit: 14 Powerful Techniques for Problem 

York: Random House, Inc., 1998.

Kahaner, Larry. Competitive Intelligence: From Black Ops to Boardr

Businesses Gather, Analyze, and Use Information to Succeed in the Global 

New York: Simon & Schuster, 1996.

Kam, Ephraim. Surprise Attack: The Victim’s Perspective. Cambridge, 

University Press, 1988.

Keithly, David M. “Leading Intelligence in the 21st Century: Past 

Defense Intelligence Journal 7, no. 1 (Spring 1998): 78-88.
40

Princeton University Press, 1949.

Kifner, John. “Raids by U.S. Agents and Tirana Police Reportedly Thwar

Embassy.” New York Times, 21 August 1998, A13.

Landers, Daniel F. “The Defense Warning System.” Defense Intelligence 

no. 1 (Spring 1994): 21-32.

Loeb, Vernon and Walter Pincus. “New Spy Satellites at Risk Because 

Uncertain, Pentagon Told.” Washington Post, 12 November 1999, A7.

Manheim, Jarol B., and Richard C. Rich. Empirical Political Analysis

Methods in Political Science. New Jersey: Prentice-Hall, Inc., 1981.

Mann, Steven R. “Chaos Theory and Strategic Thought.” Parameters 

(Autumn 1992): 54-68.

Manthorpe, William H.J. Jr. “From the Editor.” Defense Intelligence Journ

(Spring 1998): 3-5.

McCarthy, Mary. “The National Warning System: Striving for an Elus

Defense Intelligence Journal 3, no. 1 (Spring 1994): 5-19.

Murray, Thomas H., Ph.D. Former analyst and trainer for the Central I

Agency and Senior Vice President of Sequoia Associates, Inc. Interview by 

January 1999.

Neustadt, Richard, and Ernest May. Thinking in Time. New York: The 

1986.

New York Times, 7 June 1998 and 21 August 1998.

Newing, Rod. “Consumer Information is the Fuel of a New Industrial R

Financial Times, 3 February 1999, B7.

Norusis, Marija J. SPSS 8.0 Guide to Data Analysis. Upper Saddle River, N

Hall, 1998.

Nye, Joseph S. Jr. “Peering into the Future.” Foreign Affairs 73, no. 4 (J

1994): 82-93.

Nye, Joseph S. Jr., and William A. Owens. “America’s Information Edg

Affairs 75, no. 2 (March/April 1996): 20-25.

“An Orderly Approach to Intelligence Analysis.” Instructor’s manual. 

Provided on 18 January 1999 by the Senior Vice President of Sequoia Assoc

Thomas H. Murray.
41

no. 1 (Spring 1998): 25-45.

Peterson, John L. “Forecasting: It’s Not Possible.” Defense Intelligen

no. 2 (Fall 1994): 37-45.

Pilzer, Paul Zane. Unlimited Wealth. New York: Crown Publishers, Inc., 

Porteous, Samuel D. “Looking out for Economic Interests: An Incre

Intelligence.” Washington Quarterly 19, no. 4 (Autumn 1996): 191-204.

Quantitative Approaches to Political Intelligence: The CIA Experience. E

Heuer Jr. Boulder: Westview Press, Inc., 1978.

Resch, David T., Captain, USA. “Predictive Analysis: The Gap Between 

Practitioners.” Military Intelligence Professional Bulletin 21, no. 2 (April-June 

Robbins, Carla Anne. “Intelligence Community is Criticized in Review

Journal, 3 June 1998, A8.

Schweizer, Peter. “Growth in Economic Espionage.” Foreign Affai

(January/February 1996): 9-14.

Schum, David A. Evidence and Inference for the Intelligence Analyst. V

II. Lanham, MD: University Press of America, 1987.

Shirley, Edward G. “CIA Needs Reform, Not New Missions.” Wall S

19 November 1998, A22.

Seib, Philip. “Intelligence Gathering Remains a Vital Function,” 1 March 

Morning News. URL: <http://www.dia.ic.gov/admin/EARLYBIRD/990302/ 

gathering.htm>. Accessed 2 March 1999.

Siegel, Sydney. Nonparametric Statistics For the Behavioral Science

McGraw-Hill Book Company, Inc., 1956.

Studeman, William O. “Leading Intelligence along the Byways of 

Acquiring C4ISR Architectures for the 21st Century.” Defense Intelligence 

1 (Spring 1998): 47-65.

Tellis, Ashley J., Thomas S. Szayna, and James A. Winnefeld. Antici

Conflict. Santa Monica, CA: The Rand Corporation, 1997.

Toffler, Alvin, and Heidi Toffler. War and Anti-War: Survival at the Da

Century. New York: Little, Brown and Company, 1993.

Tufte, Edward R. The Visual Display of Quantitative Information. C

Graphics Press, 1983.
42

Intelligence and Counterintelligence 5, no. 3 (Fall 1991): 275-284.

U.S. Commission on the Roles and Capabilities of the United States I

Community. Preparing for the 21st Century: An Appraisal of U.S. In

Washington, DC: GPO, 1 March 1996. URL: <http://www.access.gpo.gov/su_

epubs/int/int012.html>.

Warning analyst at the National Warning Staff. Telephone interview 

5 November 1998.

Weiner, Tim, and James Risen. “Decision to Strike Factory in Sudan 

Surmise: Inferred from Evidence.” New York Times, 21 September 1998, A1.

Wohlstetter, Roberta. Pearl Harbor: Warning and Decision. Stanford, CA

University Press, 1962. 
43

Master Sergeant Robert D. Folker, Jr. received the Master of Scienc

Intelligence degree from the Joint Military Intelligence College in 1999. H

Superintendent of the 325th Operations Support Squadron’s Intelligence Fli

Air Force Base, Florida. Since enlisting in the Air Force in 1986, M

assignments have included Non-Comissioned Officer in Charge (NCO

Intelligence Team at the Strategic Command’s Intelligence and Warning Cen

Force Base, Nebraska; NCOIC, Intelligence Element at the 80th Fighter Squ

Air Base, South Korea; and NCOIC, Mission Analysis Flight and Senior An

Contingency Airborne Reconnaissance Deployable Ground Station 2, Beale A

California. MSgt Folker may be contacted at: <robert.folker@tyndall.af.mil>
45

Occasional Papers

Unclassified papers are available through <www.ntis.gov>; selecte

available through the U.S. Government Printing Office <www.gpo.gov>.

1. Classified paper.

2. Getting Intelligence Right: The Power of Logical Procedure, Capt (U

S. Brei, 1996.

3. An Office Manager’s Guide to Intelligence Readiness, Russell G. Swe

4. Classified paper.

5. A Flourishing Craft: Teaching Intelligence Studies, Papers Prepared f

1999 JMIC Conference on Teaching Intelligence Studies at 

Universities, 1999.

6. Intelligence Essentials for Everyone, Lisa Krizan, 1999.

7. Intelligence Analysis in Theater Joint Intelligence Centers: An E

Applying Structured Methods, MSgt (USAF), Robert D. Folker, Jr., 2
Occasional Paper Number Seven

JOINT MILITARY

INT

E

LLIGENCE COLLEGE

1

9

6

2

DEFENSE INTELLIGENCE AGENCY

UNITED STATES OF AMERICA