United States General Accounting Office

GAO Testimony

Before the Select Committee on Intelligence, U.S. Senate

For release on Delivery

Expected at

9:30 a.m., EST

Wednesday,

December 4, 1996

FOREIGN MISSILE

THREATS

Analytic Soundness of

National Intelligence

Estimate 95-19

Statement of Richard Davis, Director, National Security

Analysis, National Security and International Affairs

Division

G O A

years

1921 - 1996

GAO/T-NSIAD-97-53

Mr. Chairman, Mr. Vice Chairman, and Members of the Committee:

I am pleased to be here today to discuss our evaluation of National

Intelligence Estimate (NIE) 95-19, Emerging Missile Threats to North

America During the Next 15 Years. The details of our evaluation are

contained in our August 1996 report.1

In summary, we found:

• The main judgment of NIE 95-19 — “No country, other than the major

declared nuclear powers, will develop or otherwise acquire a ballistic

missile in the next 15 years that could threaten the contiguous 48 states or

Canada.” — was worded with clear (100 percent) certainty. We believe this

level of certainty was overstated, based on the caveats and the intelligence

gaps noted in NIE 95-19.

• NIE 95-19 had additional analytic shortcomings. It did not (1) quantify the

certainty level of nearly all of its key judgments, (2) identify explicitly its

critical assumptions, and (3) develop less likely (but not impossible)

scenarios referred to as “alternative futures.”

However, NIE 95-19 did acknowledge dissenting views from several

agencies and also explicitly noted certain information the Intelligence

Community does not know that bears upon the foreign missile threat.

• NIE 95-19 worded its judgments on foreign missile threats very differently

than did two 1993 NIEs on related subjects that we reviewed,2

 even though

the judgments in all three NIEs were not inconsistent with each other. In

general, the two 1993 NIEs pointed out unfavorable and unlikely outcomes

associated with foreign missile threats to the United States more often

than did NIE 95-19. Finally, the evidence in NIE 95-19 is considerably less

than that presented in the two 1993 NIEs, in both quantitative and

qualitative terms.

Our evaluation did not include whether policymakers or intelligence

officials interfered with the NIE 95-19 process. Therefore, we have no views

1

Foreign Missile Threats: Analytic Soundness of Certain National Intelligence Estimates

(GAO/NSIAD-96-225, Aug. 30, 1996). We also prepared a classified version of our report. All of our

findings are contained in our unclassified report; the classified information concerned detailed

examples drawn from the NIEs to support our findings and observations.

2

The titles and content of these NIEs are classified.

Page 1 GAO/T-NSIAD-97-53
on this matter. Also, we did not attempt to independently evaluate foreign

missile threats to the United States.

Before describing in detail what we found, I want to mention that our

evaluation was significantly impaired by a lack of cooperation by several

Executive Branch agencies. The Departments of Defense and State would

not allow us to review their records on NIE 95-19 and instead referred us to

the Director of Central Intelligence (DCI). The DCI declined to cooperate

with our review. His office maintained that our review would be contrary

to oversight arrangements for intelligence that the Congress has

established. Therefore, we were unable to obtain the DCI’s official

standards (if any exist) for the essential elements of an objective NIE,

review supporting documentation on NIE 95-19, or discuss the NIE with

cognizant officials from the National Intelligence Council (NIC) and other

agencies.

NIE 95-19 Overstated

Certainty of Its Main

Judgment

The main judgment of NIE 95-19 was worded with clear (100 percent)

certainty. We believe this level of certainty was overstated, based on the

caveats and the intelligence gaps noted in NIE 95-19.

On the issue of certainty, in 1992 then-DCI Robert Gates opined: “While we

strive for sharp and focused judgments for a clear assessment of

likelihood, we must not dismiss alternatives or exaggerate our certainty

under the guise of making the ’tough calls.’ We are analysts, not umpires,

and the game does not depend on our providing a single judgment.”

The caveats and intelligence gaps noted in NIE 95-19 do not support the

100-percent certainty level of its main judgment. For example, at the

beginning of NIE 95-19, the estimate states “as with all projections of

long-term developments, there are substantial uncertainties.” Also, NIE

95-19’s Intelligence Gaps section noted several shortcomings in the

Intelligence Community’s collection of information on foreign plans and

capabilities.

Page 2 GAO/T-NSIAD-97-53
NIE 95-19 Had

Additional Analytic

Shortcomings

We believe that at least five standards apply to an objective NIE. These

standards were synthesized from our review of the published views of nine

current and former senior intelligence officials, the reports of three

independent commissions, and a Central Intelligence Agency (CIA)

publication that addressed the issue of national intelligence estimating.3

I would like to briefly discuss each of the five standards and how we

assessed NIE 95-19 against the standards.

(1) An NIE should quantify the certainty level of its key judgments by using

percentages or “bettors’ odds,”4 where feasible, and avoid overstating the

certainty of judgments. NIE 95-19 did not quantify the certainty levels of

any of its key judgments, except for the 100-percent certainty implied by

its main judgment, previously discussed. It used unquantified words or

phrases such as “probably,” “sometimes,” and “feasible, but unlikely.”

The CIA has told its analysts to be precise in conveying the levels of

confidence they have in their conclusions because policymakers and

others rely on these assessments as they define and defend U.S. interests.

Different people can hear very different messages from the same words,

especially about probabilities, and therefore good estimates should use

quantitative measures of confidence, according to a former NIC Vice

Chairman. For example, a “small but significant” chance could mean one

chance in a hundred to one person; for another it may mean one chance in

five. Similarly, a former NIC Chairman wrote that NIEs with only words such

as “possibly” are not of much help to someone trying to make an important

decision. However, we recognize that some intelligence judgments may

not easily lend themselves to specifying a meaningful level of confidence,

using numbers.

(2) An NIE should identify explicitly its assumptions and judgments. NIE

95-19 did not explicitly identify its critical assumptions either by separately

listing them in one place or by introducing them throughout the text with

wording such as “we have assumed . . .”

3

Our sources included the published views of Robert M. Gates, former DCI and Deputy Director for

Intelligence, CIA; Joseph S. Nye, Jr., former Chairman, NIC; Harold P. Ford, former Acting Chairman,

NIC; Gregory F. Treverton, former Vice Chairman, NIC; reports by the Vice President’s National

Performance Review, the Commission on the Roles and Capabilities of the United States Intelligence

Community, and a study group on intelligence sponsored by the Council on Foreign Relations; and 

A Compendium of Analytic Tradecraft Notes, vol. I, March 1996, published by the CIA’s Product

Evaluation Staff, Directorate of Intelligence.

4

Bettors’ odds state the chance as, for example, “one out of three.”

Page 3 GAO/T-NSIAD-97-53
Critical assumptions, also known as “linchpin assumptions,” are defined

by the CIA as analysts’ debatable premises that hold the argument together

and warrant the validity of judgments. Estimative judgments are to be

defended by fully laying out the evidence and carefully explaining the

analytic logic used, according to a former Deputy Director for Intelligence,

CIA. Writing about NIEs, a former Vice Chairman of the NIC agreed. As a

general rule, the more complex and controversial an issue, the more

analytic effort is required to ensure that critical assumptions are precisely

stated and well defended, according to the CIA. A good analysis will clearly

identify its key assumptions so that policymakers are aware of the

“foundations” of the estimate and can therefore judge for themselves the

appropriateness of the assumptions and the desirability of initiating

actions to hedge against a failure of one or more assumptions.

(3) An NIE should develop and explore “alternative futures:” less likely (but

not impossible) scenarios that would dramatically change the estimate if

they occurred. NIE 95-19 did not develop alternative futures.

NIEs should “describe the range of possible outcomes, including relatively

unlikely ones that could have major impact on American interests, and

indicate which outcomes they think are most likely and why . . . The job,

after all, is not so much to predict the future as to help policymakers think

about the future,” according to a former NIC Chairman. The CIA, then-DCI

Robert Gates, and other senior NIC officials agree that NIEs should analyze

alternative futures.

(4) An NIE should allow dissenting views on predictions or interpretations.

NIE 95-19 had 12 dissents, all in the body of the estimate — most dealt with

technical issues.

According to a February 1996 statement by the current Chairman of the

NIC, “The process for producing NIEs is directed particularly at ensuring

presentation of all viewpoints. We do not impose consensus; in fact we

encourage the many agencies that participate in NIEs to state their views

and we display major differences of view in the main text. Lesser

reservations are expressed in footnotes.”

(5) An NIE should note explicitly what the Intelligence Community does

not know when the information gaps could have significant consequences

for the issues under consideration. NIE 95-19 noted information gaps at

places in the estimate’s text and in a separate Intelligence Gaps section.

This disclosure not only helps alert policymakers to the limits of the

Page 4 GAO/T-NSIAD-97-53
estimate but also informs intelligence collectors of needs for further

information, according to a former NIC Chairman.

Differences and

Similarities Between

NIE 95-19 and 1993

NIEs

NIE 95-19 worded its judgments on foreign missile threats very differently

than did the two 1993 NIEs we reviewed, even though the judgments in all

three NIEs were not inconsistent with each other. That is, while the

judgments were not synonymous, upon careful reading they did not

contradict each other.

In general, the two 1993 NIEs pointed out unfavorable and unlikely

outcomes associated with foreign missile threats to the United States more

often than did NIE 95-19. In our view, this could lead to a greater level of

concern about missile threats to the United States.

Finally, the evidence in NIE 95-19 is considerably less than that presented

in the two 1993 NIEs, in both quantitative and qualitative terms. Laying out

the evidence is important because it allows readers to judge for

themselves how much credence to give the judgments, according to a

former Vice Chairman of the NIC. In quantitative terms, each of the 1993

NIEs was over three times as long as NIE 95-19. In qualitative terms, we

believe the 1993 NIEs provided more convincing support for their key

judgments than did NIE 95-19.

Mr. Chairman, that concludes my prepared remarks. At this time, I would

be happy to answer any questions you, the Vice Chairman, and the

Members of the Committee may have.

(701107) Page 5 GAO/T-NSIAD-97-53
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100