CENTER FOR THE STUDY OF

NATIONAL RECONNAISSANCE

NOVEMBER 2015
A Methodology for 

Identifying Lessons Learned

An Integr a t i o n o f 

C a se S tudy T e c h n i ques w i t H G ener a l 

S ystems P a r a d i gm & Congruen c e T heory 1

Robert A. McDonald, Ph.D.

1 The information in this publication may not necessarily reflect the official views of the National 

Reconnaissance Office, the Intelligence Community, the Department of Defense, or any other United States 

Government entity.

Center for the S t u d y o f 

Na t i o n a l R e c onn ai s s a n c e

N ovember 2015

C SNR L essons le a rned ser i e s
Center for the Study of National Reconnaissance

The Center for the Study of National Reconnaissance (CSNR) is an independent 

National Reconnaissance Office (NRO) research body reporting to the Director/Business 

Plans and Operations Directorate, NRO. The CSNR’s primary objective is to advance national 

reconnaissance and make available to NRO leadership the analytic framework and historical 

context to make effective policy and programmatic decisions. The CSNR accomplishes its 

mission by promoting the study, dialogue, and understanding of the discipline, practice, and 

history of national reconnaissance. The CSNR studies the past, analyzes the present, and 

searches for lessons for the future.

Contact Information: Phone, 703-227-9368; or e-mail, csnr@nro.mil

To Obtain Copies: Government personnel may obtain additional printed copies directly 

from CSNR. Other requestors may purchase printed copies by contacting the Government 

Printing Office. Selected CSNR publications are available on the Internet at the NRO web site.

Disclaimer: The views expressed in the monograph are those of the author. The information 

and opinions expressed in this monograph should not be attributed to the Central Intelligence 

Agency, National Reconnaissance Office, or other government departments or agencies. 

Copyright Information: This work, where indicated, is subject to copyright 

restriction, and may not be copied or distributed except as follows: This work may be 

copied and distributed by agencies of the U.S. Intelligence Community for official U.S. 

Government purposes. 

Suggested Cataloging Data

McDonald, Robert A. A Methodology for Identifying Lessons Learned—An Integration of Case Study 

Techniques with General Systems Paradigm and Congruence Theory/Robert A. McDonald. - First Edition

vii, 85p.,28cm

Includes bibliographical references

1. Case study framework for lessons learned. 2. The general systems paradigm. 3. Congruence theory. 4. 

Integrating the general systems paradigm and congruence theory into the case study framework. 5. Inherent 

flaws in lessons-learned research and methodologies.

ISBN 978-1-937219-14-7 (pkb.) 2015

For sale by the Superintendent of Documents, U.S. Government Printing Office

Internet: bookstore.gpo.gov Phone: toll free (866) 512-1800 DC area (202) 512-1800

Fax (202) 512-2104 Mail: Stop IDCC Washington, DC. 20302-0001

“In every field of inquiry, an adequate paradigm reveals patterns of coherent 

relations in what are otherwise inexplicable random coincidences”

Richard Tarnas

Cosmos and Psyche: Intimations 

of a New World View, 2006
Table of Contents

Foreword ..................................................................................................................................... v

Preface ......................................................................................................................................... vii

Introduction ............................................................................................................................... 1

Case Study Framework for Lessons Learned ........................................................................... 3

Case Study Compared With Case History ............................................................... 4

Alternative Case Study Models .................................................................................. 5

Applicability of Case Study Methodology to IC Lessons-Learned Analysis .......... 5

The General Systems Paradigm—A Model for Lessons-Learned Data Collection ............... 7

Flaws in Traditional Approaches to Collect Organizational Data .......................... 8

The Advantage of Using the General Systems Paradigm Approach ....................... 9

Conceptualizing the Collection of Systems Data .................................................. 10

Applying General Systems Theory for Data Collection .......................................... 12

Congruence Theory—A Technique for Lessons-Learned Analysis ...................................... 16

Assessing Congruence of the Domains in the Process Component of the System .. 17

Questions for Assessing Congruence ........................................................................ 18

Considering Congruence with Other Components ................................................. 20

Labeling the Dyadic Relationships .......................................................................... 21

Integrating the General Systems Paradigm and 

 Congruence Theory into The Case Study Framework .......................................................... 23

Phase I—Developing the Research Design ................................................................. 25

Phase II—Collecting Data and Preparing the Case Description ............................ 27

Phase III—Conducting the Lessons-Learned Analyses .......................................... 30

Inherent Flaws in Lessons-Learned Research and its Methodologies ..................................... 33

Data Collection Errors ............................................................................................... 33

Avoiding Data Collection Errors ............................................................................... 36

Cognitive Errors ......................................................................................................... 36

Causes of Cognitive Error .......................................................................................... 39

Avoiding Cognitive Error .......................................................................................... 40

Conclusion .................................................................................................................................. 42

References ................................................................................................................................... 44

Appendix A—Research Design Template (With Questions to Consider) ............................ 48

Appendix B—Data Collection Framework and Examples of Questions .............................. 50

Appendix C—Interview Protocol ............................................................................................. 66

Appendix D—Memory: The Uncertanity of Recalling the Past............................................ 75

Appendix E—Outline and Checklist Questions for Case Description ................................. 85

Appendix F—Outline for Lessons-Learned Case Assessment ............................................... 97

Index ........................................................................................................................................... 98

v

Foreword

Learning is critical to individual development and survival. As children and adults we all 

have had to learn immense amounts of information and skills to function and survive in the 

world. Organizations, as with individuals, also need to learn if they are to operate effectively 

and sustain superior performance and avoid failures. 

When organizations overlook or neglect to learn lessons, they fail. Blockbuster, founded in 

1985, dominated home movie rentals in the late 1980s by distributing VHS video recordings to 

customers who visited their extensive network of stores. While Blockbuster was operating under 

that model, a new technology, digital video discs (DVDs) appeared, and a new competitor, 

Netflix, initiated a novel “rentals-by-mail” DVD marketing strategy, which proved to be very 

successful. Blockbuster then turned to a “rentals-by-mail” strategy, and at the same time tried 

to create and to hold onto a mixed model with its stores, but Netflix had moved ahead to 

digital delivery of streaming videos over the Internet. Twice Netflix leapfrogged Blockbuster, a 

company, apparently, that had not learned lessons about marketing and innovation. Blockbuster 

filed for bankruptcy in 2010.1

 In another case, the iconic company, Kodak, which had a name 

that was synonymous with film and photography during most of the 20th century, did not seem 

to see a lesson when photographers turned to digital imagery; the company filed for bankruptcy 

in 20122 (Blockbuster, n.d.; Kodak, n.d.; Newman, R., 2010; Sinek, S., 2010; History, 2013).

This monograph describes a social science methodology that an organization’s leadership 

can use to identify and learn lessons to maintain and improve the organization’s performance—

to identify those behaviors that either should be encouraged or corrected. By using this kind of 

social science methodology, the organization’s leadership can apply the research-based lessons 

to the organization’s processes so as to ensure continued successes and the avoidance of failures.

To some, the findings from this kind of social science research may appear to be merely 

formulating “statements of the obvious” or reporting “knowledge of the obvious.” This kind 

of reaction is not unexpected, as it seems to be a ubiquitous criticism of social science research 

where laypersons typically describe social science findings as nothing more than the obvious—a 

psychological phenomenon best described as hindsight bias (i.e., the “I knew it all along” effect). 

However, the reality is that these kinds of findings typically are not obvious prior to completing 

the research. It only is after observers learn of the research findings that they come to label the 

findings as self-evident facts. They do not accurately anticipate the correct findings prior to 

the research. The popular wisdom that generally is accepted as fact, most often turns out to 

be contrary to actual social science research findings (Gage, 1991; Kelley, 1992; Myers, 1994; 

Stanovich, 2009; Lilienfeld, 2012).3

But it is assessing—with a social science methodology—the experiences in our most recent 

and more distant histories where, through the structured social science research, we can identify 

1 In September 2010, Blockbuster, Inc. filed a voluntary petition under Chapter 11 of Title 11 of the United 

States Code in the United States Bankruptcy Court for the Southern District of New York (Blockbuster, n. d.).

2 On January 19, 2012, Eastman Kodak Company and its U.S. subsidiaries filed voluntary petitions for Chapter 11 

business reorganization in the U.S. Bankruptcy Court for the Southern District of New York (Koday, n.d.).

3 Social science research has produced evidence that “popular wisdom” and “folk knowledge” such as the 

follow examples are inconsistent with research findings: opposites attract; familiarity breeds contempt; 

expressing pent-up anger reduces anger; on multiple-choice tests, the test taker should stick with the original 

answer (Lilienfeld, 2012, p. 114).
vi

useful lessons, not only for our present, but also for our future. The most recent histories may 

be only days or hours ago; the more distant histories may be years or even decades ago. As 

you look back to make your assessments and learn lessons from your organization’s histories, 

I urge you to do this with a structured, social-science-based methodology, such as I have 

offered in this monograph. I believe that only in this way will you be able to find valid lessons, 

not folklore and legends. The successful assessment of any historical experience can only be 

discovered within a larger hermeneutic context—a context for the study and interpretation of 

human behavior and social institutions such as encapsulated by the general systems paradigm 

and congruence theory (Heywood & Stronach, 2005).

“A historical analysis always has to be performed with its own hermeneutics 

from the period, not to justify the events, but rather to understand them. It is 

essential to analyze history in the cultural context of the moment in which the 

events occurred....Another important issue is to analyze historical processes in 

their entirety and not remain with a fragmented interpretation because that 

fragment then becomes universalized and takes the place of the entirety, and 

it becomes a legend” (Bergoglio & Skorka, 2013).

I suggest that the methodology I have outlined in this monograph will aide you in conducting 

an analysis of your experiences within the full context–historical, cultural, and operational—

within which your organization resides. And out of that assessment you will identify lessons that 

are based on solid research and will prove useful for the success of your organization.

 Robert A. McDonald, Ph.D.

 Director, Center for the Study of National Reconnaissance
vii

Preface to An Integration of Case Study Techniques 

with General Systems Paradigm & Congruence Theory4

When I had the privilege to serve as a Professor of National Security Policy and Psychology 

at the National War College, I found that both I and the students learned some of the most 

meaningful lessons through our analyses of case histories that had documented the experiences 

of national security leaders in dealing with complex problems. During seminar we would review 

the case descriptions, formulate hypotheses that could explain what happened, hypothesize 

alternative courses of action, and challenge those hypotheses with our developing insights. 

I believe both the analytical experiences and the lessons that the students learned from their 

case analyses proved useful to them in their future assignments as flag officers in the military 

and senior executives in the national security community.

One of the greatest national security challenges during the early years of the 21st century 

was the terrorist attacks of 11 September 2001. After that event the various intelligence oversight 

bodies highlighted the importance of the Intelligence Community (IC) identifying lessons from 

retrospective analyses of past experiences. When the Weapons of Mass Destruction (WMD) 

Commission highlighted the importance of the IC learning lessons, it brought to mind the War 

College students’ search for lessons in their case studies. I wondered what was the methodology 

behind those successful learning experiences, and how might they be applied to the broader 

challenge of the WMD Commission for IC organizations to learn lessons? 

Several years after completing my assignment at the War College, I had the opportunity 

to be a student in a senior executive course at the Wharton School and was reintroduced to the 

application of systems theory to the study of organizations. It was reminiscent of my earlier 

exposure to general systems theory and congruence theory during my years as a doctoral student. 

After the Wharton course, I concluded that these concepts could be useful in identifying lessons 

for any organization, whether it be in the private sector or the government. The challenge was 

to integrate these various social science approaches for analyzing organizational behavior into 

a structured methodology that could offer insight into identifying lessons that could aid the 

organization in sustaining and improving the quality and output of its mission operations. 

It is difficult to find reflective time to integrate these kinds of concepts and to formulate 

a methodology that could be applied to social science research. However, I did find that 

opportunity during my periodic visits to Maryland’s Eastern Shore on the Delmarva Peninsula. 

I used that opportunity to take on the challenge to reflect on how case studies, systems theory, 

and congruence theory could be integrated into a structured methodology to identify lessons, 

with specific attention to IC organizations. There could be many ways to do this. I offer you the 

approach in this monograph.

Robert A. McDonald

The Eastern Shore

4 Notice: The monograph, A Methodology for Identifying Lessons Learned—An Integration of Case Study 

Techniques with General Systems Paradigm & Congruence Theory is copyrighted, and is used with permission in 

this government publication. © 2013, Robert A. McDonald, all rights reserved. The views expressed in this 

monograph are the personal views of the author.

1

Introduction

On Sunday morning, December 7, 1941 . . . over 300 Japanese planes . . . 

attacked . . . Pearl Harbor in the Territory of Hawaii. . . . We come today, over 

4 years after the event, . . . to record for posterity the facts of the disaster. In another 

sense we seek to find lessons to avoid pitfalls in the future to evolve constructive 

suggestions for the protection of our national security.

(U.S. Congress, 1946)

September 11, 2001, was a day of unprecedented shock and suffering in the 

history of the United States. . . . The law directed us to investigate “ facts and 

circumstances relating to the terrorist attacks” . . . Our aim has not been to 

assign individual blame. Our aim has been to provide the fullest possible account 

of the events surrounding 9/11 and to identify lessons learned.

(Commission on Terrorist Attacks, 2004)

The Japanese militarists surprised Americans on December 7, 1941 by attacking Pearl 

Harbor. Sixty years later the Al-Qa’ida terrorists surprised Americans on September 11, 2001 by 

attacking New York and Washington. Subsequent to the 1941 and 2001 attacks, U.S. officials 

conducted inquiries to discover facts about the attacks and to identify lessons to be learned, 

especially with regard to intelligence activities. After 11 September 2001 there was increased 

pressure on the Intelligence Community (IC) to study its past experiences in order to learn 

lessons for the future. The Weapons of Mass Destruction (WMD) Commission (2005) cited, as 

one of its most important recommendations, that the Director of National Intelligence (DNI) 

“create mechanisms to ensure that the Intelligence Community conduct ‘lessons learned’ 

that will better equip it...to identify its own strengths and weaknesses.” In this monograph 

I have suggested a research approach to help achieve that goal—a systems-oriented research 

methodology for identifying lessons learned.1

I developed this methodology out of the case study framework that long has been part of social 

science research. To guide data collection I turned to the paradigm of general systems theory. To 

guide analysis of the data I turned to congruence theory. The case study framework, along with 

these two ways of thinking—systems theory and congruence theory—can be integrated into 

a disciplined social-science-based research methodology. I believe such a methodology is ideal 

for analyzing the activities of an organization with the goal of identifying lessons to be learned 

and applied to future actions, and I believe the methodology is applicable to the analysis of any 

organization, whether it be a national security organization, such as an Intelligence Community 

agency or a military command, or any other organization, such as a law enforcement agency, an 

educational institution, a financial institution, or a business operation.

This disciplined research methodology is an organizationally focused diagnostic activity that 

examines organizational transactions within a near-current historical context. The focus is to 

gain insight into how to influence positive future organizational behavior. The objective is not to 

1 There are other more traditional ways to search for lessons, and these may very well prove helpful. However, 

I will not describe them in this monograph.
2

document a historical record of organizational behavior, not to write an interesting anecdote about 

organizational experiences, not to develop recommendations for future organizational behavior, 

and not to seek how to conduct a past practice better if there were a second opportunity. Rather, its 

objective is to analyze a recent past and to gain fundamental insight from that past experience—

insight that will serve as a basis for members of the organization to make informed decisions about 

future individual and organizational behavior. 

In doing this, the researcher must address those experiences that are both “satisfying” as well 

as “frustrating.”2

 The challenge in identifying lessons is to uncover in past experiences the insights 

that can be generalized as constructive principles that suggest options to form productive future 

behaviors. The goal is to gain insight from the past experiences in order to be able to shape future 

behavior in a positive way. The universe of historical experiences is broad and includes experiences 

that fulfill organizational objectives, as well as experiences that leave gaps in meeting objectives. In 

this methodology, we are interested in studying that full range.

I believe that many of the traditional methods that government agencies, such as in the national 

security and law enforcement communities, have used to identify lessons may be incomplete or 

flawed for long-term insight. They 

typically identify findings that 

are useful for specific situations, 

but they fail to offer fundamental 

insight that is applicable to a broad 

range of situations. In many cases 

the traditional methods also lack 

the rigor of an objective, disciplined 

approach that is founded in the 

social sciences and necessary for 

broad generalization. The law 

enforcement and national security 

communities do not have the 

luxury to rely solely on tradition, 

ad hoc methods to identify lessons 

that have narrow applications. The 

complexity of 21st century threats, 

and the speed with which those 

threats can metastasize, argue for 

more integrated and social-sciencebased research to identify lessons. 

2 I selected the terminology, “frustrating experiences,” and “satisfying experiences” to avoid the use of 

the terms, “failure” and “success.” Using the term “failure” encourages risk-adverse behavior within the 

organization; it invites blame-assigning behavior from outside the organization. There are no failures—only 

insights that offer opportunities to learn. When an individual or organization falls short of its goal, it is the 

frustration from that experience that generates the psychic energy to learn. There never can be total success—

only imperfect human behavior that provides opportunities to move toward total success. When there is a 

degree of success, it is the positive reinforcement from that satisfying experience that generates the psychic 

energy to move forward and learn more. 

Roadmap for this Monograph

The first three sections that follow this introductory section (i. e., the sections on 

case study, general systems paradigm, and congruence theory) will introduce you 

to the case-study-based framework and explain how it can integrate the general 

systems paradigm and congruence theory into a methodology for the collection 

and analysis of data in support of lessons-lessons research.

• The case study serves as a framework for the qualitative documentation 

and assessment of the facts surrounding a particular experience or event. 

• The general systems paradigm enables the researcher to view the 

organization and its processes as a complex set of integrated and 

interacting components and ensures that the researcher collects data on 

all the components that affect the functioning of the organization.

• Congruence theory provides the researcher with a framework to analyze 

the data by assessing how effectively the various components of the 

system fit together, which ultimately affects the performance and life of 

the organization. 

The final three sections (i. e., the sections on integrating the methodology’s 

components, identifying flaws, and offering concluding comments) outline the 

phases in this integrated research methodology, point out some potential flaws 

in collecting data and conducting analyses, and offer brief insight into applying 

the lessons. In this way the methodology should help identify ways that could 

enable an organization to sustain those satisfying experiences that bring about 

positive results and avoid those frustrating experiences that block attaining the 

organization’s objectives and goals.

The five appendices at the end of the monograph suggest a number of tools that 

could be useful in conducting the research.
3

Case Study Framework for Lessons Learned

Il était une fois . . . 

“Once upon a time there lived in a certain village a little country girl, the 

prettiest creature who was ever seen . . .”

—From Little Red Riding Hood by Charles Perrault

Vieną kartą . . .

“Once upon a time there lived an old man and an old woman. Every morning 

they went out to clear a nearby . . .”

—From The Swan Queen Lithuanian folk tale

Fadó, fadó, fadó a bhí ann. . . 

“A very long time ago, there suddenly appeared in old Ireland two unknown 

merchants of whom nobody had ever heard . . .”

—From The Countess Kathleen O’Shea, Irish folklore

C’era una volta . . .

“Once there was a very rich man who had a son who could not tell a bean-pod 

from a cucumber . . .”

—From Good Helpmates, Italian folk tale

“Once upon a time” is the traditional way to begin a story in many cultures and languages. 

And that is what a case study is—a story about an experience. And like the stories about 

folktales, which often have a moral or lessons, the story in a case study provides a basis for 

identifying a principle that can become a lesson.

The social science community generally accepts the case study as “the bedrock of scientific 

investigation.” Bromely (1986) reported its use “. . . in areas disparate as: administration, 

anatomy, anthropology, artificial intelligence, biochemistry, business studies, clinical 

medicine, counseling, criminology, education, gerontology, history, industrial relations, 

jurisprudence, management, military studies, personality, politics, psychiatry, social work, 

and sociology” (Bromely, 1986, p. ix). 

The term “case-study” has a different meaning for each of these disciplines: For example, 

in psychology it might relate to the study of a segment of a life-story where the psychologist 

is seeking to understand the personality of an individual; in social work it might mean the 
4

study of relationships where the social worker is seeking to understand a family in distress; for 

business it might refer to the study of organizational performance where a consultant might 

be seeking to explain the success or failure in the output of the organization (Bromely, 1986).

For our approach in lessons-learned research, the case study methodology is the detailed and 

disciplined examination of past behaviors, performance, or events with the purpose of developing 

explanations or hypotheses that might be generalizable to other experiences, either in the past 

or in the future. The case-study methodology is disciplined in the sense that it is scientifically 

based research. In this way it can offer scientifically founded, broad-based explanations about 

why organizations perform as they do. It can provide a systematic body of knowledge that 

leadership can draw on as a means to improve marginal organizational performance and to 

sustain high organizational performance (Bromely, 1986; George & Bennett, 2005).

Figure 1: Lessons-Learned Framework (Adapted from Bromely, 1986, p. 80)

Case Study Compared With Case History

The case study, as I will describe it, is different from the case history. The case study for 

lessons-learned analysis focuses on a particular issue or set of issues over the period of time that 

is relevant to the issue or issues of interest. It documents the individual’s or organization’s 

behavior relevant to those issues under study. It is not a comprehensive history of the issue 

and its origin. The case history, by contrast, typically is an extended narrative account of the 

formative and cumulative events that define a set of events and associated behaviors in the past. 

While the case history focuses on the past, the case study is not limited to a particular historical 
5

context; it actually considers all three aspects of time—the past, the present, and the future: the 

past for insight; the present for reality; and the future for expectations (Bromely, 1986, p. 79). 

The case study methodology collects data about historical performance, which is in its 

recent history, but it analyzes the data within a current context to anchor it in reality. Figure 

1 illustrates how this case-study methodology is centered on present realities, yet draws on 

historical experiences while looking toward possible futures. The focus of lessons-learned 

research is within the current reality as depicted by the oval in the center of the figure. 

The central oval overlaps into a near point within the left oval that represents the historical 

experience of the organizations. The event of the case study is depicted by the indicated dot, 

which is in the more recent history that overlaps current reality. The results of the lessonslearned analysis look toward the right into the future where the lessons can be experimented 

with as the organization evolves into what it will become. This conceptual model can help the 

researcher focus on bringing the past into the present; and project an analysis of that current 

reality into what might be by focusing on possible futures (Bromely, 1986, p. 80).

Alternative Case Study Models

There are a variety of endeavors that analysts might consider case studies, but many of 

those endeavors typically do not meet the standards of a disciplined, scientific-based case study. 

The description of an anecdotal experience that illustrates a particular point is not this kind 

of case study; a hypothetical story that is part of a narrative is not this kind of case study; 

an account that documents an ad hoc inquiry is not this kind of case study. These are sagas, 

fictional accounts, and reports, not examples of disciplined case studies based on social science 

methodology. The methodology that I refer to for a case study must comply with the disciplined 

approach of scientific inquiry. It must independently specify a research question, systematically 

collect and record data, logically analyze the data, and objectively form conclusions (Bromely, 

1986, p.268).

Applicability of Case Study Methodology to Lessons-Learned Analysis

This case-study methodology is ideal for observing and studying the nature and 

performance of organizations, especially those involved in high-tempo/high-risk operations, 

such as national security, law enforcement, and medical functions. In much of the remainder 

of this monograph, I will use the Intelligence Community, as one kind of national security 

organization, that can serve as an example of an organization with high-tempo/high-risk 

functions that can most benefit from this kind of analysis. The methodology is flexible enough 

to document and organize what might be important in a wide-range of particular issues under 

study. It is structured enough to offer insight into the fundamental nature of the organization 

and its members, and it is objective enough to suggest lessons that can be applied universally 

across organizations for future insight. Out of the analysis, the case study can offer ways to 

understand the dynamics of the case experience and identify lessons for the future, both lessons 
6

for continued positive organizational performance and lessons for modified corrective action. 

The case study is particularly helpful because it can provide useful information about the 

experience and provide a model or example based on the lessons from the experience (Bromely, 

1986, pp. ix-xi, 72-73, 268).

The case study begins with a concise research question that gives rise to the inquiry. The 

research question can be open ended and ask about an event or series of events (e.g., why did 

intelligence fail to warn of the Japanese attack at Pearl Harbor at the start of World War II?), or 

it can offer a hypothesis about a particular phenomenon (e.g., it was the strength of teamwork 

between the CIA and Air Force that resulted in the quick development and fielding of first the 

U-2 reconnaissance aircraft and later the Corona imaging reconnaissance satellite during the 

early days of the Cold War) (Bromely, 1986, p. 72). 

After the investigator defines the research question, there are two fundamental steps that 

are essential to case-study research: (1) data collection and (2) data analysis. As I indicated 

earlier, for our adaptation of case-study methodology to lessons-learned research, I used 

the general systems paradigm as the framework for data collection and congruence theory 

for data analysis. In the next section I will discuss the application of the general systems 

paradigm to data collection.
7

The General Systems Paradigm—A Model for LessonsLearned Data Collection

“Beyond Ghor there was a city . . . The populace became anxious to see the 

elephant, and some sightless from among this blind community ran . . . to find it. . 

. . [T]hey groped sightlessly, gathering information by touching some part of it. . . . 

The man whose hand had reached an ear . . . said: ‘It is a large, rough thing, wide 

and broad, like a rug.’ . . . [T]he one who had felt the trunk said: ‘ . . . It is like a 

straight and hollow pipe, awful and destructive.’ The one who had felt its feet and 

legs said: ‘It is mighty and firm, like a pillar.’ Each had felt one part out of many. 

Each had perceived it wrongly. . . . All imagined something, something incorrect.”

—From “The Blind Ones and the Matter of the Elephant”

The classic Sufi story about the blind villagers and their study of the elephant is an example 

of why general systems theory is an appropriate model for collecting lessons-learned data.3

 In this story the blind villagers examined the elephant, not as a complete, integrated, living 

system, but as individual parts of an entity. Because of this incomplete—and what turned out 

to be a failed—approach to data collection, the villagers’ subsequent analyses were flawed. 

Not one of them correctly could identify the entity as a complex living organism or elephant, 

nor were they in a position to make meaningful behavioral inferences about this as an entity. 

If they had used a systems theory approach to conceptualize the entity as a system that was a 

living organism with integrated subsystems, they could have avoided that fundamental error.

General systems theory is particularly valuable in studying social organizations because 

it offers a framework that is ideal for treating large complex organizations and uses a common 

vocabulary that can unify several social science disciplines associated with the study of both 

individual and organizational behaviors. It allows the researcher to understand the whole 

system rather than being limited to explaining the interaction of one or two of the system’s 

aspects (Buckley, 1967, p. 39; Levinson, Molinari, & Spohn, 1972, p. 27). 

General systems theory is a comprehensive approach for describing and gaining an 

understanding of the full range of phenomena associated with a particular entity. Van DeVen 

and Ferry (1980, p. 1) reported that there is general agreement among management theorists 

and practitioners that the design of organizations—i.e., the environments and structures 

with which they operate and the interactions within and outside those structures—make a 

difference in the organization’s performance. Those interactions determine the organization’s 

outcomes and are essential for the researcher to understand if the researcher is to be able to 

identify lessons that will effectively and strategically improve the organization’s performance 

(Katz & Kahn, 1966, p. 28, 453). 

3 A “Sufi” is a Muslim mystic whose teachings, Sufism, address the inner, mystical, psycho-spiritual dimension 

of Islam (Godlas, 2008).
8

It provides the researcher with both a conceptual language and a structural framework 

to observe the system’s performance and collect data about it. With this systems framework, 

the researcher can come to understand the interactive nature of the whole system rather than 

merely understanding something about a few aspects of the system. Only if researchers apply 

this kind of systems framework, which considers all parts of the organization, will they be able 

to see and understand an organization as it truly is and make meaningful inferences about the 

organization’s future behavior (Katz & Kahn, 1966; Levinson, et al., 1972; Sutherland 1973).

The systems approach, when it is applied to any field—biology, physics, or organizational 

assessments—facilitates several aspects of data collection and analysis: (a) identifying the 

interactive components of the organization; (b) finding relationships in the organizational 

system; (c) uncovering patterns of interaction for the organization; and (d) mapping the 

cycles of input, transformation, and output of the organization. By collecting organizational 

data in this way, general systems theory provides a holistic and integrated orientation for 

the collection and analysis of complex organizational data, such as what occurs in national 

security, law enforcement, and medical organizations. It is ideal for collecting data about, 

and assessing the performance of these kinds of and other high-performance/high-tempo 

organizations (Buckley, 1967, p. 39).

Flaws in Traditional Approaches to Collect Organizational Data

Traditional approaches for the study of organizations and their performance tend to be 

either narrowly narrative or mostly metric in their focus. Typically they study only a part of 

the organization or its processes; they either collect descriptive data about a component of the 

organization, or they focus on measuring selected aspects of the organization’s performance. 

Traditional Descriptive Approach. The collection of descriptive data focuses on examining 

components of the organization, discussing their tasks, and characterizing their bureaucratic 

structure. This kind of approach typically answers questions about the structure of the organizations 

(i.e., “What are the roles required to do the organizational tasks?”), questions about the nature of 

the task (i.e., “What are the components of the task?”), questions about the bureaucratic structure 

(i.e., “What are the organizational units involved in doing the task?”), or questions about the 

informal organization (i.e., “Who talks with whom under what circumstances?”). These kinds 

of questions lead to descriptive answers, which may be useful for some inquires, but they are 

flawed for uncovering lessons that can be used to improve future organizational behavior. The 

descriptive approach does not offer information about relationships that are essential to gain 

insight for improving organizational behavior (Levinson, et al., 1972, p. 5).

The descriptive approach is imperfect because it tends to focus on describing individual 

components or aspects of the organization or the processes that its components perform—but 

all as separate entities. One example would be examining an organization’s morale, another 

might be studying its financial health, and another might be looking at relationships with 

external organizations. In none of these examples does the approach offer a comprehensive 

methodology that studies the entire organization and its interactive processes. One of the 
9

fundamental flaws of this single-focus descriptive approach is that the analysis might suggest a 

lesson to resolve a problem in one part of the system, but that remedy in one area might create 

new problems for the organization in other parts of its system (Levinson, et al., 1972, p. 5).

The Traditional Metric Approach. The collection of metric data typically focuses on 

measuring indicators of organizational performance such as cost, schedule, quantity, and 

quality. These kinds of data may have their useful application, but they do not lend themselves 

to identifying lessons that could offer insight into sustaining superior performance or improving 

marginal performance. Van De Ben and Ferry (1980, p. 3) argued that these metrics do not 

explain why or how the organization achieved or failed to achieve a given level of performance. 

They merely measure output and are unable to provide information about the organization’s 

environmental conditions, which are essential to gaining insight about the organization’s 

interaction with that part of the overall system. As a result, the organization’s management will 

not have the insight into factors that might positively influence the organization’s long-term 

performance. One of the fundamental flaws of this metric approach is that the performance 

indicators do not offer insight that could yield broad lessons to positively influence long-term 

organizational behavior (Levinson, et al., 1972, p. 5; Van De Ven & Ferry, pp. 2-3).

The Advantage of Using the General Systems Paradigm Approach 

These traditional descriptive and metric approaches may be useful for some inquiries, but 

they do not lend themselves to identifying lessons needed to sustain superior organizational 

performance and improve organizational performance. They tend to overlook system 

interactions and, as a result, have a high probability of identifying lessons that might suggest 

strategies to improve a part of the organizational system. Without considering the interaction 

of all parts of the system, these strategies could create new problems in other parts of the 

organizational system (Levinson, et al., 1972, p. 5). 

Scientific progress generally is achieved through the use of a framework where the 

constructed model serves as an effective allegory of the real-world phenomena under 

investigation—a framework that approximates what it is that researchers are trying to describe, 

reconstruct, predict, or explain. General systems theory provides that kind of model for studying 

lessons-learned in organizations such as those in the national security, law enforcement, and 

medical communities. It is ideal for collecting data about, and assessing the performance of 

these kinds of and other high-performance organizations (Buckley, 1967, p. 39). 

The general systems model offers a conceptual language and lexicon of terms that are 

applicable to systems of all types. More specifically, it offers an epistemology that can be relevant 

to the study of high-performance/high-tempo organizational behavior and performance. It 

offers a technique for treating large, complex organizations and a common vocabulary that can 

unify several disciplines associated with the study of organizational behavior (Buckley, 1967, p 

39; Sutherland, 1973, p. 19). 
10

Sutherland (1973, pp 139-140) suggested the systems theory framework offers the 

researcher an information-gathering medium. As such, it can be a systematic way to collect 

data about the organization and its performance, and organize the data for analysis.

Conceptualizing the Collection of Systems Data

Lessons-learned researchers face the same challenge as the villagers who studied the 

elephant and collected data in the Sufi story. Using the Intelligence Community as a typical 

high-performance/high-tempo organizational component within the larger national security 

organization, we can see that similarity. When analysts study Intelligence Community 

components they, like the villagers, typically fail to view the target of their study as an 

integrated system. In both cases the general systems model would provide a more holistic and 

integrated orientation to the study and collection of data for what, in both cases, are complex 

systems (Buckley, 1967, p 39). 

The IC and its organizational components are, by definition, systems and share properties 

of other open-ended systems. They are composed of sets “. . . of elements standing in interrelation 

among themselves and with the environment” (von Bertalanffy, 1968). More specifically, they 

are open systems that, unlike closed systems, maintain themselves in a state of continuous 

inflow and outflow of “energy” with their environments. During this inflow, which is in the 

form of resources and information, they break down and build up that energy into various 

components, which in turn the system transforms into an outflow of energy that can be in the 

form of action or product (von Bertalanffy, 1968). 

For the Intelligence Community, agencies initially import energy such as funding, personnel 

resources, and information; then they conduct a transformation process such as analyzing 

intelligence, conducting research and development, or acquiring intelligence systems; and finally 

they export some kind of output product, such as intelligence reports, new ideas, or intelligence 

collection systems. The IC organizations are evolving systems and operate within the boundaries 

of a specific environment that limits their selection and reception of the inputs, and constrains 

the volume and content of the output (Katz & Kahn, 1966, pp. 28, 453). 

Intelligence Community organizations, as social systems, lack the tangible structures of their 

counterparts in physical and biological systems. Rather, their structures are composed of events 

and practices, and therefore the structure is inseparable from the organization’s functioning 

processes. This is why any study of the events and practices of an IC organization requires a 

systems analysis (Katz & Kahn, 1996, p. 69). 

Each agency is a subsystem of the larger IC. For example, on one level, the National 

Reconnaissance Office (NRO) and Central Intelligence Agency (CIA), are 2 of 17 generally 
11

recognized agencies that are part of the larger Intelligence Community.4

 On another level, the 

NRO, CIA, and other IC agencies are composed of subsystems, themselves—directorates, offices, 

and field elements. Those components are composed of other subsystems—program offices and 

other sub elements (Levinson, et al., 1972, p. 4). 

At the same time, the Intelligence Community is part of a larger system known as the national 

security community (with Defense, State, the National Security Staff as other components). The 

national security community, itself, is part of the larger Executive Branch of the U.S. Government, 

which—in turn—is composed of two other branches of government. In a sense, each component 

of the IC is like a Russian matryoshka, or nesting doll.5

 We can see, therefore, that any study of 

an IC component must consider that component as a system that is part of other systems and 

composed, itself, of various sub-systems. 

The study of IC organizations in terms of this kind of process, i.e., in terms of general systems 

theory, is particularly valuable because it offers a framework that is ideal for treating nested, 

large, complex organizations and uses a common vocabulary that can unify several social science 

disciplines associated with the study of both individual and organizational behaviors. It allows the 

researcher to understand the whole system rather than being limited to explaining the interaction 

of one or two of the system’s aspects (Buckley, 1967, p. 39; Levinson, et al., 1972, p. 27). 

To collect relevant data and be in a position to analyze it with the objective of identifying 

lessons for an Intelligence Community component, the researcher must study the full nature of 

the intelligence organization—i.e., the researcher must see the organization as an active, unique, 

changing entity within a particular environment. This approach for organizational assessment 

is an adaptation of research in the life sciences where the biologist routinely deal with living 

organisms that are open-ended, evolving systems (Katz & Kahn, 1966, p. 28). 

4 Sec. 1073 of The Intelligence Reform and Terrorism Prevention Act of 2004 (118 STAT. 3638 Public Law 

108–458—Dec. 17, 2004) described the term “intelligence community” to include: the Office of the Director 

of National Intelligence; the Central Intelligence Agency; the National Security Agency; Defense Intelligence 

Agency; the National Geospatial-Intelligence Agency; the National Reconnaissance Office; other offices within 

the Department of Defense for the collection of specialized national intelligence through reconnaissance 

programs; the intelligence elements of Army, Navy, Air Force, Marine Corps, Federal Bureau of Investigation, 

and the Department of Energy; the Bureau of Intelligence and Research of the Department of State; the Office 

of Intelligence and Analysis of the Department of the Treasury; the elements of the Department of Homeland 

Security concerned with the analysis of intelligence information, including the Office of Intelligence of the 

Coast Guard; and such other elements of any other department or agency as may be designated by the President, 

or designated jointly by the DNI and head of the department or agency concerned.

The Director of National Intelligence [ODNI Home page, retrieval 24 May 2013 from http://www.dni.

gov/index.php/intelligence-community/members-of-the-ic] identified17 agencies as member of the IC. 

In addition to the Office of the DNI, the DNI listed them as: Air Force Intelligence, Army Intelligence, 

Central Intelligence Agency, Coast Guard Intelligence, Defense Intelligence Agency, Department of Energy, 

Department of Homeland Security, Department of State, Department of the Treasury, Drug Enforcement 

Administration, Federal Bureau of Investigation, Marine Corps Intelligence, National Geospatial-Intelligence 

Agency, National Reconnaissance Office, National Security Agency, and Navy Intelligence.

5 The Russian matryoshka doll (sometimes called a babushka doll) is a series of small wooden dolls nested 

within larger and larger hollow wooden dolls. One explanation of the origin of the name, matryoshka (or 

matryona or matriosha), is that it originated from the Latin root, “mater” or “mother” and became associated 

with the image of a portly-figured peasant mother. Some reports explain that folk artisans in the “children’s 

Education” workshop, located near Moscow on the Abramtsevo estate, owned by Sava Mamontov, created the 

first matryoshka in 1890 (Russian Crafts, 2009).
12

Applying General Systems Theory for Data Collection

I will now explain how I adapted the general systems framework for use as a tailored 

lessons-learned systems model to collect and analyze data about organizational behavior and 

performance. I turned to Nadler & Tushman (1989) to develop this tailored model, which can be 

helpful for diagnosing organizational behavior. I adapted and expanded their general model into 

a conceptual framework focused on facilitating the collection and analysis of data associated with 

the behavior and performance of organizations. The goal is to provide the researcher and analyst 

with a tool to identify and evaluate lessons from organizational experience. This lessons-learned 

systems adaptation offers a skeletal framework or structure on which, in Boulding’s (2013, p. 208) 

words, the researcher can “hang the flesh and blood” of the various aspects of the organization so 

as to create “an orderly and coherent corpus of knowledge.”

This adaptation of the Nadler & Tushman model can help the researcher avoid a narrow focus 

typically found in frameworks associated with descriptive summaries and metric assessments. 

This adapted model provides a framework for a comprehensive, interactive assessment of the 

organization—an assessment that will better enable the researcher to look for ways to maintain 

the system in a balanced state. 

In this way, the adapted model is a framework that depicts the organization as an open system 

of systems set within the context of its environment, which is one of the major components of 

the system. The model has four major components: (1) environment, (2) input, (3) organizational 

process, and (4) outcome. Each component has two to five subcomponents or domains. See Table 

1 for a list of the four major components and their domains. 

Each domain has two to six features that define the domain. See Tables 1a through 1d for a 

listing of each domain and its associated features.

Tables that follow derived from: Nadler & Tushman, 1989; Van De Ven & Ferry, 1980, p. 10-13, 171; 

and Levinson, et al., 1972, p. 57. 

System Component Subcomponents (or Domains)

Environment societal

associational

limitations

history

Input requirements (or needs)

funding

staffing

Process formal organization

informal organization

network of individuals

information network

task

Outcome deliverables

process changes

Table 1: LessonsLearned Systems Model 

Components
13

Domains Features

Society

(the human ecology)

geopolitics

domestic politics

economics

other factors

Associational or Relational 

Domain (the external alliances 

and associations)

suppliers

users

stakeholders

Limitative/ Restrictive Domain 

(the structural and behavioral 

constraints)

laws & regulations

external oversight

fiscal

technical

other

Historical Domain 

(the history)

societal

organization

Table 1a: Environmental Component 

Domains Features

Formal Organization demographics

design

components

administrative intensity

bureaucratic constraints

work of organization

Informal Organization relationships

culture

influences

dependencies

certainty

Network of Individuals demographics

education

skills

experience

style

affect and motivation

Information Network dormalization

mode

frequency

quality & content

Task nature

difficulty

expertise

Table 1c. Process Component

Domains Features

Requirements 

(need to be satisfied)

problem

objective

complexity

Funding 

(fiscal resources)

amount

nature

Staffing 

(human resources)

level

nature

Table 1b. Input Component

Domains Features

Deliverable nature

utility

Process Change internal

external

Table 1c. Outcome Component
14

The components and their domains are organized into a functioning system. See Figure 2 

for a macro-level graphic representation of how these components, when adapted from Nadler 

& Tushman, can be organized into a dynamic systems model for lessons-learned analyses. This 

macro-level depiction of the system does not display the complexity of the various subsystems. 

At this macro-level, you can see that the system consists of recurrent cycles of input (from the 

environment), transformation (by the organizational process), and outcomes (outputs into the 

environment). This means that the organization functions as a cyclical process with various 

internal transactions that result in outcomes, i.e., the consequences of the organization’s 

behavior (Katz & Kahn, 1966). 

The “process” component that is depicted in the center, is core to the organization’s 

existence, and, as I indicated, it operates within the context of the external environment. 

The five process domains that I earlier identified—the formal organization, the informal 

organization, the network of individuals, the information network, and the tasks—accept the 

input from the environment, process that input internally, and provide output back into the 

environment as outcomes of the organization’s behavior. Notice that the process component 

is set in the system’s external environment, and the input and outcome components are the 

links of the process to the external environment. In other words, the input comes from the 

environment, and the outcomes return the outputs into the environment.

Figure 2: Lessons-Learned Systems Model (Adapted from Nadler & Tuschman 1989, p. 47) 
15

My basic theoretical assumption in adapting the Nadler & Tushman model is that 

any complex organization, like the example of high-performance/high-tempo Intelligence 

Community organizations, are what Van De Ven & Ferry (1980) described as open social 

systems that consist of a variety of structures and interrelated processes. These IC organizational 

systems, whether at the level of an agency or the level of a program office within an agency, 

operate as a result of its decision makers selecting from a series of iterative choices that arise 

during the organization’s internal processes and relationships with the environment. As I 

suggested earlier, this is a repetitive cycle of procuring, transforming, and distributing inputs 

into outputs with the system constrained by the context of its environment. At the same time, 

it is important to remember that each organizational component has one or more subsystems 

that in a more detailed analysis must be considered (Van De Ven & Ferry, 1980, p. 6).

I believe it is important to study all components of an organization as depicted in the 

model—both within the organization’s process and within the environment—and use the 

model as a basis for collecting data about both aspects of the model. This will provide the analyst 

with adequate information with which to conduct analysis of the organization’s performance, 

i.e., to specify the organization’s components and functions, to describe the interactions within 

the system, to trace those interactions through the organization’s transformation process, and to 

make an assessment about potential lessons. In other words, to conduct lessons-learned analysis. 

This is where, in the next section, I turn to using congruence theory for the analytical stage.
16

Congruence Theory—

A Technique for Lessons-Learned Analysis

“First she lay down upon the bed of the great, huge bear, but that was too high at 

the head for her. Next she lay down upon the bed of the middle-sized bear, but that 

was too high at the foot for her. Then she lay down upon the bed of the little, small, 

wee bear, and that was neither too high at the head nor foot, but just right. So she 

covered herself up comfortably and fell fast asleep.”

—From “The Story of Goldilocks and the Three Bears” (D’Aulnoy, 2007)

The story of Goldilocks provides us with a metaphor to understand and apply congruence 

theory to the analysis of systems. Goldilocks looked for the bed with the best fit for her. She 

tried all three beds, and found the best fit with wee bear’s bed. That is what congruence theory 

looks for, the best fit within and across systems.

Congruence is the relative degree of consistency or “fit” between pairs of domains (i.e., 

subcomponents) within the various components of the system. The fit could be between 

domains within the environment, input, outcome, or process component of the system. Nadler 

& Tushman (1989, p. 45) defined congruence as “the degree the needs, demands, goals, 

objectives, and/or structures of one component [of the system] are consistent with the needs, 

demands, goals, objectives, and/or structures of another component [of the system].” 

Congruence theory is a well-founded principle in the social sciences. It has been effectively 

tested in this regard and is ideal to apply to the analysis phase of our case-study methodology. 

Scholars in political science, organizational analysis, psychology, as well as other related 

disciplines, have applied congruence theory to their research. One classic example is from the 

application of congruence theory to political science. Eckstein argued in his congruence theory 

of political authority that if non-political authority patterns were congruent or similar, then 

political stability would be enhanced (Almond, 1996; Almond & Verba, 1963; Eckstein, 1997).

Eckstein (1998, p. 10), explained that congruence can be understood by considering the 

geometric notion of sameness of form, or isomorphism. In this sense Eckstein pointed out that 

congruence either exists or does not exist, and never is a matter of degree. In geometry “figures 

either can or cannot be exactly superimposed.” But, he argued, that may not be the case with 

political and social systems. For political systems it is unreasonable to expect all the multifarious 

aspects to have identical patterns on all comparative dimensions (Eckstein, 1998, p. 11).

Because government agencies are organizational social sub-systems of a political system—

the nation state—the same principle should apply. Therefore, for government organizations, 

like other political and social systems, there may be degrees of fit, as opposed to an exact fit. 

When we apply congruence theory to analysis in lessons-learned research, we probably should 

employ this more inexact application of the concept, i.e., congruence in terms of having the 

individual components of the systems being more broadly in agreement with the essentials of 

the other components, as opposed to an exact superimposition. 
17

In studying any social organization, whether it be a business organization, or religious 

institution, or a major national security government component like the Defense Department 

or IC, or a subcomponent government agency, such as the Army or the NRO—each with 

its own subsystems program offices—we need to consider the organization as a system that 

must maintain equilibrium (or congruence) among its internal subsystems. That congruence 

also must be examined for each subsystem’s relationship with its larger parent system, plus 

the relationship of each individual system with parallel systems with which it might have 

relationships. If an organization cannot maintain congruence within itself and with its higher 

level systems or relevant parallel systems, then the organization will become dysfunctional, and 

may not even be able to survive (Levinson, et al., 1972, p. 4).

Levinson, et al. (1972, p. 4) pointed out that all open systems, whether they be living 

organisms or social organizational systems, can have problems in adaptation: “Sometimes 

their internal subsystems do not function well together; sometimes their environments change 

so significantly that former modes of adaptation no longer work effectively.” Understanding 

how an organization and its processes are poised to adapt the organization can be helpful in 

determining why it might become dysfunctional, hence, the value of congruence analysis in 

understanding the organization’s processes. 

It is the organization’s processes—those activities that take in resources, do something 

with them, and then put out some kind of product or effect—that are central to understanding 

the organization’s performance, the analysis of which is critical to identifying lessons. The 

process is at the core of the organization’s performance, and it is the performance of the 

organization that is key to the organization’s output, which, in turn, is the critical measure of 

the organization’s success and survival. 

This importance of the system’s process component to the organization’s survival and 

identification of lessons for success, makes it the one component of the system that always 

must be analyzed for congruence. Because of this, I will offer a more detailed explanation of 

assessing the congruence of its domains.

Assessing Congruence of the Domains in the Process Component of 

the System

To assess the congruence of the domains in the process component of the system, the 

researcher needs to evaluate the degrees of fit between each of the processes’ subcomponents or 

domains. Each of the five processes domains (i.e., formal organizational, informal organization, 

information network, network of individuals, and tasks) shown in Figure 2 have dyadic 

relationships with each other. See Figure 3 (on the next page) for a graphic display of these 

dyadic relationships.

The lines in Figure 3 show the dyadic connections between the five domains. Each of 

the domains has a two-way relationship with each of the other four domains, which results 

in four two-way sets of dyadic connections for each of the five domains (e g., the formal 

organization has a dyadic relationship to and from the other four other domains: 1—the 
18

informal organization; 2—the information network; 3—the network of individuals; and 4—

the tasks). That gives each domain four outgoing and four incoming dyadic relationships for a 

total of 20 unique dyadic connections. As a part of congruence assessment, the analyst needs 

to evaluate each of these dyadic relationships. 

Figure 3: Graphic Display of Dyadic Relationships Between the Five Internal Domains in the 

Organizational Process Component of the Systems Model 

Questions for Assessing Congruence

The evaluation of these dyadic relationships can be accomplished by asking questions 

about the relationships. I have developed a question matrix (Table 2) as an example of how to 

assess congruence within the process component. The analyst could use such a matrix to ask 

specific questions about each dyadic relationship, the answers to which would offer insight 

into the degree of congruence. For example, for the two domains “network of individuals” and 

“information network,” the matrix suggests these two questions: (1) “To what degree can the 

cognitive capacity of individuals accommodate the flow of information?” (2) “To what degree 

does the information network provide individuals with the information they require? ”

In answering the first question, the analyst would consider the fact that there are limits 

to the amount of information that individuals can transmit, receive, and process. The analyst 
19

would evaluate the information requirements and assess the degree of congruence between the 

capability of the network of individuals to process information and then transmit it through the 

information network. When the capability of the network of individuals to process information 

is congruent with the nature of the information (i.e., the content, speed, and bandwidth of the 

information matches the capability of the network of individuals), the organization is more 

likely to be successful in producing its desired output. When this is not the case, the desired 

output would be less likely (Katz & Kahn, 1966, p. 257).

Domain Formal 

Organization

Informal 

Organization

Network of

Individuals

Information 

Network Task

Formal 

Organization

To what degree are the 

structures of the formal 

organization consistent 

with the behaviors in the 

informal organization?

To what degree 

does the formal 

organization make 

use of individual 

resources and meet 

individual needs?

To what degree 

can the capacity 

of the formal 

organization’s 

communication 

system handle the 

flow and storage of 

information?

To what degree do 

the structures of the 

formal organization 

motivate taskrelevant behavior 

and facilitate task 

completion?

Informal 

Organization

To what degree is 

the culture of the 

informal organization 

consistent with the 

goals and rewards 

of the formal 

organization?

To what degree 

does the informal 

organization make 

use of individual 

resources and meet 

individual needs?

To what degree are 

the relationships 

of the informal 

organization 

reflected in the 

information 

network?

To what degree do 

the relationships 

of the informal 

organization 

motivate taskrelevant behavior 

and facilitate task 

completion?

Network of 

Individuals

To what degree 

do individuals 

support the formal 

organization’s goals 

and make use of the 

formal organization’s 

resources to support 

their own personal 

goals?

To what degree do 

individuals support the 

informal organization’s 

culture and make 

use of the informal 

organization’s 

relationships to support 

their own personal 

goals?

To what degree 

can the cognitive 

capacity of 

individuals 

accommodate the 

flow of information?

To what degree do 

individual skills and 

abilities match task 

demands?

Information 

Network

To what degree is the 

information network 

consistent with the 

structure of the formal 

organization?

To what degree 

does the information 

network accommodate 

the communication 

expectations of the 

informal organization?

To what degree 

does the information 

network provide 

individuals with the 

information they 

require?

To what degree 

does the 

information network 

communicate 

information relevant 

to the task? 

Task To what degree are 

the demands of the 

task compatible with 

and converge with the 

mission and functions 

of the formal 

organization?

To what degree are 

the demands of the 

task compatible with 

and converge with the 

relationships of the 

informal organization? 

To what degree 

does the task meet 

individual needs?

To what degree 

are the task’s 

requirements 

reflected in the 

flow and storage of 

information?

Table 2: Question Matrix for Assessing Congruence within the Process Component of the System 

Sources: Katz & Kahn, 1966, p. 257; Nadler & Tushman, 1989
20

Considering Congruence with Other Components

For most lessons-learned case studies it may be sufficient to limit the congruence analysis 

to the organizational process component of the model. However in some more complex research 

problems it might require a congruence analysis across components. You can see the value of 

this by looking at the external environmental component.

The organization and its processes can exist in an ever-changing environment, and each 

change would demand the anticipation and execution of relevant responses to those changes. 

This is an area where the analyst could 

look for congruence between the changing 

environment and the subcomponent, 

network of individuals. (Katz & Kahn, 

1966, p. 334). 

For example, when the external 

environment is stable, and we find that 

the process subcomponent, network of 

individuals, is a highly defined structure, 

that process subcomponent would 

be congruent with the environment, 

and the process would effectively be 

able to carry out the organization’s 

mission. Alternatively when the external 

environment is constantly changing, and 

we find that the network of individuals 

remains as a highly defined structure, 

that subcomponent would not be 

congruent with the environment, and the 

process would be less likely to effectively 

carry out the organization’s mission. In 

other words, when the capability of the 

network of individuals to respond to 

the changing environment is congruent 

with the changes in the environment, 

the organization is more likely to be 

successful in producing its desired output. 

When this is not the case, the desired 

output would be less likely (Katz & Kahn, 

1966, p. 334). 

We can see this drawing on 

the Intelligence Community for an 

example. High-performance/high tempo 

Identifying a Lesson though Congruence Analysis

A Study in Responding to an Oversight 

Requirement – Obtaining & Sustaining 

a “Clean Audit”

The Case. In May 2013, the NRO announced that, for four 

consecutive years, the NRO had obtained from its independent 

auditors an unqualified opinion on its financial statements. Congress 

passed the Chief Financial Officer Act in 1990, which increased 

the oversight requirement for Federal agencies by establishing the 

requirement for financial audits. In 1991 the General Accounting 

Office (GAO) concluded that implementation of the act would be 

the beginning of a new era in Federal government operations. 

Federal agencies would now be subjected to the rigorous accounting 

standards more typical in the more open, transparent, and profitmaking world of commerce. This was alien to Intelligence Community 

agencies, which operated in a world of secrets, where the mission 

was the focus, and financial accounting a foreign language. The 

research question was, “How was the NRO able to find success in 

this cultural change, and obtain and sustain a clean audit?” 

Data Collection. The lessons-learned systems model (Figure 2) 

provided the framework for data collection, with the research team 

focusing on the system components that were most relevant to the 

audit case, which were the environment’s historical domain, the 

network of individuals, the formal organization, and the information 

network. The data collection showed that in the early 1990s the 

NRO was facing its own unique organizational and management 

challenges. It was disestablishing what had been a 30-year 

confederation of semi-autonomous, self-sufficient alphabetic 

programs that were strongly tied to external parent organizations. 

It was restructuring that kind of organization into a hieratically 

managed, geographically consolidated, and functionally managed 

organization. At the same time the NRO was moving from operating 

as an unacknowledged, highly secretive activity, buried in the hidden 

world of intelligence operations, into a publicly acknowledged agency, 

visible to all. While focusing on these management and operational 

challenges, it did not have a staff with the expertise to address 

accounting requirements.

Congruence Analysis. The analyst’s evaluation of the data showed 

that the network of individuals involved in business management at 

the NRO was not the best fit for the input requirement of accounting 

standards as defined in the laws and regulations of the limitative 

domain. As a consequence of this analysis, a lesson about expertise 

became apparent, “assign credentialed experts and train the staff.” 

The actionable insight is, “When the NRO moved to a CFO model 

with CPAs in key finance positions and implemented staff training in 

financial management, the finance staff was positioned to obtain a 

clean audit and could sustain a record of clean audits.”
21

organizations, such as the NRO operate in a dynamic state of interactions where both the task 

and environment are in constant change. In one instance the NRO might be dealing with 

evolving acquisition activities where it must regularly form new networks of individuals with 

unique skill mixes to match each new acquisition effort. In another instance the NRO might 

have to continually shift its output goals and work processes in response to ever-changing 

national security threats from the external environment. In both instances well-defined, static 

roles for the NRO’s formal organization and network of individuals would not be congruent 

with either the changing task requirements or the evolving environment. 

Labeling the Dyadic Relationships

After assessing the congruence in the system’s dyads, the next step is to designate 

each dyadic connection as either positive or negative, based on the degree of congruence 

of the relationships. Dyadic connections that are congruent are labeled as positive; dyadic 

connections that are noncongruent 

are labeled as negative. The positive 

dyadic relationships (PDRs) enhance 

organizational performance; the 

negative dyadic relationships (NDRs) 

frustrate successful organizational 

performance. 

The greater the total degree of 

congruence between the various dyads 

in the system and the relevant external 

factors, the more efficacious will be the 

organization in terms of the degree that 

the actual outputs are consistent with 

the desired outputs. This then allows the 

analyst to develop hypotheses regarding 

what organizational behavior is most 

likely to bring the desired results from 

the organization’s performance; in other 

words, to form hypothesis statements 

that could be potential lessons (Nadler 

& Tushman, 1989, p. 45). 

The analyst formulates these 

potential lessons by explaining the 

contingent organizational behavior 

that results in either a PDR or NDR. 

The analyst tests these hypotheses by 

assessing their utility in explaining or 

Identifying a Lesson though Congruence Analysis

A Case Study in IC Integration – Badge 

Interoperability

The Case. In 2007 the Intelligence Community (IC) succeeded in a 

significant step toward IC integration when it implemented the first phase 

of security badge interoperability for the “big five” IC agencies (i.e., CIA, 

DIA, NGA, NRO, and NSA,). This permitted the employees of any one 

of these agencies to use their agency’s badge to gain access to any of 

the other four agencies, thereby facilitating the sharing of information 

between agencies. However, it took four years to accomplish this step. 

There are valuable lessons for other, yet unaccomplished integration 

tasks. To help identify some of those lessons in this case, a key research 

question was, “What discernible factors or conditions enabled the 

success of badge interoperability, a necessary step in the IC’s road to the 

integration of the community’s intelligence collection and analysis?”

Data Collection. The lessons-learned systems model (Figure 2) provided 

a useful framework for data collection, with the research team focusing 

on the system components that were most relevant to the case, in this 

instance, the environment’s historical domain, the geopolitical segment 

of the societal domain, the network of individuals, and the information 

network. The data collection showed, among other things, that the IC had 

attempted, without success, badge interoperability two earlier times over 

the previous 37 years, once in 1970 and the other in 2000, each period 

presenting a very different geopolitical environment. 

Congruence Analysis. The analyst’s evaluation of the data showed 

that in 2003, when the DCI directed badge interoperability for this third 

attempt, there was a better fit in the system between the environment’s 

geopolitical dimension and the task component of the system (i.e., 

in 2004, a post- 9/11 geopolitical threat made it an imperative for IC 

agencies to facilitate sharing of intelligence in order to be able to “connect 

the dots”). As a consequence of this analysis, the lesson about being 

committed became apparent, i.e., “When the IC commits to addressing a 

well-defined need, success for interagency programs is more likely.” The 

actionable insight is, “When attempting a task related to IC integration, 

ICS leadership can increase the likelihood of success by framing the 

requirement as a national security imperative in light of a threatening 

geopolitical environment.”
22

predicting the outcomes of phenomena in the particular case under study. In doing this, the 

analyst must determine whether the differences in case outcomes are causal or spurious. If 

the potential lesson consistently predicts the case’s outcome, that serves as evidence that the 

potential lesson can be a generalizable explanation of the phenomena; if the potential lesson 

fails to explain the case’s outcome, that argues for discarding this as a potential lesson (George 

& Bennett, 2005, p. 181-2). 

George & Bennett (2005, pp. 6-7) described this step of “hypothesis testing” as “processtracing” where the researcher attempts to trace links between the possible causes and the observed 

outcomes. In effect, the researcher must examine the instances of congruence and noncongruence 

to determine whether the hypothesized cause suggested by the data is revealed in the patterns of 

congruence and noncongruence in the case. In more direct language, when applying congruence 

theory to lessons-learned analysis, the researcher is looking for the Goldilocks effect, i.e., the 

best fit in the various relationships within and external to the organization.
23

Integrating the General Systems Paradigm and 

Congruence Theory into The Case Study Framework

Grandma is baking a cake. She asks the child if he would like a snack, which 

of course he does.

“Here. Have some cooking oil.”

“Yuck,” says the boy.

“How about a couple of raw eggs?”

“Gross, Grandma.”

“Would you like some flour then? Or maybe baking soda?”

“Grandma, those are all yucky!”

To which the Grandma replies: “Yes, all those things seem bad all by 

themselves. But when they are put together in the right way, they make a 

wonderfully delicious cake!”

—From Wisdom Commons web page, by unknown author

In the previous three sections, I described the three lessons-learned research methodology 

components: (1) the case study framework, (2) the general systems paradigm, and (3) congruence 

theory. Like grandma’s cake, they need to be put together in the right way to make a “wonderfully 

delicious” disciplined lessons-learned research methodology. The recipe to integrate them into 

a coherent and successful approach is a three-phased process. After some brief introductory 

comments about the nature of this integrated methodology, I will describe how to complete 

each phase. 

Using the disciplined, lessons-learned, case study methodology is not pure research for the 

purpose of developing theoretical social science constructs; rather it is applied research that is 

comparable to the medical model because of its focus on diagnosis and treatment. There are 

three characteristics that argue for this similarity: (1) lessons-learned research involves studying 

organizations that are open-ended, ever-changing systems, which are like the living organisms of 

medical interest; (2) lessons-learned research has as its objective the assessment of a phenomenon 

with the identification of lessons for application to future organizational behavior, which is 

comparable to the medical practice of diagnosis and prescriptions for treatment or continued 

wellness; and (3) lessons-learned research has as its overall goal providing insight to sustain or 

improve the organization’s process, which is comparable to medical practice where the goal is to 

use insight from the diagnosis to improve the patient’s long-term personal health and well-being. 
24

Like the medical model, the lessons-learned research model is interactive in nature with 

the success of each step dependent on completing the previous step. Because the organizational 

system under study is in constant change, the researcher regularly may need to return to a 

previous step to modify and refine what he or she did in that step. As the researcher observes 

the changes in the system, the researcher will need to adapt his or her approach to applying the 

methodology. This makes lessons-learned research closer to the practice of medicine than to 

research using the classical scientific method.

In the classical scientific method, the researcher typically frames hypotheses at the start of 

the research and keeps the hypotheses intact throughout the data collection and analytical steps. 

In the medical model, the practitioner forms hypotheses, but he or she modifies them in the face 

of a changing system and emerging data. At the same time, the practitioner searches for new 

data in the face of the evolving hypotheses. 

In this way the success of each step clearly is dependent on the researcher successfully 

completing the previous step, and it may require the researcher to return to a previous step when 

new insights emerge. In other words, the three phases and their steps in the research process are 

separate, but interrelated.

Whereas the medical model may be more open-ended because it deals with a living 

organism, the methodology for lessons-learned research is more structured because it deals 

with a more abstract organizational system. It is composed of three phases that integrate the 

case study framework with general systems paradigm and congruence theory: (1) developing 

the research design; (2) preparing the description of the case or cases; and (3) conducting the 

lessons-learned analysis. Each phase has three or four steps, which you can find outlined in 

Table 3, and discussed in more detail immediately following the table. 

Phase I—Developing of the Research Design

1 Defining the Research Problem

2 Selecting the Case or Case(s) to be Studied

3 Developing the Research Schedule and Spend Plan

4 Writing the Research Design

Phase II—Collecting Data and Preparing the Case Description

1 Collecting Data

2 Reviewing the Collected Data

3 Writing the Case Description

Phase III—Conducting the Lessons-Learned Analyses

1 Assessing Congruence and Noncongruence

2 Formulating Tentative Lessons as Hypotheses

3 Testing Hypotheses and Identifying Lessons

4 Writing the Lessons-Learned Case Assessment

Table 3: Case Study Research Phases
25

Phase I—Developing of the Research Design

The research-design phase may be the most critical. It provides the foundation and 

direction for the two remaining phases of research. It focuses the inquiry, defines the scope of 

the research strategy, and structures the timeframe for the research. 

The researcher must first clearly define the problem prior to being able to select the case 

that he or she will study. A clear definition of the research problem gives direction to overall 

development of the research and provides a framework for selecting the case for study. This 

ensures that the selected case supports the research objective. The writing of the research design 

is dependent on, and a reflection of, this analytical process.

Not only must the case that the researcher selects be responsive to the research problem, 

but also the overall research design must be sustainable by the available resources. In other 

words, the scope of the research must be feasible within the level of allocated resources and 

available time.

As the researcher encounters the data and becomes familiar with the phenomenon under 

study, he or she also may have to initiate iterations and respecifications of the research design. 

New data and evolving analyses might offer new insight into the problem and expose additional 

areas for inquiry that will demand adjustments to the research design. The researcher may have 

to return to phase I to interactively refine the problem statement, modify the case definition, 

and adjust the research.

Unlike some other social science researchers whose work is experimental or theoretical in 

nature, the lessons-learned researchers should not be surprised if additional insight occurs well 

into the research with the researcher having to suspend work and redesign the study more than 

once. This is the nature of research that studies a dynamic organizational system. Suspending 

work and redesigning the study in response to new data and ongoing analyses will increase 

the likelihood that the lessons-learned research project will produce results that are valid and 

generalizable (George & Bennett, 2005, p. 88)

The key to success in Phase I is to address completely and interactively its four interdependent 

steps—(1) define the research problem, (2) select the case or cases to be studied, (3) develop 

the research schedule and spend plan for the research project, and (4) write the research design 

plan. Here are some considerations for dealing with each of these steps. 

Defining the Research Problem. The research problem is the broad research question 

that the researcher is seeking to answer. It should be a concise statement of the question or issue 

that gave rise to the research. The question should relate to the kind of lesson the researcher 

is seeking to learn. For example, if the researcher is trying to learn lessons about the kind of 

strategy that resulted in the development of a technical intelligence collection system ahead of 

schedule, the researcher might frame the research problem in the form of this kind of question: 

“Why was the Agency able to deliver ‘program A’ one year ahead of schedule, while it delivered 

‘program B’ two years late?” (Bromely 1986, p. 72)

Defining the problem and associated research objectives is the most important step in 

designing the research. This step establishes the constraints of the study and focuses the nature 
26

of the research. If the researcher creates a well-reasoned, clear statement of the problem, then 

this will move the research in the correct direction (George & Bennett, 2005, pp. 74, 79).

The researcher needs to be careful not to define the problem too broadly, or he or she 

will risk losing important differences among the factors under study. At the same time, the 

researcher needs to take care not to define the problem too narrowly, or the researcher might 

unnecessarily limit the scope and relevance of the study. Finally, the researcher needs to view 

the problem statement as a tentative research question and remain open to future refinements 

during the course of the study as the researcher collects data and conducts analyses (George & 

Bennett, 2005, pp. 74, 79).

Selecting the Case or Case(s) to be Studied. The case or cases that the researcher selects 

for analysis should be a specific incident or process that is typical of the phenomenon to be 

studied. Lessons-learned researchers could begin the selection process with a predetermined 

case in search of a lesson to be identified or a lesson in search of a case with which to test a 

hypothesized lesson. Either approach should work.

The researcher should select a case that is consistent with and can become an integral part of 

a research strategy focused on achieving the objectives of the study. The selection criteria should 

not be that the case is interesting, important, or has readily available data for research. The 

selected case must focus on a narrow, specific issue that is associated with the phenomenon under 

study. The issue or incident must be constrained by limits both in time and space; it should not 

be a broad, comprehensive history of the phenomena. The researcher should not select a case that 

reflects an anomaly that falsely supports the viability of a lesson that is not generalizable to the 

population of experiences. The researcher also needs to take care to avoid bias from any internal 

or external sources when selecting the case (George & Bennett , 2005, p. 83).

As part of the case selection process, the researcher must determine if the research project is 

to be a single-case study that examines one incident or a comparative case study that compares 

two or more incidents. The researcher should base this decision on his or her understanding of 

the research question and associated research objectives that he or she defined in the previous 

step of this phase. Because of the complexity of systems-based case studies it might be best to 

study one case, or at the most, compare only two cases when doing comparative case studies. 

Depending on the research objectives, comparative case studies can involve cases that either are 

instances of the same subclass of events or different subclasses. The pair could be before-after 

instances within the same organization, or they could be the most similar or the least similar 

organizations involved in similar or dissimilar instances (George & Bennett , 2005, p. 83).

During the course of the research study, especially during the next two phases, the 

researcher might find reason to move away from the initial case or cases that he or she selected. 

It may be necessary to return to the step for selecting the case and select a better case to support 

the research question. For example, the initially selected case might draw a researcher to a 

particular hypothesized lesson different from the preliminary assessment. After initial data 

collection and analyses, the researcher might gain insight that would move him or her away 

from the initially selected case to another case or series of cases that might appear to be more 
27

appropriate to test the validity and universality of that lesson (George & Bennett , 2005, pp. 

38, 83-84; King, Keohanne, & Verba, 1994, pp. 17-18).

Developing the Research Schedule and Spend Plan. The development of the research 

schedule and spend plan makes explicit the available resources, and brings them together with 

the research strategy in the form of a project timetable. This is that part of developing the 

research design where the researcher determines how realistic and feasible the research objectives 

are, given the available resources and allocated time. This step answers questions such as: does 

the researcher have sufficient resources to accomplish the research; can the researcher complete 

the research project in time for the findings to be useful; does the researcher have to go back to 

the problem definition to reduce the scope of the project to match the resources and satisfy the 

time requirement for delivery of the results? 

As with the other steps of this phase of the research, the researcher might find it necessary 

to return to this step later in the research study in order to adjust the schedule or modify 

the spend plan. There could be a whole range of reasons for this. The researcher might find 

that data collection is taking longer or is being completed more quickly; he or she might 

find that more or less funds are necessary to accomplish the research project; unanticipated 

events might occur; or the organization might change in a way that would make it more or 

less urgent to deliver the results. Because of the uncertain nature of the research schedule and 

funding requirements, the researcher should seek to build flexibility into the research schedule 

and spend plan to accommodate these unanticipated factors that inevitably will occur during 

almost any research study.

Writing the Research Design. This step involves the actual drafting of the research 

design. The written research design should be relatively brief and address six areas: (1) the 

research problem, (2) definition of terms, (3) scope of research (including identification of 

the selected case), (4) methodology, (5) research schedule, and (6) spend plan. (See Appendix 

A for a research design outline with questions to consider.) The statement of the problem 

should be a concise statement and include a one- or two-paragraph background summary. 

The case selection discussion in the “scope” section should include a brief summary of the 

rationale for selecting the case and an explanation of whether the study will be a single case 

study or a comparative case study. The methodology portion briefly should explain how the 

researcher will apply the disciplined lessons-learned research methodology to the particular 

case under study. The funding and schedule portions of the design should be a brief outline of 

the spend plan and project schedule. As you can see, the emphasis is on a “brief, but focused” 

product. The researcher should use his or her energy on the creative and analytical processes in 

preparing the research design, rather than writing unending prose.

Phase II—Collecting Data and Preparing the Case Description

In order to describe the case, the researcher first must collect relevant data about the case. 

All case data, however, are embedded in the historical and environmental domains in which 

the organizational system existed at the time of the events relevant to the case. The various 
28

environmental domains—societal, associational, restrictive, and historical—shape and limit 

the data that can be collected about the case. The researcher needs to be conscious of those 

limitations and how they structure the availability of case data. 

The data collection for the case also is influenced by a parallel series of environmental 

domains—those that are prevailing at the time of the research. Each datum from a past event 

will be interpreted in terms of the more current environmental domains that exist during the 

time of the data collection. The nature of these more recent environmental domains already 

will have influenced the initiation of the case study. During data collection, they additionally 

will influence: the selection of data, the perspective for evaluating collected data, and the 

framework of norms and values that will be used to identify the lessons. The implementation 

of the lessons, within the context of those norms and values, will, in turn, influence the future 

performance of the organization. This is why the collection and evaluation of data in this phase 

of the research are so critically important (Bromely, 1986, p.267).

Data collection involves describing all relevant aspects and interactions of the system, 

and it can become the most laborious and detailed aspect of the research. This phase of the 

lessons-learned research project focuses on both collecting the data by conducting a broadbased examination of the organizational system that is associated with the case incident. 

The goal is to identify the factors that will serve as the basis to formulate lessons-learned 

hypotheses. This phase of the research has three steps: (1) collecting the data to learn about 

the organizational system, (2) reviewing collected data, and (3) writing a descriptive case 

study based on the collected data.

Collecting Data. This step involves the collection of detailed data about the organizational 

system under study. The collection of data is a central step in conducting lessons-learned research. 

The collected data should be empirical evidence that is relevant, confirmable, and reliable. The 

data collection step should result in compiling considerable documentary evidence. However, 

the researcher needs to avoid filtering the data and making any interpretations, inferences, or 

assertions that go beyond the empirical evidence. Those activities should be deferred to the 

analytical stage when the analyst will need to examine the data critically. If the researcher 

collects sufficient relevant, detailed, confirmable, and reliable data, it will assist the researcher 

in formulating and testing hypotheses as he or she moves through the later research phases 

(Levinson, et al., 1972, p. 7; Bromely, 1986, pp. 69, 71-72). 

The researcher needs to ensure that he or she collects data for all decisive points in 

the case narrative and anchors those points by supporting the associated observations with 

documentary evidence. The collected data must be comprehensive, to include details not only 

about the system, but also about its environment, which includes its history. Research evidence 

has shown that individuals select, interpret, and make decisions based on the context of the 

environment. The environment, with its history becomes a major component in determining 

the frame of reference for all subsequent analyses. The frames of reference become “the internal 

standards or cognitive filters a person uses” in evaluating a situation and making analytical 

decisions (Van De Ven & Ferry, 1980, pp. 56-58; Bromely, 1986, pp.69, 71). 
29

Referring to the lessons-learned systems model (Figure 2) during data collection can 

help the researcher become aware of latent cognitive filters and ensure that the researcher 

considers the full range of potential data that may be embedded in the organizational system, 

i.e., within the process, the environment, the input, and the outcome. Use of the model also 

can help the researcher in formulating data collection questions that might be unique to the 

case, in organizing the collected data, and in uncovering insightful relationships between data 

elements (Levinson, et al., 1972, p. 7).

I have identified a set of comprehensive, normalized questions that the researcher can use 

as a baseline for initiating data collection. I have listed them in Appendix B (Data Collection 

Framework and Examples of Questions). They offer a methodical and comprehensive way to 

address the relevant aspects of the lessons-learned systems model (Figure 2) in terms of the 

way the specific case operates. The use of these normalized questions can serve as a checklist 

to help ensure standardized data collection with analogous inquiries, both within the case 

and across cases. This will allow for the comparability of data and help ensure that the data 

can be accumulated, matched, and systematically analyzed in the subsequent steps of research 

(George & Bennett , 2005, p. 86). 

When answering the data collection questions, the researcher should use all available sources 

for data collection: documentary material (publications, transcripts, and photographs), archival 

records (government and corporate), correspondence (written, electronic, and recorded), surveys 

(psychometric measurements and other social science assessments), recollections (memoirs, 

interviews, and oral histories). When selecting data sources, the researcher should give preference 

to original, primary sources,6

 rather than secondary sources.7 For the most part, the researcher 

should avoid relying on tertiary sources (Bromely, 1986, pp. 71-72).8

Although primary sources are the ideal for data collection, they can produce information 

with inaccuracies. Any individual primary source requires validation from other firsthand, 

contemporaneous primary sources, which could be documented in the form of written records, 

digital files, photographs, or multi-media recordings. Secondary documentary material—such as 

contemporary secondhand accounts and subsequent scholarly assessments—also can be helpful 

in evaluating primary sources. In addition, they can facilitate the researcher’s understanding of 

the context in which the case operates and help raise questions about the accuracy of the primary 

sources (George & Bennett, 2005, pp. 96-97).

One very common approach to collecting primary source data is to use oral history 

interviews. This method is at risk of collecting spurious data for a number of reasons. The 

6 Primary sources are original reference materials from the time period involved. They would not have been 

filtered through interpretation or evaluation over time. They usually are the first formal documentation of new 

information, original thinking, or a discovery recorded in physical, print or electronic format (UMD, 2012). 

7 Secondary sources are interpretations and evaluations of accounts in primary sources written after the 

fact with the benefit of hindsight. They are not evidence, but most often discussions or commentaries about 

evidence (UMD, 2012).

8 Tertiary sources are collections of information that are a distillation of primary and secondary sources 

(UMD, 2012).
30

interviewee’s perspective can be influenced by innate biases; the interviewer might have created 

questions that were affected by his or her own biases; or there may have been fundamental flaws 

in the design and structure of the oral history interview process. See Appendix C (Interview 

Protocol) for guidelines on conducting interviews to minimize these kinds of errors (Bromely, 

1986, pp. 71-72).

Reviewing the Collected Data. The purpose of this step is to fill in any gaps in the 

collected data. During this step, the researcher first reviews the systems model (Figure 2) as 

applied to the organization under study. The researcher then examines the collected data using 

the standardized questions as a check list (Appendix B) to identify gaps in the collected data. 

The research needs to verify that he or she has collected all the data that would be essential to 

describing and understanding the case. Should any data be missing, the researcher needs to 

return to the data collection step, and fill in the gaps.

Writing the Case Description. This step involves writing a description of the system 

being studied; it is the descriptive case study. The descriptive case study should explain the 

entire system (Figure 2), the organization’s process, its environment, the input for the process, 

and the outcome of the process. The researcher should include sufficient detail about the 

system in order to be in a position to conduct the lessons-learned case assessment in Phase III 

of the research process. Refer to Appendix E (Outline and Checklist for Case Description) for 

guidance on drafting a narrative case description. 

Phase III—Conducting the Lessons-Learned Analyses

The lessons-learned analyses may be the most difficult part of the research project. This 

phase is fundamentally an analytical and writing activity. It focuses on developing and testing 

the lessons-learned hypotheses. It has four steps: (1) assessing congruence; (2) formulating 

lessons-learned hypotheses; (3) testing hypotheses and identifying lessons, and (4) writing the 

lessons-learned case assessment. 

Assessing Congruence and Noncongruence. This step involves a three-part process: (1) to 

assess the congruent fit of domains within the system’s process component (i.e., the congruence 

of the dyadic relationships of the domains that are part of the process component as shown 

in Figure 3 on p. 18 in this publication—i.e., the domains of formal organization, informal 

organization, network of individuals, information network, and task); (2) to determine if—other 

than in the process component—there are any other relevant dyadic relationships that need to 

be assessed for congruence (i.e., the need to assess domains in the other system components 

as shown in Figure 2 on p. 14 in this publication— i.e., the environment, the input, and the 

outcome); and (3) if there are other dyadic relationships that need to be assessed, to assess them. 

As I explained in more detail earlier, when the researcher is evaluating dyadic relationships, 

he or she is looking for both (1) congruent, or positive dyadic relationships (PDRs)—those 

that enable and enhance optimal organizational performance, and (2) any non-congruent or 

negative dyadic relationships (NDRs)—those that frustrate the organization from meeting its 

goals—i.e., block or distort optimal organizational performance. 
31

When assessing congruence within the process component of the system, the analyst 

will be evaluating the fit between each dyad within the process’ domains. A question matrix, 

such as in Table 2 (p. 19 in this publication), can be helpful in identifying PDRs and NDRs 

in the process domains.

When assessing congruence in the domains of components other than the process 

component, the analyst should consider asking two fundamental questions: (1) to what degree 

does this dyad reinforce organizational performance, and (2) to what degree does the dyad 

block or distort the system’s output and contribute to the organization’s production failures? 

The answers to these two questions will be helpful in identifying PDRs and NDRs for these 

components of the system.

The compilation of PDRs and NDRs from the analysis of dyadic relationships becomes 

the basis for formulating lessons-learned hypotheses. 

Formulating Tentative Lessons as Hypotheses. This is a critical step where the analyst 

develops tentative lessons to serve as hypotheses to be tested in the next step. It involves 

formulating into generalized lessons, specific explanations for the effects that resulted from 

congruence and non-congruence in the system.

The analyst should formulate the lessons in a behavior/consequence phraseology, i.e., 

state a conditional behavior (by an individual or organization), and then state a contingency 

consequence that results from that behavior. This puts the lesson in an operational form that 

is actionable for the individual or organization. Two hypothetical examples of such a statement 

would be: “When a manager limits his or her direct reports to a number less than 10, the 

manager will have better communication and control in the organization.” “When a supervisor 

delegates the whole task to a single employee, the employee more frequently completes the task 

in a timely manner.”

By definition, when a dyad comparison results in an NDR, it implies an output failure 

related to a behavior that the organization (or individual) should avoid. However, the analyst 

should seek to formulate the behavior/consequence lesson for avoidance in a positive way. 

Looking at an example from early childhood learning, the analyst would avoid formulating 

the lesson this way: “When a parent prevents a child who is learning to walk from taking the 

risk of falling, the child will not have an opportunity to attain balance and will take longer to 

learn to walk.” The preferred positive formulation for the lesson would be, “When a child who 

is learning to walk is given the opportunity to risk falling, the child is more likely to attain 

balance and learn to walk quicker.” 

The analyst will next test the hypotheses that he or she developed from the 

congruence analysis.

Testing Hypotheses and Identifying Lessons. This is the core step in this phase 

where the researcher tests the validity of the hypothesized lessons. To test the hypotheses, 

the researcher should evaluate the hypothesized lessons using the case description from Phase 

II. The case description will contain evidence that either supports or rejects the hypothesis. 
32

The hypothesized explanation only can be considered a lesson when the analyst can point to 

consistent empirical data that supports the hypothesized generic lesson. 

The researcher must take care when explaining the causes of the organization’s performance. 

First, the researcher must avoid relying on correlations as the only basis for explaining an 

organization’s behavior—correlation is not necessarily a reflection of causation. Second, the 

researcher must consider the role of equifinality and multifinality in identifying and explaining 

system outcomes. In equifinality, a given system outcome can be reached from any number of 

paths. In multifinality, similar initial conditions may lead to dissimilar system outcomes.

Finally, the researcher must be cautious to avoid inadvertently identifying spurious 

lessons. George & Bennett (2005) identified one example of a spurious lesson in a case where 

the researcher observed that organizational decision-making was decisively affected by the 

absence of a key participant who had become ill. A spurious conclusion would be to frame 

the lessons with this statement, “When key decision-makers are free of illness, the quality 

and timeliness of decision-making is more likely to improve.” Alternatively, this might be a 

better lessons statement, “When all key participants are present, the quality and timeliness of 

decision-making is more likely to improve” (George & Bennett , 2005, p. 92-3).

Writing the Lessons-Learned Case Assessment. This step involves writing the lessonslearned case assessment (See Appendix F - Outline for Lessons Learned Case Assessment). 

As with the case description, this report should state its main points at the beginning. In 

particular, this first part of the report should highlight the generalizable lessons learned from 

the research. It also needs to include some number of empirical examples drawn from the 

descriptive case summary to support the hypothesized general lesson. If the statement of the 

general lessons is so generic as to lack actionable specificity, then the analyst should frame a 

secondary lesson that would be more specific to the case and would offer actionable insight.

Both the statement of the general lessons and the statement of any actionable insights 

should be formulated in the behavior/consequence phraseology that I discussed earlier. The 

analyst also should explain the analytical rationale that lead to the formulation of the lessons. 

In the final section of the written case assessment, the analyst also should identify some of 

the unanswered questions and identify any areas for further research. Because of the tentative 

nature of all social science research, this final section of the case assessment also should provide 

an evaluation of the research and a statement on the level of confidence for the findings. Much 

of the uncertainty in the research may come from the inherent flaws in social science research, 

which I discuss in the next section.
33

Inherent Flaws in Lessons-Learned Research and its 

Methodologies

“Watch out, Achilles, for although you are a mighty warrior, you are still just 

a mortal man.” Achilles ignored Apollo and fought on with savagery. . . . Atop 

the safety of the city walls, Paris sat with his bow and arrows. He drew a single 

feathered arrow from his quiver . . . and inserted it in his bow. When Paris let the 

arrow fly, Apollo, the god of archery, used his great powers to guide the arrow so 

that it hit Achilles in his heel. This was the one spot on his body where Achilles was 

vulnerable. Achilles fell back on top of the bodies of many slain warriors. Soon his 

eyes, too, misted over in death, and he died, just as was fated. “

—From The Iliad and the Odyssey in Greek Mythology by Karen Bornemann 

Spies, 2002

All research methodologies have their vulnerabilities, just as Achilles had a fatal 

vulnerability in his heel. The lessons-learned case-study methodology that I have outlined 

is not unlike any other social science research methodology; it suffers from inherent flaws 

characteristic of the social science and history disciplines. The analyst must avoid the arrogance 

of Achilles, lest the flaws in the methodology destroy the research as the arrow did Achilles. 

There are two categories that are of particular importance and make the methodology most 

vulnerable. One is related to data errors that arise during data collection, and the other is 

related to the cognitive errors that occur during analysis. These deserve further discussion, and 

I will briefly discuss them in the next two sections.

Data Collection Errors

Lessons-learned analysts draw on a wide range of data that includes current records, 

historical documentation, and personal information. Dealing with any of these kinds of data 

potentially is problematic. Data errors can arise from two sources—either from deficiencies 

within the datum or from the assumptions the researcher makes as he or she collects the data. 

There are four data errors that the researcher could make: assume that answers are in the 

available records; rely on an apparent single authoritative source; find the complete record 

unavailable; and collect unreliable data (George & Bennett, 2005, p. 95). 

Assume that Answers are in the Available Records. The available records may include 

previously conducted lessons-learned analyses or after-action reports; however, if the researcher 

assumes that the answers in those reports are the correct answers for the current research, 

that can lead to data collection errors. First, the assessments in previously conducted afteraction reports and other historical inquiries are dependent on either the specific questions 

that those earlier investigators asked or dependent on the context of their earlier inquiries. 

These earlier assessments may not be consistent with the current research question. Second, 

the earlier investigators may not properly have weighted the evidentiary significance of the 
34

data they reviewed, or they may not have asked the full range of research questions appropriate 

for the general systems model. As a result, the current lessons-learned analysts must be careful 

not to take for granted that these earlier inquiries provide the answers to the current research 

questions. They must look for gaps in any earlier inquiries, particularly with regard to answering 

the research questions in the general systems model. They must seek not only to fill any gaps 

in earlier research, but they must independently evaluate all the relevant available data (George 

& Bennett, 2005, pp. 96, 99).

Rely on an Apparent Single Authoritative Source. Relying on a single source of what 

appears to be authoritative data can be problematic. For most cases of high interest, there are 

at least some after-action reports or case histories that appear to be authoritative reviews of 

the case. These reports can be misleading in the current research if the lesson-learned analyst 

heavily relies on them as background narratives to extract neutral data for framing the problem 

or testing hypotheses. These earlier reports tend to be accounts of the events that are based on 

the original author’s perspective. They are written from the perspective of that author’s personal 

commitment to his or her world view of the issue, and that author’s methodological decisions 

that governed his or her research. While they may be flawed, they can prove useful. The lessonslearned analyst can look for opportunities to translate historical narratives and after-action 

reports into competing hypotheses to challenge his or her own tentative assessments (George & 

Bennett 2005, p. 95-96).

Find the Complete Record Unavailable. One of the most limiting aspects of historical 

and lesson-learned research is the unavailability of the entire set of documentary records for 

the organization’s history. As a result, the collected data will be incomplete and not tell the full 

story. This could be the result of a variety of factors. First, for many organizations, the records 

may not have been preserved—and if preserved, not consolidated, indexed, and archived. 

This especially could be case for when an agency or business organization was restructured, 

consolidated, or disbanded over its history, and as a result the records may have become 

dispersed or lost. Second, when the records are predominately in digital form (such as in e-mail), 

those records may not have been incorporated into the formal records management system and 

subsequently incorporated into the organization’s archive. Third, because of the contemporary 

nature of analytical lessons-learned research, many of the most relevant records may not yet 

be available in the archive because they still are in use. These factors make it imperative that 

during the research stage, the analyst take into consideration that the available data reveal only 

part of the story, and the analyst must be persistent in looking for additional data sources. 

During the assessment stage, the analyst must take into consideration the possibility that gaps 

continue to remain in the collected data, and these gaps are likely to influence the analysis. 

The analyst must keep asking the question, “What data are likely to be missing, and how is 

this influencing my analysis?”

Collect Unreliable Data. Some of the data that a researcher collects may prove to be 

unreliable, whether they be based on scholarly and scientific secondary sources or drawn from 

primary source documentary records and personal recollections. With regard to secondary 
35

sources, the authors may have done incomplete or biased research. When we look at primary 

sources, which we might expect to be more accurate and reliable because they are firsthand records, we find that they can be equally flawed. The originators of primary source 

documentary material could have entered distortions into the record either consciously with 

the intent to deceive or unconsciously as a result of cognitive biases. Any information in a 

primary source could be corrupted by perceptual errors, be documented out of context, or be 

based on hearsay, rather than being derived from first-hand observations (Bromely, 1986, p. 71; 

George & Bennett, 2005, pp. 99-101). 

Memory can present a more problematic challenge when dealing with primary sources, 

especially those that rely heavily on personal recollections, such as oral history interviews, 

autobiographies, memoirs, and diaries. The information in these kinds of sources, all of which 

are memory dependent, can be flawed because of the introduction of a range of errors that are 

inherent in the memory process. Appendix D is a commentary on memory and offers some 

insight into the range of errors associated with memory and recall.

Data Error Explanation of Error Questions to Minimize Errors

Assume Answers are in 

Available Records

Assuming data in the records contain 

answers to the questions in the 

current research

What is the validity of the records in the archive 

under study?

For what purpose did the author(s) create the 

record(s)?

How do the records fit into the process under study?

What is the relation of the information in the 

record(s) to the stream of other communications and 

documentation?

Rely on a Single 

Authoritative Source

Relying on what appears to be a 

single definitive authoritative source

How relevant is the information in the source to the 

current research question?

How objective was the collection of data for this 

“authoritative source”?

Complete Record 

Unavailable

The entire sent of documentary 

records is not available

How likely is it that the available data reveal only part 

of the case story?

What data are likely to be missing?

How is the information that is in the available records 

influencing the search for additional data?

What information might be missing from the record?

How is the limited amount of available data likely to 

influence the subsequent analysis?

Collect Unreliable Data Some of the data in the documentary 

record or interview transcripts may be 

unreliable

What distortions might the originator of the data have 

introduced into the record?

What was the objective of the originator of the record 

in preparing the record?

What were the biases of the interview subjects?

What is the likelihood that the source of data could 

have entered self-serving information into the record?

Table 4: Questions to Minimize Risks From Data Errors
36

Avoiding Data Collection Errors 

The analyst must assess the evidentiary worth of all data in terms of not only what was 

preserved in the actual record, but also in terms of the context and purpose of what was 

preserved. The lessons-learned analyst should evaluate who is speaking to whom, for what 

purpose and under what circumstances. Knowing the purpose for which the original drafter 

prepared the document or conducted the interview should offer insight into its reliability. 

Often originators of documentary records and participants in interviews desire to enter selfserving information into the historical record. The analyst must carefully assess these error 

sources. Table 4 is a guide dealing with data collection errors. It lists the four data collection 

errors I discussed in this section, explains the errors, and offers a series of questions that you 

can use to help minimize the risks associated with data collection errors.

Cognitive Errors

Cognitive errors are the result of distortions and biases in our thinking as we search for 

answers to understand our environment. These errors are difficult to guard against. Groopman 

(2007a) reported that physicians experience these kinds of errors when diagnosing illnesses, yet 

the physicians typically are unaware of their faulty thinking and the consequent errors in their 

assessments. I would suggest that this phenomenon also occurs with other analytical thinkers 

such as social scientists, historians, and intelligence officers (Groopman, 2007a, pp. 40, 147, 263). 

There are several opportunities where different kinds of cognitive bias can become introduced 

into lessons-learned analysis. I will briefly discuss six examples of cognitive error: categorical 

thinking bias, rationality assumption error, tribal moral community bias, cognitive dissonance 

bias, anchoring errors, and search satisfaction error.

Categorical Thinking Bias. Categtorical thinking bias is where the analyst distorts his 

or her ability to see differences and similarities between facts as a result of placing the facts 

in categories and paying too much attention to the categorical boundaries. When humans 

analyze complex subjects, we think categorically about the information and assign labels to the 

concepts in order to simplify and explain the phenomenon. For example, we arbitrarily organize 

the visual electromagnetic spectrum into arbitrary categories that we label as discrete colors. 

Yet the spectrum is continuous. Arbitrary divisions often can result in different conclusions. 

We could divide the following nine letters into three arbitrary categories: “Cho,” “Pho,” and 

“Use.” Alternatively, we arbitrarily could place them in two categories: “Chop” and “House.” To 

compensate for this potential bias, the analyst must avoid focusing on category boundaries and 

think broadly rather than categorically (Sapolsky, 2005).

Rationality Assumption Error. Rationality assumption error occurs when the analyst assumes 

a more rationale process than may be justified by the data. This easily can occur when the analyst 

attempts to reconstruct a decision-making process and looks for an orderly process with a univariate 

explanation. Typically it is not the case that there is a single, clear explanation for the phenomenon. 

To avoid this error, the analyst should turn to the systems model for help in identifying the range of 

multivariate factors that may be influencing the process (George & Bennett, 2005, p. 98). 
37

Tribal-Moral Community Bias. Tribal-moral community bias is a narrowing of the focus 

of research that results from being part of what Haidt (2011) called a “tribal-moral community 

[united by] sacred values.” Any profession (e.g., military, intelligence, engineering, legal, 

social science, and history) over time could come to embrace certain values that have a moral 

underpinning. The members then can come to ascribe to what has become the moral value of 

that professional “community.” The community could have come to subscribe to a particular 

philosophy, favor a particular political point of view, or adopt a particular set of religious values. 

The lessons-learned analyst often is a member of a variety of communities and is subject to the 

bias of any one or more of those multiple communities. If the researcher “circles around sacred 

values” of a particular community, the researcher might embrace specific research whenever 

it supports the “moral community’s” values, but ignore it whenever it threatens those sacred 

values. To mitigate against this bias, the researcher must assess the perspective of his or her 

own moral communities, make those often unknown “sacred values” explicit, and attempt to 

conduct research independent of those values—a difficult task (Haidt, 2011).

Cognitive Dissonance Bias. This is a bias where the analyst more readily accepts new 

information that is consistent with his or her existing mind-set and rejects data that are 

inconsistent with his or her existing mind-set. In this way, the analyst eliminates potential 

cognitive dissonance, i.e. avoids the discomfort of holding disparate points of view. As a result, 

the analyst will tend to reject information that challenges the existing mind-set and have a much 

higher threshold for accepting data that are consistent with existing assessments. To mitigate 

against this bias, the lessons-learned analyst needs to be open to revising initial assessments in 

the face of new data, even for what might appear to have been definitive explanations (George 

& Bennett, 2005, p. 99).

Anchoring Errors. Anchoring errors are what I see as a collection of similar errors where 

the analyst latches onto a single possibility and fails to consider the range of possibilities—i.e., 

“throws down an anchor” where the analyst believes it needs to be. These “anchoring” errors 

can contribute to serious flaws in analysis. I will cite five variations of anchoring errors: earlyfind anchor error, availability anchor error, confirmation anchor error, attribution anchor error, 

and affective anchor error (Groopman, 2007a, p. 65; Groopman, 2007b).

Early-Find Anchor Error. This is the case where the analyst anchors the initial 

data that he or she comes to see as the “definitive evidence.” Groopman (2007a) 

argued that these typically are the data that the analyst first encounters. In this 

error the analyst makes a quick judgment and “shoots from the hip.” In the 

medical field this is most common in hospital emergency departments. In 

the intelligence field it is most common in indications and warning (I&W) 

watch centers. Both are settings where the analyst has to make “life and death” 

assessments with short deadlines (Groopman, 2007a, p.75).

Availability Anchor Error. This is an anchoring error where the analyst has 

a tendency to make a judgment about the likelihood of an event by the ease 

with which relevant examples come to mind. For example, if a large number 
38

of satellite launches fail because of fuel contamination, the analyst is more 

likely to focus on fuel contamination as the key contributing factor in the 

next failure. An example from imagery analysis would be where the analyst 

who recently has located a series of terrorist camps subsequently interprets 

what he or she sees as indicators of a terrorist camp and stops analysis 

at that point, failing to consider other possibilities that might be better 

explanations. Availability anchoring error is particularly likely in cases 

where there are similarities in data, which can result in analysts ignoring 

key differences. For example, when a law enforcement officer investigates 

a teenage death, the investigator—who recently may have investigated a 

series of suicides might ignore the signs of a complicated accident or foul 

play that might have caused the death, (Groopman, 2007a, pp. 64-65, 188; 

Tversky & Kahneman, 1973, pp. 207-232).

Confirmation Anchor Error. This is an error where the analyst confirms what 

he or she expects to find by selectively accepting or ignoring information. 

Groopman called this “cherry-picking” data and described it as wishing for 

a certain outcome. An example from a military operation would be where 

a reconnaissance patrol looks at its map and studies the terrain only to see 

landmarks that it expects to see and neglects those that would correctly tell 

the unit that it is on the other side of the border. The faulty reading of the 

map “confirms” the mistaken assumption that the unit anchored (Groopman, 

2007a, p. 65; Tversky & Kahneman, 1973).

Attribution Anchor Error. Attribution anchoring error is similar to the 

confirmation anchoring error. However, in the case of attribution error, the 

analyst attributes the phenomenon to a similar stereotypical or representative 

case. Analysts typically rely on pattern recognition and draw on pattern 

stereotypes to make assessments. For example, when a patient is intoxicated, 

the physician could attribute a set of symptoms to cirrhosis of the liver when 

the actual cause could be a more unusual condition like Wilson’s disease that 

has symptoms that include clumsiness, difficulty speaking, and jaundice 

(Groopman, 2007a, pp. 45 ff).

Affective Anchor. Affective anchoring resembles confirmation anchoring in the 

selectivity of examing data. It is the tendency for the analyst to prefer what he 

or she hopes will happen over the less appealing alternatives. The analyst tends 

to lull him or herself into thinking that what he or she wishes for will occur 

when he or she gets the first suggestion, however fragmentary, that the wish 

will come true. It is driven by the expectation that the initial, more desirable 

hypothesis was correct (Groopman, 2007a, pp. 47, 65).

To mitigate against anchoring errors, the analyst needs to ask questions that would 

prompt the analyst to recognize the anchor and then move to other explanations—to move 

away from the most available and “preferred” explanation. The analyst should keep asking 
39

the question, “Is it possible there is more than one explanation?” This question should trigger 

consideration of multiple causes and assist the analyst in casting a wider net of thinking and 

asking additional questions not posed earlier in the analysis. One of the most effective ways to 

counter these errors is to generate a list of alternative explanations, even when there is a high 

degree of certainty of the hypothesis (Groopman, 2007a, p. 66).

Search Satisfaction Error. This error is where the analyst stops searching when he or she 

uncovers a major finding, even though there may be multiple causes or parallel causes. For 

example, a complex engineering project might have failed because of insufficient funding but 

the absence of experienced engineers also could have been a major contributor to the failure. 

This is an easy trap for the analyst because there is a cognitive tendency to stop searching 

after there is a major finding. However, that finding might not be the only explanation. An 

example from everyday life is when you are rushing out the door and can’t find your wallet. 

You search the various places and find it on your desk. You then rush out the door and go to 

your car only to find you don’t have your keys—there was more than one item to find. Like 

with the anchoring error, a helpful way to avoid the search satisfaction error is to keep asking 

the question, “Is there something more?” (Groopman, 2007a, pp. 169-70, 185) 

Causes of Cognitive Error

Cognitive errors are caused by faulty thinking influenced by a variety of factors. These 

factors may include inner thoughts or feelings that analysts typically don’t recognize as affecting 

their thinking. There can be negative feelings about the problem under study or the people 

involved in the problem. There can be a fascination with some part of the problem that diverts 

the analyst’s attention. There can be events and things in the environment that channel thinking 

in a particular direction, e.g., in medicine the literature has evidence that internists give less 

attention to and are less thorough with patients believed to have a psychological disorder. 

Urgency in completing analysis is another significant factor that can induce cognitive error. 

Haste in the analysis makes the analyst vulnerable to the whole range of cognitive errors and 

increases the likelihood of those errors (Groopman, 2007a, pp. 39-40, 88).
40

Avoiding Cognitive Error

There is no single script or methology to avoid cognitive error. The researcher and analyst 

must be on constant guard against these errors. There are a series of steps or check lists that 

you, as an analyst, can use to help guard against the errors. 

First, consistently question your work:

Question the precision of your language: Are you accurately describing the 

data and your analysis? 

Question your data collection: Should you collect additional data?

Question your analysis: What else could it be; how else can it be explained? 

Even ask the question, “What is the least likely explanation?” 

Second, slow down your observation of data and its analysis. This offers you 

an opportunity to exam the full range of data—not just data that support 

the hypotheses. It also affords you an opportunity to identify a wider range 

of factors that could influence the data. In other words, it is important 

to look beyond a narrow data set or the initial analysis, e.g., if you are 

trying to explain the failure of a flight system and focus only on failed 

electrical components as the causative factor, you might not see that the 

electrical problem could have been caused by nearby mechanical pressure 

on the electronics—or the electrical failure could have been caused by a 

temperature differential—or it could have been caused by radiation—or it 

could have been caused by a combination of those factors, or yet something 

else (Groopman, 2007a, p.182).

Third, generate a list of alternative explanations after you have generated a list of 

alternative explanations. In other words, do it more than once. You should 

do this even when you are convinced that you have strong supporting data 

and good analysis. The obvious explanation of the immediate event may 

not reveal the underlying cause. A medical example can best illustrate this 

point. If a patient complains of pain in the ankle, and the physician—with 

the benefit of a clinical exam and x-rays—concludes the cause is a sprained 

ankle, there may be more to the case. The physician could have ignored the 

contributing and more fundamental causes, such as the patient acquired 

the sprained ankle because of a fall that in turn was caused by anemia (or a 

transient ischemic attack, a TIA, or an irregular heart beat that reduced the 

blood flow to the brain) (Groopman, 2007a, p. 66).
41

Table 5: Questions to Minimize Analytical Risks From Cognitive Errors

Cognitive errors typically operate below the level of conscious thinking, and it is important 

to consciously go through this kind of checklist. Attempting to correct potential errors helps 

the analyst think of an explanation that he or she didn’t previously consider. Doing so will 

bring the errors closer to the surface of thinking and reduce the uncertainty of analysis. Table 

5 is a guide dealing with cognitive errors. It lists the six cognitive errors I discussed in this 

section, explains the errors, and offers a series of questions that you can use to help minimize 

the risks associated with cognitive errors. 

Cognitive 

Error

Explanation of

Error Questions to Minimize Analytical Risk

Categorical 

Thinking Bias

Paying too much attention to 

categorical boundaries

How can I consider this fact or element of the case outside of 

its boundary?

What is a broader category in which to place this fact or 

element of the case? 

Rationality 

Assumption Error

Assuming a simplistic more 

rationale process 

What other multivariate factors might influence the case and 

its processes?

What random, unexpected factors might be playing a role in 

the processes of the case?

What aspects of components in the system have you not 

considered?

Tribal-Moral 

Community Bias

Narrowing research focus as a 

result of being a member of a 

“community”

What is the perspective of your particular tribe or 

community?

What are the “sacred values” of your tribe or community?

How can you expand your research beyond your “tribe’s” 

perspective and values?

Cognitive 

Dissonance Bias

More readily accepting data 

that is more consistent with 

your existing mind-set

How do the new data challenge my original mind set?

How can I revise my initial assessment based on new, 

sometimes inconsistent, data?

Anchoring Errors

Latching onto a single 

possibility and failing to 

consider a range of possibilities

Are there any data that I have overlooked in identifying 

lessons?

Are the lessons more complex and multivariate?

Are there data that contradict the working hypothesis?

Is there really a different lesson here?

Search 

Satisfaction Error

Stopping the research when 

you uncover a major finding

Is there something more to this case?

What other explanations are there?
42

Conclusion

“A wealthy merchant bought many baskets of salt and tied them on his donkey 

. . . The donkey slipped beneath his heavy load and fell into the water. . . .

[H]alf the salt . . . washed away. . . . The baskets were only half as heavy. . . . The 

merchant sold . . . the salt and bought several bags of sponges. . . . When they came 

to the river, the donkey . . . pretended to slip and rolled into the water, hoping his 

load would be lightened. But the sponges . . . soaked up water until they swelled fat 

and heavy. . . . The beast staggered home under a load secondary.”

—From “The Donkey and Load of Salt,” Aesop’s Fables

Fundamental to the lessons-learned analytical process is identifying the lessons. Equally 

important, however, is knowing how and when to apply the lessons to ensure a particular 

outcome. In the fable, the donkey thought that he learned a useful lesson about how to 

reduce the weight of his load. He applied it with eagerness only to find that the lesson was not 

applicable to the new situation. In fact, when he applied the lesson, he made his situation worse. 

Failing to correctly learn or apply lessons can be fatal not only for our donkey, but 

also for organizations. Kodak’s experience is one example. During the 20th century Kodak 

was the dominant provider of film to both the public (for its Brownie, Pentax, and other 

home cameras) and to the Intelligence Community (for NRO’s classified film-return satellite 

reconnaissance cameras like Corona and Hexagon). As the century came to a close there 

was a revolution in photography with the emergence of transformational digital technology, 

but Kodak’s organizational behavior hindered a fast response to this new technology that 

changed the process of capturing and sharing images. It was applying the wrong lessons. 

Lucas and Goh (2009) reported that as a result Kodak suffered an almost 80% decline in its 

workforce, lost market share, and suffered falling stock prices (Ehrenberg, 2012; Lucas & 

Goh, 2009; McDonald, 1995).

The Kodak example demonstrates a principle for all organizations, which by definition 

are open systems. If they are to survive, they must acquire what some would describe as 

negative entropy to counter the law of entropy. This is the law of nature that states all systems 

move toward disorganization or death. Biological organisms run down, disintegrate, and die; 

sociological organizations also have a natural tendency to become disorganized and dissipate 

themselves. To counter this, the open system must acquire more energy from its environment 

than it expends as part of its transformation process. Learning and incorporating lessons into 

the organizational system brings energy to the system. Unless the organization can continually 

import and store more energy than it expends (thereby acquiring negative entropy), it will 

disintegrate and fail in its organizational mission. Applying lessons-learned can bring this new 

energy into the system (Katz & Kahn, 1966, p. 21).
43

Each lessons-learned experience with its new energy is unique, often with unknown causes 

and embedded secondary lessons not readily apparent. The challenge is to collect data, do 

an analysis, and develop a series of hypotheses that could be lessons. The goal is to identify 

a contingency statement to serve as the lesson in the form of, “When the player(s) exercise a 

particular behavior, it increases the probability of a particular outcome.”

Identifying and applying lessons is an ongoing challenge for all individuals and 

organizations. The IC, as with most organizations, has a long record of both meeting its 

goals and falling short of its goals. And it is through learning from these experiences that 

an organization maintains and improves itself. Evidence in the literature supports the view 

that expertise is largely acquired through experience, but there also is evidence that receiving 

feedback through lessons helps individuals and organizations understand technical errors and 

misguided decisions. The challenge for individuals and organizations is to recognize when they 

meet their goals, and celebrate it. While at the same time, identify those experiences where they 

miss their goals, and admit it. In both cases, they should analyze the experiences with a view 

to learn lessons to sustain excellence and improve performance (Groopman, 2007a, postscript).

Identifying lessons and sharing them across the Intelligence Community has the potential 

to improve performance for the entire Community. Pfeffer and Salanick (1978, p. 74) have 

pointed out that when lessons are learned and shared within an organization, it focuses 

organizational energy, attention, and demand on the identified lessons so the organization can 

improve its capabilities. This kind of sharing lessons makes it more likely that those lessons will 

be learned and applied in future decision making. An aggressive lessons-learned program can 

give managers the tools to improve performance (Van De Ven & Ferry, 1980, p. 3). 

I offer the methodology in this monograph as a social-science based approach to gain an 

understanding of ways to improve on organizational efficiency and output—ways that can 

identify lessons that offer insight on how to sustain and improve organizational performance. 

The methodology is applicable to a wide range of organizations, both within and outside 

the IC. The methodology incorporates the best insight from the case-study framework as a 

foundation for the integration of the general systems paradigm with congruence theory. It can 

offer insights to reinforce highly productive behaviors associated with satisfying performance, 

as well as offer insights to modify—in a positive direction—less productive behaviors associated 

with the more frustrating experiences.
44

References

Almond, G. (1996). The Civic Culture: Prehistory, Retrospect, and Prospect. Center for the 

Study of Democracy Working Paper, University of California, Irvine. Retrieved May 4, 2014, from 

http://escholarship.org/uc/item/4mm1285j?query=Prehistory,%20 Retrospect,%20 and%20

Prospect 

Almond, G. & Verba, S. (1963). The Civic Culture: Political Attitudes and Democracy in Five Nations. 

Princeton: Princeton University Press. 

Bergoglio, J. M. & Skorka, A. (2013). On Heaven and Earth. New York: Image.

Blockbuster. (n.d.). Company Overview. Retrieved August 10, 2013, from: http://www.blockbuster.

com/corporate/aboutBlockbuster

Boulding, Kenneth E. (2013). Key Text: General Systems Theory: The Skeleton of Science. In 

Interdisciplinary Economics—Kenneth E. Boulding’s Engagement in the Sciences, Dolfsma, W. & 

Kesting, S. (Eds). 2013. Milton Park, Abingdon, Oxon, UK: Routledge.[Article previously 

published in Management Science, 1956, Vol 2(3) pp 197-208.]

Bromely, D. B. (1986). The Case-Study Method in Psychology and Related Disciplines. New York: John 

Wiley & Sons.

Buckley, W. (1967). Sociology and Modern Systems Theory. Englewood Cliffs, NJ: Prentice-Hall, Inc. 

Commission on the Intelligence Capabilities of the United States Regarding Weapons of 

Mass Destruction (Commission on WMD). (2005). Report to the President. Retrieved June 

28, 2008, from http://govinfo.library.unt.edu/wmd/report/wmd_report.pdf [President 

George W. Bush established the Commission by Executive Order 13328, The Commission 

presented its report to the President on 31 March 2005.]

D’Aulnoy,M.C.B. (2007). The Story of Goldilocks and the Three Bears. Retrieved April 11, 2007, from 

http://www.dltk-teach.com/rhymes/goldilocks_story.htm [Originally published in 1892 in 

The Fairy Tales of Madame D’Avlnoy, translated by Annie MacDonell and “Miss Lee.”]

Eckstein, H. (1998). Congruence Theory Explained. In Eckstein, H., Fleron Jr. F., Hoffmann, E., 

& Resisinger, W. with Ahl, R., Bova, R. & Roeder, P., Can Democracy take root in Post-Soviet 

Russia? Explorations in State-Society Relationships. New York: Rowman Littlefield Publishers, Inc. 

Eckstein, H. (1997). Congruence Theory Explained. Center for the Study of Democracy Paper 97-

07, University of California, Irvine [Posted at the eScholarship Repository, University of 

California, http://repositories.cdlib.org/csd/97-07]

Ehrenberg, R. (2012). The Digital Camera Revolution. Science News, 181(2), 22.

Gage, N. L. (1991). “The Obviousness of Social and Educational Research Results.” Educational 

Researcher, 20, 10-16.

Gandel, S. (2010, October 17). How Blockbuster Failed at Failing. Time, Retrieved August 10, 

2013, from http://www.time.com/time/magazine/article/0,9171,2022624,00.html
45

George, A. L, & Bennett, A. (2005). Case Studies and Theory Development in the Social Sciences. 

Cambridge, MA: MIT Press.

Godlas. A. (2008). Surfism’s Many Paths. Retrieved June 28, 2008, from http://www.uga.edu/

islam/Sufism.html, 28 June 2008.

Good Helpmates. (2009). Italian Folktales. Retrieved April 11, 2009, from http://oaks.nvg.org/

italian-tales1-3.html [Gold Scales web site hosted by Norway’s Technical-Natural-Scientific 

University.].

Groopman, J. (2007a). How Doctors Think. Boston: Houghton Mifflin.

Groopman, J. (2007b). Interview on PBS NewsHour with Jim Lehrer, PBS, 15 May 2007. Retrieved 

May 16, 2007, from http://www.pbs.org/newshour/health/ [The Online Newshour.]

Groopman, J. (2007c, January 29). What’s the Trouble. The New Yorker. 29 January. 

Retrieved May 16, 2007, from http://www.newyorker.com/ 

reporting/2007/01/29/070129fa_fact_groopman?printable=true [The New Yorker web 

page.] 

Heywood, D., & Stronach, I. (2005). Philosophy and Hermeneutics. In B. Somekh and C. Lewin, 

Research Methods in the Social Sciences. London: Sage Publications.

Haidt, Jonathan (2011). The Bright Future of Post-Partisan Social Psychology, 2020: Visions 

for the Next Decade of Social and Personality Psychology. Opening Session and Presidential 

Symposium, 12th Annual Meeting, Society for Personality and Social Psychology, San Antonio, TX, 

27-29 January 2011. [Talk given at the 2011 meeting. See conference program at http://c.

ymcdn.com/sites/www.spsp.org/resource/resmgr/conference_programs/spsp2011_program.

pdf?hhSearchTerms=%22Post-Partisan+and+Social+and+Psychology%22,transcript of talk 

retrieved May 4, 2014, from http://edge.org/conversation/the-bright-future-of-post-partisansocial-psychology.].

History Channel Web Page. (2013). Oct 19, 1985: First Blockbuster Store Opens. Retrieved August 

27, 2013, from http://www.history.com/this-day-in-history/first-blockbuster-store-opens

Katz, D. & Kahn, R. L. (1966). The Social Psychology of Organizations. New York: John Wiley & 

Sons, Inc.

Kelley, H. H. (1992). Common-sense Psychology and Scientific Psychology. Annual Review of 

Psychology, 43, 1-24.

King, G., Keohane, R. O., & Verba, S. (1994). Designing Social Inquiry: Scientific Interference in 

Qualitative Research. Princeton, NJ: Princeton University Press.

Kodak. (n. d.). Information about Kodak’s Reorganization. Retrieved August 10, 2013, from http://

www.kodak.com/ek/U.S./en/Kodak_Transforms/Home.htm

Levinson, H., with Molinari, J., & Spohn, A. (1972). Organizational Diagnosis. Cambridge, MA: 

Harvard University Press.
46

Lilienfeld, S. O. (2012). Public Skepticism of Psychology. American Psychologist, 67(2), 111-129.

Lucas, H. C., Jr. & Goh, J. M. (2009). Disruptive Technology: How Kodak Missed the Digital 

Photography Revolution. The Journal of Strategic Information Systems, 18, 46–55.

McDonald, R. A. (1995). Corona: Success For Space Reconnaissance—A Look Into the Cold War, 

and A Revolution For Intelligence. Photogrammetric Engineering & Remote Sensing, 61(6).

Myers, D. (1994). Exploring Social Psychology. New York, NY: McGraw-Hill.

National Commission on Terrorist Attacks upon the United States (9/11 Commission). (2004). 

The 9/11 Commission Report: Final Report of the National Commission on Terrorist Attacks upon the 

United States. New York: Norton. [Also available from the Superintendent of Documents, U.S. 

Government Printing Office, online via GPO Access: http://www.gpo.gov/fdsys/pkg/GPO911REPORT/pdf/GPO-911REPORT.pdf, accessed May 4, 2014. Commission established 

under Public Law 107-306, November 27, 2002; final report released 22 July 2004.]

Nadler, D. A. & Tushman, M. L. (1989). A Model for Diagnosing Organizational Behavior: 

Applying a Congruence Perspective. In M. L. Tushman, R. O’Reilly, and D. A. Nadler (Eds.), 

The Management of Organizations: Strategies, Tactics, Analyses (pp. 91-106). New York: Harper & 

Row, Publishers (Ballinger Publishing Company).

Nordregio. (1999). Theme 1.2. Economic Strength, Strand 1: Spatial Criteria and their Indicators, 

Study Programme on European Spatial Planning. Retrieved April 30, 2007, from http://www.

nordregio.se/ [Published by Nordregio, a European Research Centre in the field of Spatial 

Development, established by the Nordic Council of Ministers.]

Perrault, Charles (2009). Little Red Riding Hood. Retrieved April 11, 2009, from http://www.pitt.

edu/~dash/type0333.html#perrault [University of Pittsburg web site.]

Pfeffer, J. & Salanick, G. R. (1978). The External Control of Organizations: A Resource Dependence 

Perspective. New York: Harper and Row.

Pinkney, J. (2000). Aesop’s Fables. New York: Seastar Books.

Russian Crafts. (2009). Nesting Dolls. Retrieved May 24, 2009, from http://russian-crafts.com/

nesting-dolls/history.html [Russian-crafts.com web page]

Sapolsky, R. M. (2005). Lecture 1: Biology and Behavior—An Introduction. In Biology and Human 

Behavior: the Neurological Origins of Individuality, [CD] Chantilly, VA: The Teaching Company.

Sinek, S. (2010, February 11). Why Great Companies Fail. Forbes. Retrieved August 10, 2013, 

from http://www.forbes.com/2010/02/10/microsoft-apple-google-facebook-why-companiesfail-cmo-network-simon-sinek.html Stanovich, K. (2009). How to Think Straight About Psychology 

(9th ed.). Boston, MA: Pearson.

Sutherland, J. W. (1973). A General Systems Philosophy for the Social and Behavioral Sciences. New 

York: George Braziller.
47

The Countess Kathleen O’Shea (2009). Retrieved April 11, 2009, from http://www.sacred-texts.

com/neu/yeats/fip/fip66.htm [Irish folklore on the Internet Sacred Text Archive web site, 

an online archive of writings about religion, mythology, and folklore.]

The Swan Queen (2009). Fairy-Tales web site. Retrieved April 11, 2009, from http://www. 

fairy-tales.parnas.info/en/lithunian/swan-queen.html [A Lithuanian Folk Tale.]

Tversky, A. & Kahneman, D. (1973). Availability: A Heuristic for Judging Frequencyand 

Probabilty. Cognitive Psychology, 5, 207-232.

University of Maryland Libraries (UMD). (2012). Guides to Information Resources. 

Retrieved April 9, 2012, from http://www.lib.umd.edu/guides/primary-sources.html

U.S. Congress, Joint Committee on the Investigation of the Pearl Harbor Attack (1946). 

Investigation of the Pearl Harbor Attack: Report of the Joint Committee on the Investigation of the 

Pearl Harbor Attack Congress of the United States and Additional Views of Mr. Keefe Together 

Minority views of Mr. Ferguson and Mr. Brewster. Washington, DC: US Government 

Printing Office. [79th Congress, Senate Document, 2d Session No. 244; pursuant 

to S. Con Res. 27, A Concurrent Resolution Authorizing an Investigation of the Attack on 

Pearl Harbor on December 7, 1941, and Event and Circumstances Relating Thereto.]

U. S. General Accounting Office (U.S. GAO). (1991). The Chief Financial Officers Act: 

A Mandate for Federal Financial Management Reform. Washington, DC: U.S. 

General Accounting Office.

Van De Ven, A. H. & Ferry, D. L. (1980). Measuring and Assessing Organizations. New York: John 

Wiley & Sons.

von Bertalanffy, L. (1968). General System Theory: Foundations, Development, Applications. New York: 

Braziller [Also see L. Von Bertalanffy, General System Theory: A New Approach to Unity of 

Science, Human Biology, 23, 303-361.]
48

Appendix A—Research Design Template 

(With Questions to Consider)

Overview of Appendix A

This Appendix provides one example of a template for drafting a concise, but focused 

research design. While you can find a variety of examples for drafting a research design in 

the social science literature, this example, with its associated questions, should be helpful in 

ensuring that you address six areas that are critical to consider when you design your lessonslearned research: 1) statement of the research problem; 2) definition of terms/concepts; 3) scope 

and limitations of the research, including identification of your selected case study; 4) how you 

will apply a disciplined lessons-learned methodology to your case; 5) research schedule; and, 6) 

spend plan and needed resources. For each of these topics in the template you will find a series 

of questions to consider when you draft that section of the research design.

Research Design for [Insert Subject of Research]

Statement of the Problem (This is the most important part of the research design. It defines 

the focus of the research and suggests the steps to find the results.)

• What is your main research question?

• What are the sub problems and associated research questions? 

• What is your working hypothesis, if you have one?

Definition of Terms (This step makes explicit the concepts you will be addressing)

• What are the terms that have a special meaning for this research?

• How will you be defining those terms?

Scope & Limitations of the Research (This step bounds your research)

• How are you defining the case you will be studying?

• How will ethical, legal, security, and other limitations constrain the study?

• What kinds of constraints do time, geographical limitations, and resources 

present?

• What are the conceptual limits of the research?

• What is the form and scope of the final product (i.e., is the final product 

a written journal article, monograph, or book; is it an oral presentation of 

specific length; is it a multimedia production of a particular form and length)?
49

Methodology (This step explains how you will do your research)

• Will this be an exploratory study, descriptive study, comparative study, single 

case study, or have some other form?

• What is your strategy for data collection?

• What approach to data collection will you use (observational, interviews, 

surveys, documentary research)?

• To what extent, and how will you be using a historical method, quantitative 

analysis, qualitative analysis, general systems theory, congruence analysis, the 

experimental method, and other methodologies?

Schedule (This step explains the allocation of time for the research and identifies target dates 

for completion of the various parts of the research.)

• How long will this research project take?

• How much time will you be spending on each phase of the research project 

(e. g., data collection, analysis, writing, and reviews/revisions)?

Resources (This step identifies any needed resources for the research)

• What will it cost to conduct the research, and what is your source of funding?

• How will you allocate your funding resources?

Attachments (Through the use of attachments you have an opportunity to provide some detail 

about your research plan)

• Preliminary outline for the final product

• Tentative list of potential interview subjects

• List of potential sources for documentary research

• Preliminary list of the kinds of peer reviewers
50

Appendix B—Data Collection Framework and 

Examples of Questions1

Overview of Appendix B

This appendix outlines a general data collection framework with specific examples of the 

kinds of questions that might be helpful for a political science case study related to national 

security. For other kinds of case studies, you might want to develop a different set of questions. 

I have organized the questions according to the components of the lessons-learned systems 

model (See Figure 2 on page 14). There is a separate section in the framework for each of the 

four system components of the model (See Table 1 on page 12). 

While the specific questions in this example might be most appropriate for studying 

national security organizations, the questions serves to illustrate the scope of data that should 

be collected in order to be able to describe and explain each component of the model. In 

developing your specific list of questions, you should ensure that: (a) your questions consider 

each component of the lessons-learned systems model, (b) they are relevant to your basic 

research question, and (c) the listing is comprehensive enough to enable you to adequately 

describe each system component. After data collection, you can use your list of questions as a 

checklist to ensure that you collected ample, relevant data for each component in the model. 

The answers to your data collection questions can then serve as the required input for writing 

your description of the case study. 

Section 1—Environment Components

Societal Domain 

(The Human Ecology)

Geopolitical

1. What is the focus of U.S. National Security Policy during the period 

under study?

2. What geo-spatial factors are impacting U.S. national security?

3. What is the focus of U.S. National Intelligence Strategy?

4. What is the focus of U.S. military strategy?

1 References: Levinson, Molinari, & Spohn, 1972; Nadler & Tushman, 1989; Nordregio, 1999; Van De Ven 

& Ferry, 1980. 
51

5. What are the major national security threats to the U.S.?

6. What are the major U.S. military operations?

Domestic Political

1. What political party controls the Executive Branch, and what are the 

administration’s philosophy and goals?

2. What political party controls the Legislative Branch, and what are 

Congress’s philosophy and goals?

3. What role is the Judicial Branch playing, and what judicial philosophy does 

it exhibit?

4. What role is the press playing in matters related to the organization’s 

mission?

5. What pressures and influences are the Administration, Congress, and the 

Press placing on the organization and its leadership?

Economic

1. What is the strength of the economy (productivity, sectoral structure, future 

orientation, wealth, and labor market)2?

2. What is the economic growth rate?

3. What is the state of the Federal Budget (e.g., size, deficit, taxation, 

expenditure trend)?

4. What is the relative strength and direction of the Intelligence Community 

budget?3

5. What are the funding priorities for intelligence?

6. What is the relative distribution of funding in the Intelligence Community 

budget?

2 Productivity as GDP/employee; sectoral structure in terms of proportion in agriculture, industry, etc; future 

orientation in terms of proportion of research and development employment; wealth in terms of GDP per 

head; and labor market in terms of unemployment rate.

3 Relative in comparison with other national security and budget sectors.
52

Other Societal Factors

1. What other kinds of factors are part of the environment’s societal domain 

for this case?

2. What are the impact and implications of these factors?

Associational and Relational Domain 

(The External Alliances and Associations—

The CUPS: Customers, Users, Providers, & Stakeholders)

Dependencies

1. What is the policy coordination dependency for the organization?

2. What is the technological dependency?

3. What is the fiscal dependency?

4. What is the supplier dependency?

Communications4

1. How formal are communications with the CUPS? 

2. What is the typical mode of communication?

3. What is the frequency of communication?

4. What is the quality and content of communication?

4 See the “Information Network” domain section for questions related to organization’s internal information 

network.
53

Limitative and Restrictive Domain

(The Structural and Behavioral Constraints)

Legal & Regulatory Constraints

1. Under what legal authority does the agency head and program manager 

direct the agency and project?

2. What regulatory regime controls how the agency head and program 

manager direct the agency and project? 

3. What constraints do these laws and regulations impose on the agency head 

and program manager?

External Oversight

1. What kinds of oversight are there from the Executive Branch (e.g., ODNI, 

USD/I, and OMB)?

2. What kinds of oversight are there from Congress?

3. What kinds of oversight are there from independent bodies (e.g., PFIAB, 

national commissions)?

4. What kind of attention is the press giving the organization and/or task?

Fiscal Constraints

1. Who has fiscal authority to manage the funding for the project?

2. How much flexibility is there in the funding (e.g., what authorities does the 

program manager have to change to the specifics of the funding within the 

overall funding ceiling)?

Technical Constraints

1. What are the technical constraints imposed by the limits of engineering and 

scientific knowledge?

2. What are the technical constraints that result from human performance 

limitations?
54

4. What are the technical constraints that result from policy and funding? 

5. What are the limits and implications of the technical constraints?

Other Limitative Factors

1. What other kinds of constraints are part of the case’s limitative domain?

2. What are the limits and consequences of those constraints? 

Historical Domain

(The Historical Context)

Societal

1. What was the geopolitical history during the origin of the 

organization and at the time of task input?

2. What was the domestic political history?

3. What was the scientific and technological history?

Organizational

1. What was the origin of the organization?

2. What is the organization’s performance history?

3. What major crises did the organization experience?

4. What is the organizational folklore?
55

Section 2—Input Components

The Requirement

(The Need that is to be Satisfied by the Process)

Problem

1. What is a specific, detailed explanation of the problem or problems 

to be addressed by the task given to or assumed by the organization?

2. What issue or issues are expected to be resolved by the task given to 

or assumed by the organization?

Objective

1. What is the deliverable that is expected to satisfy the need?

2. What is the desired end-state?

3. What is (are) the indicator(s) of success?

Complexity

1. How difficult is it to separate the different parts of the problem and/or 

deliverable(s)?

2. How many interest groups are stakeholders?

3. What are the conditions that could hinder reaching the objective?

4. What kind of constraints are associated with meeting the requirement?

5. What is the degree of uncertainty associated with satisfying the requirement?
56

Funding for the Task 

(Fiscal Resources Dedicated to Satisfying the Requirement)

Amount of Funding

1. How much funding does the budget allocate to the activity under study?

2. Who has execution authority?

Nature of Funding

1. To what degree does the funding match the requirement?

2. Who has execution authority?

Staffing for the Task 

(Human Resources Dedicated to Satisfying the Requirement)

Level of Staffing

1. How many government and contractor personnel are authorized for the 

project?

2. What is the vacancy rate for the authorized positions?

Nature of Staffing

1. What is the skill mix defined for the staffing?

2. What is the proportion of government and contractor personnel authorized 

for the project?
57

Section 3—Process Components

Formal Organization 

(The Formal Arrangements between Components

of the Organization)

Demographics of Organization

1. What kind or type of organization is under study in the case?

2. What is the size of the organization? 

3. What is the age of the organization, or how long did the organization exist?

4. What is the growth stage of the organization?

Design of the Organization

1. How is the organization structured (vertical, horizontal, etc.)?

2. What is the form of the sub-components (by function, program, matrix, 

location, etc.)?

3. What is the vertical differentiation (i.e., the number of supervisory levels)?

4. What is the horizontal differentiation (i.e., the number of subunits)? 

5. What is the mix of government and contractor employees?

6. What is the distribution of authority? 

Administrative Intensity

1. What is the supervisor to staff ratio?

2. What is the top manager’s span of control (include non-supervisory

personnel)?

3. How extensive is middle management?
58

Bureaucratic Constraints Within the Organization

1. What are the organizational affiliations and allegiances?

2. What are the internal HR constraints?

3. What are the internal fiscal constraints?

Work of Organization

1. What is the nature of the organization’s work environment?

2. Where is the work done in relation to the organizational components?

3. What is the complexity of the work (i.e., number of different products/

services, territories of operation)?

Informal Organization 

(The Informal Arrangements Within and

Extending Beyond the Organization)

Relationships

1. What is the degree of awareness that the organizations’ components have 

with each other?

2. What are the intramural relationships associated with this awareness?

3. To what extent are there informal extramural arrangements that extend 

beyond the formal structure of the organization?

4. What are these extramural relationships?

5. What is the intensity of the intramural and extramural informal 

relationships? 

6. What is the impact of turnover rates on these informal relationships?
59

 Culture

1. What is the organizational ideology?

2. How is time regarded (e.g., value as an investable commodity; orientation 

to past, present, and future)?

3. What are the customs, taboos, and prohibitions?

4. What are the organizational symbols and slogans?

Influences

1. What is the distribution of power (i.e., ability of “A” to get “B” to change)?

2. What are the influence patterns?

3. How much discretion do individuals have (i.e., the discretion individuals 

can exercise in making task-related decision)?

4. What is the organizations time span and rhythm (planning cycles, degree 

activities regulated by time)?

Dependencies

1. What are the internal dependencies?

2. What are the external dependencies?

Certainty

1. How much agreement is there on goals and priorities?

2. How much clarity is there in how to respond to events?
60

Network of Individuals

(The Organization’s Collection of Individuals)

Demographics

1. What is the gender ratio of the individuals?

2. What is the age range and typical age of the individuals?

3. What is the cultural composition of the individuals?

Education

1. What formal education do individuals have?

2. What informal education do individuals have?

Skills

1. What are the skills directly related to the task?

2. What are the skills associated with related tasks?

Experience

1. What is the typical tenure of individuals?

2. What experience do individuals have in the disciplines of national security 

(e.g., national reconnaissance, intelligence, and diplomacy)?

3. What experience do individuals have that is directly related to the task?

4. What experience do individuals have in related industrial sectors?
61

Style

1. What behavioral and attitudinal preferences do the individuals have?

2. Are individuals task-oriented or relationship-oriented?

3. Are individuals introverts or extroverts?

4. What communication style do the individuals exhibit?

5. How do individuals resolve conflict?

6. What behavioral patterns do individuals exhibit?

Affect and Motivation

1. What are the expectations of the individuals?

2. What are the needs of the individuals?

3. What rewards do the individuals seek?

Information Network

(The Arrangements within the Organization 

for Communicating Information)

Formalization

1. What is the degree to which rules govern information exchange between 

individuals within the organization?

2. What is the degree to which rules govern information exchange between 

groups?

3. To what degree is there centralized control of information sharing?

4. How spontaneous are the communications?
62

 Mode

1. What is the range of information sharing modes (i.e., face-to-face 

conversations, telephone, memos, email)?

2. What is the distribution of information sharing across the modes?

3. How do meetings compare with individual conversations as modes?

4. What kinds of internal employee publications (such as newsletters) are there?

Frequency

1. What is the frequency of personal communications?

2. What is the frequency of group communications?

3. What is the frequency of organizational component communications?

4. What is the timing, rhythm, and urgency of communications?

Quality & Content

1. How much communication is task-related?

2. How much communication is socially oriented?

3. How difficult is it to make contact with people?

4. How often do communications get lost?

5. How useful do individuals find communications?
63

The Task 

(The Organization’s Work Activity 

Directed toward Satisfying the Requirement)

Nature of Task

1. What kind of task is involved?

2. How much time is required for the task?

3. What is the task variability (the number of exceptions encountered 

in the course of the work)?

4. What is the degree of uncertainty associated with the task?

Difficulty of Task

1. What is the analyzability of the task (i.e., the ease and clarity of knowing 

the nature and order of what needs to be done)?

2. What is the predictability of the task (i.e., the ease with which individuals 

can determine the outcome of a particular sequence in advance)?

Expertise for Task

1. What skills are required for the task?

2. What specialization does the task require?
64

Section 4—Outcome Components

The Deliverable(s)

(The Product of the Organization’s Work—

—What the Process Produced While Trying to Satisfy the Requirement)

Nature of the Deliverable

1. What is the deliverable?

2. What kind of deliverable (e.g., hardware, software, information, intellectual 

property, an organization, a building, a system)?

3. What is the value of the deliverable?

4. What is the expected lifespan of the deliverable?

Utility of the Deliverable

1. What are the functions and capabilities of the deliverable? 

2. How efficient is the deliverable?

3. What are the applications and uses for the deliverable?

4. How adaptable is the deliverable?

Process Change 

(How the System and Its Environment 

Changed During the Process)

Internal

1. What are the changes that have taken place within the system?

2. What kinds of changes are these?

3. What is the scope of these changes?

4. How different is the system at the time of output compared with the time 

of input?

5. What is the likely impact of these changes?
65

External

1. What are the changes that have taken place within the environment?

2. What kinds of changes are these?

3. What is the scope of these changes?

4. How different is the environment at the time of output compared with the 

time of input?

5. What is the likely impact of these changes?
66

Appendix C—Interview Protocol

Overview of Appendix C

This appendix offers a protocol for conducting social science research interviews. 

I organized the appendix into three annexes:

Annex 1 – Types of Interviews. 

In this annex you will find a table that lists and explains the three types of 

interviews. In planning and conducting an interview, it is important to know 

the objective so you can research and structure your questions accordingly.

Annex 2 – Checklist for Conducting Social Science Research Interviews.

In this annex you will find a table that lists the various factors you need to 

consider as you prepare for and conduct the interview. For each factor you will 

find a checklist of the tasks you need to address as you deal with each factor. 

The checklist in the table makes reference to three documents you should 

prepare as part of the interview process. You will find attached to this annex 

sample templates for these three documents.

Attachment 1 – Sample Template for Pre-Interview “Introduction Letter”

Attachment 2 – Sample Interview Consent Form

Attachment 3 – Sample Template for Post Interview “Thank You Letter”

Annex 3 – Questions to Consider and Avoid. 

In this annex you will find a table that lists the types of questions you should 

consider using in your interview and the types of question you generally 

should avoid using during your interview session.

***
67

Annex 1 – Types of Interviews

Interview 

Type Explanation

Fact-Based Interview

Objective To document facts related to an event, activity, or experience—either at one point in time or over 

a period of time.

Examples Oral history interview, investigative interview, medical history interview.

Special 

Considerations

Tend to be unstructured interviews and conversational. As the interviewee responds, be sure 

to ask the interviewee to specify dates and locations related to the events, activities, and 

experiences. When the interviewee names other players, get the full names (and spelling, 

especially if unusual) and enough information to understand the contextual relationship of the 

named players to your discussion.

Opinion-Based Interview

Objective To document the interviewee’s opinion or opinions about a particular subject.

Examples Seeking opinions on a topic or question, eliciting interpretations of policy.

Special 

Considerations

This should be a more structured interview with consistent administration across interviewees. 

The questions tend to be closed-ended.

Mixed Fact-Based Opinion Interview

Objective To document the facts about an experience and the lessons the interviewee believes he or she 

learned from that experience.

Examples Lessons-learned interview

Special 

Considerations

Tend to be semi-structured interviews and conversational. If a lessons-learned interview, try to 

draw out lessons, rather than explanations of what happened. Attempt to defer discussions of 

the interviewee’s assessment and explanations of what happened until the end of the interview. 

In general, question the interviewee about the facts associated with the experience in order to 

document the context for the opinions. Frame questions to solicit the interviewee’s assessment 

and opinion about the topic under discussion. Seek to find out the interviewee’s rationale that 

brought him or her to that assessment. 
68

Annex 2 – Checklist for Conducting Social Science Research Interviews

Factors Checklist Items

Before the interview

Prepare topical outline and questions.

Vet questions with research team.

Send “introduction letter” (signed by research director) to interviewee. See Attachment 

1 to this annex for a suggested template. Enclose the topical outline (or if a “factdocumenting” interview, you may include the questions. Withhold questions if an 

“opinion-based” interview).

Environment

Schedule a time when there will be no interruptions.

Select a room with comfortable temperature and humidity.

Be sure the room is quiet.

Exclude any observers (unless you are interviewing a person who wishes to have his or 

her executive assistant or legal adviser present, or when it might be necessary to have a 

security officer present).

Interview Team

Whenever possible, there should be a team of two conducting the interview, an 

“interviewer” and a “recorder” or “scribe.”

The “interviewer” should be the exclusive questioner (except for those cases where a 

discussion-type interview format might be more appropriate. You should clear the use of 

this format with the research manager or project director).

The “recorder/scribe” should take notes on key points and document relevant body 

language and other contextual elements.

The interviewing team members should avoid interjecting their own opinions about the 

subject under study.

The team composition should avoid members who might “influence” or “intimidate” the 

interviewee (e.g., a uniformed high-ranking military officer interviewing junior enlisted 

personnel, or a foreign national participating on the interview team).

During the interview

First, be sure the interviewee signs an interview agreement form, such as the example in 

Attachment 2 to obtain informed consent.

Review the ground rules (summarize key points in the interview agreement form, and 

state the role of others who might be present).

Always record the interview, preferably with a digital recorder.

Have the interviewee specify dates, names, and places; explain jargon; and, give context. 

Follow-up with probing questions, as appropriate.

After turning on the recorder, start with an introductory statement like:

“I am recording an interview with [interviewee], [position & and affiliation]. I am 

[interviewer], and today is [date]. We are conducting this interview in [city/state, 

building/room]. The classification level will be up to [indicate].” [If others are in the 

room ask them to state their name and affiliation].

After the Interview

Send a written thank you; invite follow-up information. 

Transcribe, correct/edit, and document the interview.
69

Annex 2 (Checklist)

Attachment 1—Sample Template for Pre-Interview “Introduction Letter”

 [Insert DATE] 

[Insert NAME]

[Insert ADDRESS 1]

[Insert ADDRESS 2]

[Insert ADDRESS 3]

Dear [Insert NAME]:

I request an opportunity to interview you in support of a current research project that is under 

the overall direction of [Insert the name of the overall director]. The subject of the research project is 

[Insert a phrase that describes subject under research]. 

Mr./Ms./Dr. [Name and position title of project manager], is the project manager for this research 

activity. Mr./Ms./Dr. [Interviewer] is one of the interviewers and is the research interviewer whom 

we identified to interview you. Mr./Ms./Dr. [Interviewer] [Insert something about the interviewer to 

introduce him or her, e.g. credentials, experience, etc].

Our research is focusing on [Insert an explanation of the scope and purpose of the research]. 

We are interested in hearing your views about [Insert what it is we want from the interviewee]. As 

the [Insert interviewee’s relationship with the matter under study], you are in a unique position to 

comment on [Insert whatever it is]. We are seeking an open and frank discussion of your personal 

opinions on the matters under our study. For your background, I have enclosed a list of topics that 

[Interviewer] plans to discuss with you.

[Interviewer] will record the interview. Resources permitting, we will transcribe it over the 

next several weeks. Should we be able to do that, we plan to provide you with the initial draft 

transcript for your review and editing. We will then correct or revise the draft transcript and use 

the corrected transcript as one of our sources to research and write about [Insert whatever it is]. 

At the end of this research project, we will maintain a copy of the draft transcript for follow-up 

research in [State where].

We will conduct this interview at the [Insert classification] level. (If classified, state that the 

interview will take place at a secure facility, and indicate where that will be.)

As part of our research protocol [Interviewer] will request your consent for us to interview you 

and ask you to sign our standard interview consent form, which also explains how we handle the 

interview information we collect. I have enclosed a copy of that form for your advanced reading.

We will contact you within the next [Insert number of days] days to discuss your availability for 

the interview and any relevant security constraints. Should you have any questions in the interim, 

feel free to contact either [Interviewer] or me [Insert phone number and/or e-mail address].

 Sincerely,

Enclosures:

Potential Topics List

Interview Agreement 
70

Annex 2 (Checklist)

Attachment 2 — Sample Interview Consent Form

Research Information & Interview Consent FORM

(For Oral History and Other Research Interviews) 

 1. I, [Insert name of interviewee], voluntarily agree to participate in a recorded interview, 

which will be conducted by [Insert name of interviewer], representing [Insert name of organization 

conducting/sponsoring the research]. I understand that I may withdraw from participation in the 

interview at any time during the course of the interview.

2. Nature of Interview. I understand: (a) the interview is part of [Insert name of organization 

conducting the research]’s ongoing consultative and deliberative process of documenting historical 

experiences and assessing lessons learned from those experiences, and (b) the [Insert name of 

organization conducting the research] is seeking open and frank discussion of my personal opinions 

on the matters to be addressed. 

3. Purpose of Research & Contacts. The purpose of this research is [Insert a brief explanation 

of the purpose of the research]. For more information about this research project, you may contact 

[Insert contact information].

4. Record of the Interview. I understand that the written transcript, and any associated 

recording, may be accessioned into the [Insert name of organization conducting/sponsoring the 

research] reference collection after an accuracy check by the interviewee, as resources, time and 

opportunity permit. 

a. The [Insert the name of the organization conducting/sponsoring the research] treats all recordings 

and/or interviewer notes as draft, raw research data and considers any content that reflects your 

personal opinions, interpretations, and advice to the [Insert title of the senior director of the research 

organization] to be “draft and pre-decisional deliberative.” These raw research data typically will be 

segregated from other research data and retained in a controlled environment.

b. Access to such draft, pre-decisional research data is controlled by the [Insert title of the 

senior director of the research organization], and that access limited to those with a validated research 

requirement. Researchers may use records of the interview as primary source research material only 

when authorized by the [Insert title of the senior director of the research organization] (or designated 

representative).

Interviewee: ___________________ Interviewer: __________________

Location:_____________________ Date: ________________________

page 1
71

c. The [Insert title of the senior director of the research organization] clears finished research 

products prior to publication, and will attempt to honor any restrictions the interviewee might 

request to be placed on interview materials, in the event the interviewee makes such a request.

d. (Include a paragraph along these lines only if the research sponsor is a Federal Government 

organization.) Records about me that are collected through my participation in this research activity 

are covered by [Insert an explanation of the authority for collecting the privacy data, i. e., identify the 

system of records notice published in the Federal Register, explaining where the notice is available, and 

provide the subject with a “Privacy Act Statement” that includes information about the notice.]

5. Release of Interview Record. (Include this paragraph only when a freedom of information 

(FOIA) law might be applicable to the sponsoring organization) Should any record of an interview 

be deemed responsive to a FOIA request, it would be reviewed in accordance with [Insert name of 

the relevant governmental agency’s] review and release policy and any drafts that are pre-decisional/

deliberative personal opinions would be redacted or denied in full based on the applicable FOIA 

exemption. However, it could be possible that some cleared and published material that is responsive 

to a FOIA request after relevant redactions, may be released to the public contrary to the wishes of 

the interviewee. 

6. Classified Information. (Include this paragraph only if national security classified 

information is involved and the interview subject has a pre-publication review requirement.) 

I understand that any classified information that I might discuss during the interview must be 

controlled and protected according to the security policy for that information. 

a. I acknowledge that I am not authorized to publish any information related to this interview 

without first obtaining the written approval of the [Insert the name of the relevant government 

organization] prepublication review authority.

b. Prior to publication, or making arrangements for publication, of the interview records(s) or 

any work that I create that includes all, or any portion(s), or the record(s) of the interview, I agree to 

obtain pre-publication approval from the [Insert the name of the relevant government organization], or 

its successor organization, and to publish only when such approval is granted.

7. Access to Interview Material. Consistent with any dissemination controls levied by 

the [Insert the name of the relevant parent organization], I understand that I—or my executors, 

administrators, heirs, and assigns—may be given access to records of this interview to review or to 

use for research and publication purposes.

8. Use of Interview Material. As the subject of this interview, I grant, as qualified by any 

conditions I might append to this agreement, permission for the [Insert the name of the organization 

conducting/sponsoring the research] to use, duplicate, reproduce, and distribute my name and any 

photographs, artwork, videotapes, electronic representations or sound recordings associated with 

this interview, on an ongoing and cost-free basis for: scholarly research, documenting history and 

lessons-learned, and purposes such as promotional, news and public relations, or other comparable 

uses in print or electronic media.

page 2
72

9. Gratuitous Service. I also understand that my participation shall be considered gratuitous 

in nature, and the performance of services under this agreement is without remuneration. The [Insert 

the name of the organization conducting/sponsoring the research] shall not be liable to the interviewee 

for any expenses paid or incurred by the interviewee in connection with this interview (Include the 

following phrase when the parent organization will be reimbursing the subject for travel expenses), 

except for when the [Insert the title of the senior director of the research organization] has approved the 

reimbursement of travel expenses in writing in advance of the expenditure. 

________________________________ _____________________

(Interviewee) (Date)

_______________________________ _____________________

(Interviewer) (Date)

_______________________________ _____________________

(Accepted on behalf of the [Research organization]) (Date)

page 3
73

Annex 2 (Checklist)

Attachment 3—Sample Template for Post Interview “Thank you Letter”

 [Insert DATE]

[Insert Name of Interviewee]

[Insert Address 1]

[Insert Address 2]

Dear [Insert Name],

Thank you for participating in an interview for [Name of research project].

As our interviewer explained prior to the interview, we will be holding the interview material 

for further review as part of our deliberative research process. Our post-interview process is:

•Mark interview materials with the designation, “Draft Pre-decisional Deliberative” along 

with any required provisional classification markings;

•As time, funding and priorities permit, transcribe the recorded interview and make a copy 

of the draft transcript available to you for review and clarification;

•When time, funding and priorities permit, we will fact check the historical content of the 

draft transcript;

•Accession of the pre-decisional/deliberative recording and transcript into [Identify where 

you will be holding this draft, primary source material].

We recognize that the responses you provided during the interview are what you recall 

as facts; accordingly, we typically defer release or publication of a finished product until we 

can evaluate your information during our research and editorial deliberative review processes. 

We want to avoid the premature disclosure of preliminary assessments or proposed advice, 

and we want to protect against confusion that might result from the publication and release of 

information that, in fact, is not the ultimate grounds for assessments based on the insight we 

learn from our interviews.

Thank you, again, for joining us to participate in this history/lessons-learned research 

project, and providing us with your open and frank discussion. It is the insight we gain from 

these kinds of interviews that are one of the foundations of our research.

 Sincerely,
74

Annex 3 –Questions to Consider and Avoid

Questions Definition Examples

Helpful Consider using these kinds of questions

Descriptive

A question that elicits a narrative description. These 

questions would include those that elicit information about 

broad experiences, a description of a specific experience, 

an example of an experience, or an anecdotal story.

Tell me about a typical day at the 

ground station. Tell me about the 

morning staff meeting. Share an 

example of when the committee 

cannot agree on the tasking 

requirement. Tell me the story of 

your experience during the launch.

Structural

A question that seeks an answer that provides structure to 

a particular phenomenon. These questions would include 

those that seek answers that are related to sequence’s 

rationale, spatial relationships, causes, attributions.

Tell me about the steps you go 

through to plan a mission. What 

were the reasons for cancelling 

the program? At what places 

do you conduct the engineering 

tests? What were the causes of 

the failed launch? Describe the 

experience necessary to design 

the system.

Comparative

A question that seeks an answer that compares two 

entities. These questions would include those that seek 

contrasts and similarities.

What are the differences between 

the East Coast and West Coast 

operations? Describe the common 

themes in the requirements from 

the two agencies.

Problematic Generally avoid these kinds of questions

Binary A question that elicits a “yes” or “no” answer. Were there any managers who 

disagreed with the proposal?

Leading

A question with assumptions that could influence the 

answer.

How often did you violate the rules? 

(Assumes the interviewee violated 

the rules). When are you shutting 

down the program? (Assumes the 

interviewee will be shutting down 

the program).

Emotional A question with emotionally-laden words. What was it like working for a loudmouth boss?

Compound

A question that is actually more than one question. Describe the various players, 

and explain the functions they 

performed.

Jargon-heavy

A question that uses technical terms and acronyms. How did cognitive dissonance 

impact on the decision for the 

radsint program at JMARC?

Vague A broad question where it is not clear what the interviewer 

is looking for.

What is in the future?

Double Negative A question that uses two forms of negation. Tell me about a program where you 

did not receive insufficient funding?
75

Appendix D—Memory: The Uncertainty of Recalling 

the Past—An Commentary on the Phenomena of 

Perceiving, Forgetting, and Fabricating

Robert A. McDonald

“It’s a poor sort of memory that only works backwards”

Spoken by the White Queen in Through the Looking Glass1

Most people assume that their memory is reliable, and it is an accurate record of their 

looking back at past events and experiences. In fact, memory is unreliable and faulty. It not 

only looks backward, but it also exists in the present and looks forward. 

Recollections of past events are a prime source of data for historical and social science 

research. Yet there is strong evidence in the social science literature that reports based on 

memory are subject to biases and errors, and any memory-based reports might prove to be 

the least reliable for documenting past experiences and events. The kind of memory that we 

use to recall events or episodes from our personal experiences is labeled, episodic memory, 

as compared with semantic memory. Semantic memory refers to memory about facts in the 

world, e.g., the name of the President, the capital of a state, a math formula, while episodic 

memory is the source of recollections of our past experiences, and, therefore the information 

drawn out during oral history and other social science research interviews.2

 (Means & Loftus, 

1991, p. 297; Corballis, 2013, p. 63).

It is not only social scientists who have concluded that memory becomes a challenge to 

the objective of learning realities from the past. Those who teach writing of autobiographical 

narratives acknowledge that memory is faulty, and recognize that this defect is a key challenge 

in writing memoirs and related accounts that are drawn from memory. In creating, or perhaps 

we might say “fabricating” the story from memory, some explain that this fabrication in the 

writing of memoirs is neither lying nor the creation of fiction, but merely the experience of 

challenging the limits of our memory. Regrettably, the challenge to the reader (especially one 

who is a social scientist trying to validate the authenticity of the narrative) is to determine when 

the limits of reality in the story have been exceeded (Roorbach & Keckler, 2008, p.28).

Unfortunately, most of us generally believe that our memories preserve precise accounts 

of what we perceived and experienced. At the same time, we are unaware of the errors in our 

Notice: This commentary is copyrighted and is used with permission in this government publication. The views 

expressed in this essay are the personal views of the author. © 2015, Robert A. McDonald, all rights reserved.

1 Chapter V, “Wool and Water” 

2 Episodic memory appears to be centered in the hippocampus part of the brain. In one case a patient, 

Clive Wearing, suffered damage to the hippocampus as a result of a viral infection. That deprived him of 

episodic memory and recollections of his past life experiences, although his semantic memory appeared to 

remain intact, e. g., he could still play the piano (Corballis, 2013, p. 63).
76

memory—we simply do not know how much of our past experiences we have forgotten. As a 

result, we assume that our memory is reliable and accurate, and we believe that the representations 

of the past that are stored in our memory are experiences we can recall with great detail and 

precision—and we believe the recalled experiences accurately can reproduce the reality of the 

past episodes. Unfortunately, the reality that we construct in our memory is unreliable and 

faulty. What we perceive as our experience is imprecise; what we retain as memory is scanty; 

what we remember can become distorted by conflicting information; what we recall from our 

memory often is based on inference. For the following discussion I have organized some of the 

more significant error sources into three broad general categories: input error, contamination 

error, and recall error (Chabris & Simons, 2009, p. 62; Corballis, 2013, p. 60). 

Input Error

For there to be any memory, there first needs to be the input of information. The 

environment, however, presents the senses with much more information than is required for 

decision making or action, and more than can be retained. After perceiving the broad spectrum 

of information in the environment, the brain selects and retains only those data that seem to 

be important and discards the data that appear to be uninteresting or irrelevant. In other 

words the brain only retains information that is necessary to respond to environmental stimuli, 

essentially ignoring and forgetting the rest. This process introduces errors during the input of 

information (Aamodet & Wang, 2008, pp. 2-4, 151; Corballis, 2013, p. 60). 

These input errors that are introduced while experiencing an event and perceiving 

information in the particular scene associated with that event are varied and can include such 

factors as inattentional blindness, unconscious cognitive bias, and pattern fabrication.

Inattentional Blindness. One of the most fundamental input errors is inattentional 

blindness. This results when there is a lack of attention to the unexpected. Simons and Chabris 

(1999, pp 1059) conducted what has become a classic study where the subjects watched a video 

of two teams moving around and passing basketballs. The experimenters asked the subjects 

to watch the video and count the number of passes made by the players wearing white and 

to ignore any passes made by the players wearing black. Halfway through the video, a young 

woman wearing a full-body gorilla suit walked across the scene stopping in the middle and 

thumped her chest, then exited the scene. When the experimenters questioned the subjects after 

watching the video, about half of them did not see the gorilla. The study demonstrated that 

when individuals attend to a particular aspect of a scene, they tend not to see something that 

is unexpected, even when they are looking directly at it, and it is something most prominent 

(Chabris & Simons, 2009, p.6-7; Simons and Chabris, 1999, pp1059).

There are real-world instances that can be explained by inattentional blindness. 

Two examples: In 2006 Martha Fleishman, who had a perfect driving record, was driving 

her Chrysler New Yorker on Second Ave in Pittsburg. While turning left she hit a Suzuki 

motorcycle that professional football player Ben Roethlisberger was riding. Even though the 

motorcycle was in front of Fleishmen, she did not see Roethlisberger, and drove directly into 
77

him. In 1977 KLM flight 4805 began its take-off role down the runway and collided at full 

speed with Pan Am flight 1736 taxiing down the same runway in the opposite direction. This 

is another case of an individual not seeing what the actor was looking at. In both cases, what 

those involved hit was unexpected and “unseen,” even though it was in front of the actors. This 

inattentional blindness places into question the reliability of recall elicited during research 

interviews (Chabris & Simons, 2009, p.6-7, 14, 20; Simons and Chabris, 1999, pp 1059).

Unconscious Cognitive Bias. Another fundamental input error is unconscious cognitive 

bias. Individual beliefs and attitudes are shaped by automatic and unconscious cognitive 

processes, where we store and retrieve information that is influenced by unconscious cognitive 

processes. Without deliberate thought and awareness, we interpret people, behavior, and 

situations oftentimes inaccurately, and then store the information as recollections of the 

episode. There is a growing body of research across a variety of disciplines, including social 

work, public health, law, and medicine that supports this view. For example, Haider, et. al., 

(2011) found that in a survey3

 of medical students at Johns Hopkins School of Medicine about 

70% to 85 % of the students had an implicit preference toward those in the upper social class. 

The implicit bias recorded in the survey was significantly different from the students’ stated 

preferences (Teal, Gill, et. al., 2012).

Unconscious biases are ingrained in our cognitive processes where we use social categories 

to acquire, process, and recall information about our environment. This unconscious 

stereotyping helps us organize complex information. These processes permit us to use social 

categorization to reduce what can be a complex social world into manageable chunks of 

information. The stereotyping fills gaps in meaning, especially when the implications of the 

episode are unclear or ambiguous. When there is a heavy cognitive load, such as when we are 

under time pressure or suffering from fatigue, the cognitive processing relies more heavily 

on stereotyping to process incoming information from the environment. This cognitive bias 

is outside our consciousness, but the meaning assigned to the experience appears to us to be 

objective, rational, and justified. This process calls into question the quality of information 

that is being perceived and stored in memory (Haider, et. al., 2011; Hart and Jaccard, 2008; 

Teal, et. al., 2012; van Ryn and Saha, 2011).

Pattern Fabrication. Because we tend to rely on pattern perception to recognize patterns 

that give us insight, we may inaccurately interpret a series of random indicators as meaningful 

patterns even when no meaningful pattern exists. On the one hand, experts are trained to look 

for patterns that fit their expectations, and this can be helpful. Physicians look for patterns that 

are helpful in matching symptoms to specific diseases; military commanders look for patterns 

that are helpful in the tactical deployment of forces; teachers look for patterns that are helpful 

in developing teaching methods. On the other hand, the expectations of individuals, including 

experts, can lead individuals to see patterns that do not exist. (Chabris & Simons, 2009, p. 

154-159)

3 The Implicit Association Test (IAT) is a survey instrument that assesses a range of biases that include 

factors such as age, religious affiliation, ethnicity, immigrant status, race, obesity, sexual orientation (Teal, et. 

al., 2012, p. 83).
78

Redelmeier and Tversky (1966) reported on experiments that support this view. In 

one experiment the experimenters showed subjects pairs of numbers, one set purported to 

be indicators of patients’ pain level on a particular day and the other purported to be the 

barometric pressure for that day. Up to 87% of the subjects saw a positive relationship, yet the 

data were fake. Much of what is stored in memory actually can be distorted assessments that 

match our conception of what we believe we saw rather than the reality. Our understanding, 

and therefore our memory of a particular episode, can be of the recollection of a pattern that 

actually is a biased fabrication rather than an awareness of the randomness of the actual events 

that occurred (Chabris & Simons, 2009, p. 154-159; Redelmeier and Tversky, 1966).

Contamination Errors

Information acquired during an episodic experience is stored as chunks of data in our 

memory. These chunks of data can become contaminated after the initial experience while 

they are stored in memory. The contamination errors that are introduced after data are stored 

in memory can come from a number of sources and can include such factors as the override 

from new information, source memory failure, melding of similar events, and creation of false 

memories (Aamodet & Wang, 2008, pp. 151).

Override from New Information. The exposure of new information after an experience 

or event can modify the information held in the memory. Loftus (1996) reported on a study 

where subjects were shown a video recording of eight student demonstrators. As part of a postviewing questionnaire, Loftus gave one half of the subjects false information that there were 

twelve demonstrators, and gave the other half of the subjects false information that there were 

four demonstrators. One week later Loftus asked the subjects how many demonstrators they 

saw. Loftus found that, in general, the false information influenced the subjects’ answers to the 

question about the quantitative fact. The average response by the first group was 8.50, and the 

second group was 6.67 (Loftus, 1975, p 566).

Source Memory Failure. Failure of source memory is when we are unable to accurately 

have knowledge about how we came to have the memory. We can forget the source of a 

memory and even may appropriate another person’s story, mistakenly attributing someone else’s 

encounters with our own experiences. Source memory failures can be quite common and easily 

can be demonstrated. Wade, Garry, et. al. (2002) presented subjects with altered photographs 

showing each subject as a child enjoying a hot air balloon ride; even though the subjects never 

had taken hot air balloon rides. During recall some of the subjects incorporated information 

from the photograph into memories of their personal experiences; 50 % of the subjected created 

a false memory of experiencing a balloon ride—some even embellishing the recollection beyond 

what they had seen in the photographs (Chabris & Simons, 2009, p. 63-64).

Melding of Similar Events. There is a tendency for us to meld together similar events 

in our memory and create a variation of the actual experience. Press reports are replete with 

reports of politicians and celebrities who reported what turned out to be false information. In 

2008 Hillary Rodham Clinton, when Secretary of State, claimed to have been under sniper 
79

fire in Bosnia in 1996, but press reports and photographs from that time showed that not to 

be the case. In 2012, former Massachusetts Governor Mitt Romney recounted a story about 

witnessing an automobile jubilee celebration in Detroit, but he hadn’t been born at the time 

of the celebration. In 2015 Brian Williams, while NBC News anchor, told a story he had been 

telling for years about an attack against a helicopter he was in over Iraq some twelve years 

earlier, but he retracted the story when crew members told Stars and Stripes that the attack 

never happened (The Boston Globe, 2012;Flint, 2015; Linton, 1986; Means & Loftus, 1991, p. 

298; The Washington Post, 2008).

Creation of False Memories. Hyman and Pentland (1996) reported that research 

supports the premise that individuals can create false memories of complete, emotional, and 

self-involving events. Eyewitness memory research has reported that memory is reconstructive. 

Because memory is a constructive process, it combines the content of self-knowledge with 

external suggestions. Therefore, a recollection may be the combined product of information 

stored in our memory combined with suggestions from external sources. Hyman and Pentland 

(1996) conducted a study where participants who were unable to recall an event were asked 

to form a mental image of the event. They then were more likely to “recover a memory,” but 

a memory that created a false event. Braun, K. A., Ellis, R., & Loftus, E. F. (2002) reported 

on an experiment where exposure to an autobiographical advertisement altered the subject’s 

recollection of a past childhood experience and even created a memory of an experience that 

never happened. The subjects viewed a Disney ad that suggested that they shook hands with a 

non-Disney character (e.g., Bugs Bunny). Exposure to the ad increased the subjects’ belief that 

they personally shook hands with the character, which in this case was an impossible character. 

The suggested autobiographical referencing in this case lead to the creation of distorted or a 

false memory (Hyman & Pentland, 1996, p 111).

Recall Errors

When the data chunks stored in memory are recalled, the recollection process can 

introduce additional errors. The recall errors that are introduced during the recall process are 

influenced by a number of causes and can include such factors as temporal changes, age-related 

deficits, re-creation distortions, and re-remembering modifications.

Temporal Changes. The length of time between an experience and the recollection of 

that experience is a crucial interval, and the length of that interval, and the events that take 

place during that interval affect the content of the recall. The findings in the social science 

literature generally agree that memory is less complete and less accurate after a longer retention 

interval than a short one. More of the data in our memory appears to be lost or unrecoverable 

the longer the interval (Loftus, 1996, p. 53).

The literature does not offer good explanations as to why some episodes in our past 

experiences are retained in memory and why others do not. Sigmund Freud (1938) offered 

the hypothesis that we repress memories of trauma; however, more recent research fails to 

support this. It now appears that memories of emotional experiences are more strongly retained 
80

than other experiences that are more mundane4

. From an evolutionary perspective, this makes 

more sense because those events related to survival more often have an emotional component 

(Corballis, 2013, p. 60-61).

The accuracy of these emotionally-related memories may be questionable. During recall the 

brain invents the details to fill the gaps in order to create a more coherent story. The left side of 

the brain seems to seek logic and order. This is so intense that when something doesn’t make 

sense, the brain will invent a plausible explanation (Aamodet & Wang, 2008, pp. 2-4. 151).

Age-related Deficits. The length of time between perception and recall also is related 

to the aging of the individual making the recall. While aspects of cognition that are based on 

accumulated knowledge and experience actually may improve with age, memory—along with 

other cognitive functions such as attention, processing speed, and the ability to switch between 

tasks—declines over time with age. That results in errors of omission in the remembrance that 

ultimately is recalled, and puts into question the reliability of those memories that are from 

episodes that occurred at a younger age (Chabris, C. & Simons, D., 2009 p. 26).

Re-creation Distortions. The length of time the memory is stored and the aging of the 

individual recalling the memory interact with each other during the re-creation process to 

generate distortions in the accuracy and completeness of the recollection. We not only see what 

we expect to see, but we also remember what we expect to remember by making sense of the 

memory during recall. Brewer & Treyens (1981) conducted an experiment where they asked 

the subjects to wait in a graduate student’s office for a short while before having them move 

to another room where they asked them to write down what they saw in the office. Almost all 

the subjects listed desk chairs, shelves—items that were in that office and most other typical 

graduate student offices. About 30% of the subjects recalled seeing books, and about 10% 

recalled seeing file cabinets—both of which were not in the office. During recall, the memory 

reconstructed objects in the room based on both what actually was there, as well as what 

the subjects expected should be there. What we recall is not a replica of reality, but rather a 

recreation of what we expect that reality to be (Chabris & Simons, 2009, pp.48-49).

Re-remembering Modifications. The experience of recall and re-remembering also can 

modify the facts in the memory. During a series of recalls, there appears to be variability in the 

content of information that is recalled. In one case from World War I, a soldier’s succession 

of accounts written between 1914 and the 1970s changed significantly over the years. One 

explanation could be that the remembering of certain kinds of experiences, especially highly 

emotional-experienced events, such as war time experiences, can be part of an individual’s 

search for ways to deal with the feelings experienced during the initial events; it therefore 

becomes necessary to modify the story in the memory to more easily deal with the feelings. 

As the individual recalls the experience to resolve emotional conflicts, the result is a blurring 

4 An interesting observation is that in cases of amnesia. It is the episodic memories that more severely 

are affected, while the semantic memories may be undamaged. Episodic memories generally also are more 

intensely affected by Alzheimer’s disease. Does this suggest that episodic memories are more fragile, and, 

therefore, at much greater risk of being lost or distorted? (Corballis, 2013, p. 63-64)
81

of distinctions between ‘fact and fiction,’ and ‘real and imaginary’ (Dawson, 1994; Roper; 

2000, p 181).

When the Challenger exploded on 28 January 1986, Neisser and Harsch (1992) used it as 

an opportunity to demonstrate recall errors. They asked the students in their class at Emory 

University to write a description of how they learned of the explosion and to answer a series of 

questions about the accident. This immediate documentation of the details should provide an 

accurate baseline description of the event. Two and a half years later Neisser and Harsch asked 

the same students to complete another detailed questionnaire. The memories that the former 

students reported at this time were dramatically different from the details initially reported. 

They incorporated “facts” into their responses that were plausible, but never actually happened. 

In these later responses some former students remembered hearing about the accident from 

different people, at a different time, and in a different situation (Chabris & Simons, 2009, p77; 

Neisser and Harsch 1992).

A classic real-world example of the variability in recall is in the recollections of British 

World War I soldier, Colonel Thomas Edward Lawrence (T.E. Lawrence, known as “Lawrence 

of Arabia”), who dramatically led Arab forces across the desert to numerous battlefield 

successes. The Turkish army captured him in Deraa and subjected him to torture. In at least 

three different autobiographical accounts of this particular emotionally-laid episode in his life, 

Lawrence recalled three different narratives (Anderson, 2013, p 400-401; Lawrence, 1991).

Lawrence, with the accounts of his various exploits, has become an iconic figure, and 

biographers and historians highjacked his story to make it their own. This has resulted in ever 

changing, and sometimes mythical and conflicting accounts of his exploits with the Arabs 

against the Turkish military. As a result, he has become a legendary and enigmatic figure in 

the history of World War I. He has become a twentieth century icon who is seen to have led 

an Arab army in pivotal encounters against the Turkish military. He has become a hero who 

contributed to ending World War I and influencing the creation of what would become a 

complex Middle East, the Middle East that would have to be dealt with over the next century 

and beyond (Anderson, 2013, p 2; Dawson, 1994; Lawrence, 1991). 

Conclusion

From the above discussion it is clear that the evidence in the psychological and medical 

literature strongly supports the view that memory is highly unreliable. Nevertheless, there 

continues to be an illusion that through careful elicitation and recall, we can tap an accurate 

record of the past that resides in our mind. At the same time, most of us generally have a high 

degree of confidence in both our and other’s memory. In one survey 47% of the respondents 

believed that memory does not change, and 63% of the respondents believed that memory 

accurately records events experienced or observed in an episode for accurate recall and review 

in the future. The reality is that we are unaware of what we have forgotten because that 

information is lost. We cannot separate those aspects of our memory that are an accurate 

reflection of the past from those aspects that are based on new information that we may have 
82

acquired, and subsequently altered. The altered information is the only information about the 

past episode that is in our memory (Chabris & Simons, 2009, p. 63). 

Despite the strong evidence that memory is flawed, there is a range of common 

practices—conducting police investigations, taking medical histories, conducting oral history 

interviews—that continue to rely on individual memories as reliable sources of information 

gathering. The record of evidence, argues for a reality that is to the contrary. All the disciplines 

that elicit information from memory must be very cautious in how they use the data acquired 

from individuals’ memories, i.e., data acquired from research interviews where the subjects 

describe events as recalled in their episodic memories. These data are not facts, but stories. 

Tobias Wolff (1989), a memoir author, explained early in one book that his memoir was a story 

from his memory, and in writing it he did his best to tell a truthful story from that memory, 

but he pointed out that memory does not always tell a truthful story because “memory has its 

own story to tell.” At best memory is a vague impression or explanation of the past; at its worst 

it is a distorted description of a past experience or observation. In all cases it is a questionable 

source for social science and historical research.

“I have been through some terrible things in my life, some of which actually happened.”

Author Unknown5

5 This quote often is attributed to Mark Twain, but Fred Shapiro, the editor of The Yale Book of Quotations 

(Yale University Press, 2006) has concluded that the statement cannot be verified as having been written by 

Mark Twain. Nevertheless, the quote sums up the reality of memory, its ambiguous source citation is an 

example of the difficulty in documenting episodes from the past.
83

References

Aamodt, S. & Wang, S. (2008). Welcome to Your Brain: Why You Lose Your Car Keys but Never 

Forget How to Drive and Other Puzzles of Everyday Life. New York: Bloomsbury.

American Psychological Association, APA (2015). Glossary of Psychological Terms. 

http://www.apa.org/research/action/glossary.aspx, Accessed 5 Jul 2015

Anderson, S. (2013). Lawrence in Arabia: War, Deceit, Imperial Folly and the Making of the Modern 

Middle East. New York: Doubleday

The Boston Globe (2012). Romney’s memory is “foggy” on Detroit’s Golden Jubilee. Feb 28, 

BostonGlobe.com, https://www.bostonglobe.com/news/nation/2012/02/28/mitt-romneystumbles-recalling-historic-auto-event-detroit/xy6oajx6LR8rQXHWnkd0bN/story.html

Accessed 5 July 2015.

Braun, K. A., Ellis, R., & Loftus, E. F. (2002). Make my memory: How advertising can change our 

memories of the past [Electronic version]. Retrieved [10 August 2015], from Cornell University, 

SHA School site: http://scholarship.sha.cornell.edu/articles/332

Brewer, W. F. and Treyens, J. C. (1981). “Role of Schemata in Memory for Places,” Cognitive 

Psychology, 13: 207-230. 

Carroll, L. (1991). Through the Looking-Glass. The Millennium Fulcrum Edition 1.7, Project 

Guttenberg, http://www.gutenberg.org/files/12/12-h/12-h.htm [Release date Feb 1991; Last 

updated 8 Jan 2013, Accessed 18 July 2015]

Chabris, C. & Simons, D. (2009). The Invisible Gorilla and Other Ways Our Intuitions Deceive Us.

New York: Broadway Paperbacks.

Corballis, M. C. (2013). A Very Short Tour of the Mind: 21 Short Walks Around the Human Brain, New 

York: Overlook Duckworth.

Dawson, Graham (1994) Soldier heroes : British Adventure, Empire, and the Imagining of Masculinities. 

London: Routledge. 

Flint, J. (2015). NBC’s Brian William’s Recants Tale of Helicopter Attack. The Wall Street Journal, 

http://www.wsj.com/articles/nbcs-brian-williams-recants-tale-1423101759 Accessed 5 July 2015.

Freud, S. (1938). The Interpretation of Dreams. In A. A. Brill (Ed., Trans.), The Basic Writings of 

Sigmund Freud (pp. 181-549). New York: The Modern Library. (Original work published 1900)

Haider, A. H.; Sexton, J.; Sriram, N.; Cooper, L.; Efron, D. T.; Swoboda, S.; Villegas, C. V.; 

Haut, E. R.; Bonds, M.; Pronovost, P. J.; Lipsett, P. A.; Freischlag, J. A.; and Cornwell, E. E. 

(2011). “Association of Unconscious Race and Social class Bias with Vignette-Based Clinical 

Assessments by Medical Students,” JAMA, 306(9), pp. 942-951.

Hart, B. and Jaccard, J. (2008). “Unconscious Racism: A Concept in Pursuit of a Measure,” 

Annual Review of Sociology, 34, pp. 277-97).
84

Hyman, I. E. & Pentland, J. (1996). The Role of Mental Imagery in the Creation of False 

Childhood Memories, Memory and Language 35, 101-117.

Lawrence, T. E. (1991). Seven Pillars of Wisdom—A Triumph, New York: Anchor books.

Linton, M. (1986). Ways of Searching and the Context of Memory. In D. C. Rubin (Ed.), 

Autobiographical Memory (pp. 50-67). Cambridge: Cambridge University Press.

Loftus, E. (1975). Leading Questions and the Eyewitness Report. 

Cognitive Psychology, 7, 560-572.

Loftus, E. (1996). Eyewitness Testimony. Cambridge: Harvard University Press.

Means, B. & Loftus, E. F. (1991). When Personal History Repeats Itself: Decomposing Memories 

for Recurring Events, Applied Cognitive Psychology, 5, 297-318.

Neisser, U. & Harsch, N. (1992). “Phantom Flashbulbs: False Recollections of Hearing the News 

About Challenger.” In Affect and Accuracy in Recall: Studies of “Flashbulb” Memories, Ed. By E. 

Winograd and U. Neisser, Cambridge: Cambridge University Press.

Redelmeier, D. A. and Tversky, A. (1996). “On Belief That Arthritis Pain is Related to the 

Weather” In Proceedings of the National Academy of Sciences, 93: pp. 2895-2996. 

Roorbach & Keckler (2008). Writing Life Stories. Cincinnati, OH: Writer’s Digest Books.

Simons, D. J. & Chabris, C. F. (1999). Gorillas in our Midst: Sustained Inattentional Blindness 

for Dynamic Events, Perception 28, 1059-1074.

Teal, C. R.; Gill, A. C.; Green, A. R.; Crandall, S. (2012). “Helping Medical Learners Recognize and 

Manage Unconscious Bias Toward Certain Patient Groups,” Medical Education, 46, pp. 80-88.

Twain, Mark (n.d.). QuotesWave.com. Retrieved July 18, 2015, from QuotesWave.com. Website: 

http://www.quoteswave.com/picture-quotes/381387 Read more at http://www.quoteswave.

com/picture-quotes/381387#pjCu1yTjy4Q40eAT.99

Wade, K. A.; Garry, M.; Read, J.D.; and Lindsay, S. (2002). “A Picture Is Worth a Thousand Lies: 

Using False Photographs to Create False Childhood Memories.” In Psychonomic Bulletin and 

Review, 9: pp 597-603.

Van Ryn, M. and Saha, S. (2011). “Exploring Unconscious Bias in Disparities Research and 

Medical Education,” JAMA, 306: 9, pp 995 ff. 

Washington Post (2008). Sniper Fire, and Holes in Clinton’s Recollection. Washingtonpost.

com, March 22. http://www.washingtonpost.com/wp-dyn/content/article/2008/03/21/

AR2008032102989.html Accessed 5 July 2015.

Wolff, Tobias (1989). This Boy’s Life—A Memoir. New York: Grove Press.
85

Appendix E—Outline and Checklist Questions for 

Case Description

Overview of Appendix E

This appendix provides an outline with a checklist of questions for you to use in writing 

the case description. I based the outline on the various components of the general systems 

paradigm. By using this outline and checklist you will be sure to consider all the components 

of the system as you write the case description. 

***

Outline with Checklist Questions

 Abstract

(Draft a short one- or two-paragraph abstract to serve as a high-level summary.)

 Executive Summary

(Draft a two- to three-page summary that describes the essence of the 

organizational system and the case.) 

Introduction

(Draft a brief discussion to introduce and define the organization, the systems with 

which it is associated, and the issue that prompted the preparation of a lessonslearned case study.) 

 Main Report

(Draft a descriptive discussion of each of the four components of the system, 

and organize the report into four sections, one for each component. Describe 

each component by discussing its subcomponents and associated features. 

Use the questions listed under each feature as a checklist of points to cover. 

Not all of subcomponents and features may be relevant to your case, but 

answering the checklist questions should help you determine which features 

you need to discuss.)
86

Section 1—Environmental Components 

(Describe the environmental context in which the organization exists.)

1. Societal Domain 

(Describe the “human ecology”)

a) Geopolitics

1. To what degree do geographical factors such as the Earth’s 

landscapes, peoples, places, and environments influence the case?

2. To what degree is the spatial allocation of territory, the distribution 

of natural resources, and the distribution of populations relevant 

to the case?

3. What national security threats impact on the case?

4. What foreign policy constraints are relevant?

b) Domestic Politics

1. To what degree does the national election cycle have an impact on 

the case?

2. How do the dynamics of Congressional politics affect the case?

3. How do the dynamics between the Congress and the Executive 

affect the case?

4. What political scandals might impact the case?

c) Economics

1. How do the domestic economic conditions (e.g., unemployment, 

inflation, economic growth) affect the case?

2. What is the impact of monetary policy on the case?

3. What is the impact of tax policy on the case?

d) Other Societal Factors

1. How does population migration affect the case?

2. How does the changing age distribution of the population 

affect the case?

3. How do factors such as crime and domestic terrorism affect the 

case?

4. How do the changing dynamics of the family affect the case?

5. What other societal factors are relevant to the case?
87

2. Associational and Relational Domain 

(Describe the external alliances and associations with suppliers, users, and stakeholders)

a) Dependencies

1. What external organizations are the sources for input?

2. What external organizations depend on the organization’s output?

b) Communications5

1. What are the key external sources of communications input for the 

organization?

2. What external organizations depend on communications output 

from the organization?

3. Limitative and Restrictive Domain 

(Describe the structural and behavioral constraints)

a) Laws & Regulations

1. What legislation creates constraints for factors in the case?

2. What regulatory constraints impact the case?

3. How do Constitutional issues such as 1st Amendment rights (i.e., 

speech, press, religion), 2nd Amendment rights (i.e., bear arms), 

4th amendment (i.e., search and seizure), and 10th Amendment 

(states’ rights) impact the case?

4. How do security constraints impact the case?

5. How do restrictions on domestic activities impact the case?

b) External Oversight

1. How does Congressional oversight impact the case?

2. How does Executive oversight (e.g., Director of National 

Intelligence, Undersecretary of Defense for Intelligence, Office of 

Management and Budget) impact the case?

c) Fiscal Constraints

1. How does the budget cycle affect the case?

2. How do budget ceilings and budget authorities affect the case?

5 See the section on “Information Network” for questions related to organization’s internal information network.
88

d) Technical constraints

1. How do the physical laws of science, such as the laws of physics (e.g., 

law of gravity, laws of thermodynamics, speed of light,) and the laws 

of chemistry (e.g., Boyle’s law,6

 Charles law7) affect the case?

2. How does time, such as the time-required-to-complete a project 

(i.e., the best-case fixed amount of time it takes to complete the 

project) and the time required to finish a step before starting the 

next step (i.e., the requirement to complete each prior task before

moving on to the next task) affect the case?

3. How do the physical limits of the human body (e.g., limits of 

staying awake, survival in heat and cold, survival without food and 

water, ability to withstand disease) affect the case?

4. How do the psychological limits of the human being (e.g., limits 

of perception, perceptual-motor abilities, ability to understand 

language, ability to withstand stress) affect the case?

e) Other limitative factors

1. What other limitative factors are unique to this organization?

2. What is the impact of these factors on the case?

4. Historical Domain 

(Describe the historical context)

a) Societal Environment

1. What was the political and social history at the time the 

organization came into being?

2. How was the political and social history changed over the life of 

the organization?

b) Organization

1. What is the origin of the organization?

2. What is the history of the organization?

6 At constant temperature, the volume of a confined gas is inversely proportional to the pressure to which 

it is subjected.

7 At constant pressure, the volume of a confined gas is directly proportional to the absolute temperature.
89

Section 2—Input Components

(Describe the input component of the organization.)

1. Requirement 

(Describe the requirement to be satisfied by the output)

a) The Problem

1. What is a specific, detailed explanation of the problem or problems 

to be addressed by the task given to or assumed by the organization?

2. What issue or issues are expected to be resolved by the task given 

to or assumed by the organization?

b) Objective

1. What is the deliverable that is expected to satisfy the need?

2. What is the desired end-state?

3. What is (are) the indicator(s) of success?

c) Complexity

1. How difficult is it to separate the different parts of the problem 

and/or deliverable(s)?

2. How many interest groups are stakeholders?

3. What are the conditions that could hinder reaching the objective?

4. What kinds of constraints are associated with meeting the 

requirement?

5. What is the degree of uncertainty associated with satisfying the 

requirement?

2. Funding

 (Describe the fiscal resources dedicated to satisfying the requirement)

a) Amount of Funding

1. How much funding does the budget allocate to the activity under 

study?

2. Who has execution authority?

b) Nature of Funding

1. To what degree does the funding match the requirement?

2. Who has execution authority?
90

3. Staffing 

(Describe the human resources dedicated to satisfying the requirement)

a) Level of Staffing

1. How many government and contractor personnel are authorized for 

the project?

2. What is the vacancy rate for the authorized positions?

b) Nature of Staffing

1. What is the skill mix defined for the staffing?

2. What is the proportion of government and contractor personnel 

authorized for the project?
91

Section 3—The Process Components

(Describe the process component of the organization.)

1. Formal Organization 

(Describe the formal arrangements between components of the organization.)

a) Demographics of Organization

1. What kind or type of organization is under study in the case?

2. What is the size of the organization? 

3. What is the age of the organization, or how long did the 

organization exist?

4. What is the growth stage of the organization?

b) Design of the Organization

1. How is the organization structured (vertical, horizontal, etc.)?

2. What is the form of the sub-components (by function, program, 

matrix, location, etc.)?

3. What is the vertical differentiation (i.e., the number of supervisory 

levels)?

4. What is the horizontal differentiation (i.e., the number of subunits)? 

5. What is the mix of government and contractor employees?

6. What is the distribution of authority? 

c) Administrative Intensity

1. What is the supervisor-to-staff ratio?

2. What is the top manager’s span of control (include non-supervisory 

personnel)?

3. How extensive is middle management?

d) Bureaucratic Constraints within the Organization

1. What are the organizational affiliations and allegiances?

2. What are the internal HR constraints?

3. What are the internal fiscal constraints?

e) Work of Organization

1. What is the nature of the organization’s work environment?

2. Where is the work done in relation to the organizational components?

3. What is the complexity of the work (i.e., number of different 

products/services, territories of operation)?
92

2. Informal Organization

(Describe the informal arrangements within and extending beyond the organization.)

a) Relationships

1. What is the degree of awareness that the organizations’ components 

have with each other?

2. What are the intramural relationships associated with this 

awareness?

3. To what extent are there informal extramural arrangements that 

extend beyond the formal structure of the organization?

4. What are these extramural relationships?

5. What is the intensity of the intramural and extramural informal 

relationships? 

6. What is the impact of turnover rates on these informal relationships?

b) Culture

1. What is the organizational ideology?

2. How is time regarded (e.g., value as an investable commodity; 

orientation to past, present, and future)?

3. What are the customs, taboos, and prohibitions?

4. What are the organizational symbols and slogans?

c) Influences

1. What is the distribution of power (i.e., ability of “A” to get “B” to 

change)?

2. What are the influence patterns?

3. How much discretion do individuals have (i.e., the discretion 

individuals can exercise in making task-related decisions)?

4. What is the time span and rhythm of the organizations (planning 

cycles, degree activities regulated by time)?

d) Dependencies

1. What are the internal dependencies?

2. What are the external dependencies?

e) Certainty

1. How much agreement is there on goals and priorities?

2. How much clarity is there in how to respond to events?
93

3. Network of Individuals 

(Describe the organization’s collection of individuals.)

a) Demographics

1. What is the gender ratio of the individuals?

2. What is the age range and typical age of the individuals?

3. What is the cultural composition of the individuals?

b) Education

1. What formal education do the individuals have?

2. What informal education do the individuals have?

c) Skills

1. What are the skills directly related to the task?

2. What are the skills associated with related tasks?

d) Experience

1. What is the typical tenure of individuals?

2. What experience do they have in the discipline of national 

reconnaissance?

3. What experience do individuals have directly related to the task?

4. What experience do they have in related industrial sectors?

e) Style

1. What behavioral and attitudinal preferences do the individuals 

have?

2. Are individuals task-oriented or relationship-oriented?

3. Are individuals introverts or extroverts?

4. What communication styles do the individuals exhibit?

5. How do individuals resolve conflict?

6. What behavioral patterns do individuals exhibit?

f) Affect and Motivation

1. What are the expectations of the individuals?

2. What are the needs of the individuals?

3. What rewards do the individuals seek?
94

4. Information Network 

(Describe the arrangements within the organization for communicating information.)

a) Formalization

1. What is the degree to which rules govern information exchange 

between individuals within the organization?

2. What is the degree to which rules govern information exchange 

between groups?

3. To what degree is there centralized control of information sharing?

4. How spontaneous are the communications?

b) Mode of Communication

1. What is the range of information-sharing modes (i.e., face-to-face 

conversations, telephone, memos, email)?

2. What is the distribution of information sharing across the modes?

3. How do meetings compare with individual conversations as modes?

4. What kinds of internal employee publications (such as newsletters) 

are there?

c) Frequency

1. What is the frequency of personal communications?

2. What is the frequency of group communications?

3. What is the frequency of organizational component communications?

4. What is the timing, rhythm, and urgency of communications?

d) Quality & Content

1. How much communication is task-related?

2. How much communication is socially oriented?

3. How difficult is it to make contact with people?

4. How often do communications get lost?

5. How useful do individuals find communications?
95

5. The Task 

(Describe the organization’s work activity directed toward satisfying the requirement.)

a) Nature of Task

1. What kind of task is involved?

2. How much time is required for the task?

3. What is the task variability (the number of exceptions encountered 

in the course of the work)?

4. What is the degree of uncertainty associated with the task?

b) Difficulty of Task

1. What is the analyzability of the task (i.e., the ease and clarity of knowing 

the nature and order of what needs to be done)?

2. What is the predictability of the task (i.e., the ease with which individuals 

can determine the outcome of a particular sequence in advance)?

c) Expertise for Task

1. What skills are required for the task?

2. What specialization does the task require?
96

Section 4—Outcome Components

(Describe the outcome component of the organization.) 

1. The Deliverable(s) 

(Describe the product of the organization’s work and the process that resulted from 

trying to satisfy the requirement.)

a) Nature of the Deliverable

1. What is the deliverable?

2. What kind of deliverable (e.g., hardware, software, information, 

intellectual property, an organization, a building, system)?

3. What is the value of the deliverable?

4. What is the expected lifespan of the deliverable?

b) Utility of the Deliverable

1. What are the functions and capabilities of the deliverable? 

2. How efficient is the deliverable?

3. What are the applications and uses for the deliverable?

4. How adaptable is the deliverable?

2. Process Change 

(Describe how the system and its environment changed during the process.)

a) Internal

1. What are the changes that have taken place within the system?

2. What kinds of changes are these?

3. What is the scope of these changes?

4. How different is the system at the time of output compared with 

the time of input?

5. What is the likely impact of these changes?

b) External

1. What are the changes that have taken place within the environment?

2. What kinds of changes are these?

3. What is the scope of these changes?

4. How different is the environment at the time of output compared 

with the time of input?

5. What is the likely impact of these changes?
97

Appendix F—Outline for Lessons-Learned Case Assessment

Overview of Appendix F

This appendix provides an outline for writing the lessons-learned case assessment. You will 

notice that the appendix to the case assessment is the case description outlined in Appendix E 

to the monograph.

Abstract

(Draft a short one- or two-paragraph abstract to serve as a high-level summary.)

Executive Summary 

(Draft a two- to three-page summary of the assessment that identifies and explains the lessons.) 

The Lessons

(Start the executive summary with a short statement that summarizes the lessons.)

Outlook

(Include a brief summary of the outlook section from the main report.)

Main Report

(Draft the report with the main points up front. In other words, lessons first; explanations 

next; background last)

The Lessons

(State and explain what your lessons are about. Formulate your statement of the lessons in 

a behavior/consequence phraseology as explained earlier in this monograph.)

Outlook

(Discuss the significance of the lessons and what role their application would likely play on 

the organization’s outcome, also address the consequences of ignoring the lessons.)

Background

(Include any historical context in this section)

Appendix – The Case Description

(Incorporate the case description prepared for phase II of the research as an appendix.)
98

Index

A

Anchoring Errors 36-38, 41

Applying Lessons 42-43

Associational 12, 28

Assume Availability 33, 35

Authoritative Source 34-35

Avoiding Data Collection Errors 36

B

Behaviors 2, 4, 7, 11, 19, 43

C

Case Assessment 24, 32, 99

Case Description 24, 27, 30, 87

Case History 4

Case Study 3, 4, 5, 21, 23, 24

Case Study Framework 1, 23, 24

Case Study Models 5

Categorical Thinking Bias 36

Cognitive Dissonance Bias 36-37

Cognitive Errors 36, 41

Collecting Data 2, 8, 9, 15

Congruence 16

Congruence Theory 1, 2, 6, 15, 16, 22, 23, 24, 43

Contamination Errors 80

D

Data Analysis 6

Data Collection 1, 6-8, 20, 21, 24, 26-30, 33, 36, 40, 51-52

Data Collection Errors 33, 36

Data Collection Questions 29, 52

Data Errors 33, 35

Domains 12, 14, 16-18, 27-28, 30-31

Dyadic Relationships 18, 21

E

Environment 10-12, 14-16, 20-21, 28-30, 36, 39, 42, 54, 60, 67, 72, 78-79, 93, 98

Environmental Component 13, 20

Errors 30, 33, 35-37, 39-41, 43, 77-78, 80-83
99

F

False Memory 80-81

Flaws 2, 8-9, 30, 32-33, 37

Formal Organization 13, 19, 59, 93

H

Historical Domain 13, 56, 90

Hypothesis 6, 21-22, 31, 38-39, 41, 50, 81

I

Informal Organization 13, 19, 60, 94

Information Network 12, 14, 17-21, 30, 54, 89

Input Errors 78

Intelligence Community 1, 5, 10-11, 15, 20-21, 42, 53

Interview Agreement 70. See also Interview Consent

Interview Consent 68, 71-72

Interview Team 70

L

Lessons-Learned 4-7, 9, 12, 14-16, 20-25, 27-34, 36-37, 42-43, 50, 52, 69, 73, 75, 87, 99

Letter 68, 71, 75

Limitative Domain 20, 56

M

Medical Model 23, 24

Melding of Similar Events 80

N

Nadler & Tushman 12, 14-16, 19

Network of Individuals 13, 19, 62, 95

Noncongruence 22, 24, 30

O

Organizational Components 10, 60, 93

Override from New Information 80

P

Paradigm 1, 2, 6-7, 9, 23, 24, 43, 87

Pattern Fabrication 78-79

Process Component 13-14, 16-20, 30-31, 93
100

R

Rationality Assumption Error 36

Recall Errors 81, 83

Record Unavailable 34-35

Relational Domain 13, 54, 89

Research Design 24-25, 27, 50

Research Design Template 50

Research Problem 25, 27, 50

Roadmap for this Monograph 2

S

Search Satisfaction Error 36, 39

Social Science Research 1, 32, 33, 68, 77

Source Memory Failure 80

Subcomponent 17, 20

Subsystem 10, 17

T

Task 8, 12-13, 19, 21, 30-31, 37, 55-58, 61-65, 90-91, 94-97

Tribal-Moral Community Bias 37, 41, 49

U

Unconscious Cognitive Bias 78, 79

Unreliable Data 34, 35

W

Writing of the Research Design 25

Writing the Case Description 24, 30

Writing the Lessons-Learned Case Assessment 30, 32, 99
CENTER FOR THE STUDY OF

NATIONAL RECONNAISSANCE

NOVEMBER 2015