Technical Report 1167 

Concept Development for Future Domains: 

A New Method of Knowledge Elicitation 

Scott B. Shadrick and James W. Lussier 

U.S. Army Research Institute 

Robin Hinkle 

University of Louisville 

Consortium Research Fellows Program 

 June 2005 

 

 United States Army Research Institute 

for for the Behavioral and Social Sciences 

Approved for public release; distribution unlimited 
U.S. Army Research Institute 

for the Behavioral and Social Sciences 

A Directorate of the Department of the Army 

Deputy Chief of Staff, G1 

 ZITA M. SIMUTIS 

 Director 

Technical review by 

Angela Karrasch, U.S. Army Research Institute 

Paula J. Durlach, U.S. Army Research Institute 

NOTICES 

DISTRIBUTION: Primary distribution of this Technical Report has been made by ARI. 

Please address correspondence concerning distribution of reports to: U.S. Army 

Research Institute for the Behavioral and Social Sciences, Attn: DAPE-ARI-MS, 

2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926 

FINAL DISPOSITION: This Technical Report may be destroyed when it is no longer 

needed. Please do not return it to the U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

NOTE: The findings in this Technical Report are not to be construed as an official 

Department of the Army position, unless so designated by other authorized documents. 
i 

REPORT DOCUMENTATION PAGE 

1. REPORT DATE (dd-mm-yy)

 June 2005

2. REPORT TYPE 

 Final 

3. DATES COVERED (from. . . to) 

 March 2004 - January 2005 

4. TITLE AND SUBTITLE 5a. CONTRACT OR GRANT NUMBER

Concept Development for Future Domains: A New 

Method of Knowledge Elicitation 5b. PROGRAM ELEMENT NUMBER 

 622785 

5c. PROJECT NUMBER 

 A790 

6. AUTHOR(S) 

Scott B. Shadrick, James W. Lussier (U.S. Army Research 

Institute), and Robin Hinkle (University of Louisville) 5d. TASK NUMBER 

 211 

5e. WORK UNIT NUMBER 

7. PERFORMING ORGANIZATION NAME(S) AND 

ADDRESS(ES) 

U.S. Army Research Institute for the Behavioral and Social 

Sciences 

ATTN: DAPE-ARI-IK 

2423 Morande Street 

Fort Knox, KY 40121-5620 

8. PERFORMING ORGANIZATION REPORT 

NUMBER 

10. MONITOR ACRONYM

ARI 

9. SPONSORING/MONITORING AGENCY NAME(S) AND 

ADDRESS(ES) 

11. MONITOR REPORT NUMBER 

 Technical Report 1167 

12. DISTRIBUTION/AVAILABILITY STATEMENT 

Approved for public release; distribution is unlimited. 

13. SUPPLEMENTARY NOTES 

Subject Matter POC: Scott B. Shadrick 

14. ABSTRACT (Maximum 200 words): 

During the development of operational concepts for the Future Combat System of Systems’ Unit of Employment and 

Unit of Action it became clear that the Army needed a more effective and efficient method for envisioning the future. 

A review of existing knowledge elicitation techniques indicated that current methods are not easily applied to the 

development of future concepts. They do not adequately address the cognitive impact new concepts and technology 

have on Soldiers. In addition, existing methods do not thoroughly examine the potential unforeseen impacts the 

introduction of the new technology will have. The inherent difficulty in envisioning new concepts requires a more 

systematic approach to elicit knowledge from domain experts. This report reviews existing methods and describes a 

new method of knowledge elicitation to more effectively support the development of future concepts, evaluate the 

impact of new technology, and solve difficult problems where information and expertise is dispersed among many 

individuals. 

15. SUBJECT TERMS 

Concept Development; Knowledge Elicitation; Knowledge Acquisition; Cognitive Task Analysis 

SECURITY CLASSIFICATION OF 

16. REPORT

Unclassified 

17. ABSTRACT

Unclassified 

18. THIS PAGE 

Unclassified 

19. LIMITATION 

OF ABSTRACT

Unlimited 

20. NUMBER

OF PAGES 

21. RESPONSIBLE PERSON 

 

Ellen Kinzer Technical Publication Specialist

703-602-8047

U.S. Army Research Institute for the Behavioral and Social 

Sciences 

 2511 Jefferson Davis Highway 

Arlington, VA 22202-3926 
ii 
Technical Report 1167 

Concept Development for Future Domains: 

A New Method of Knowledge Elicitation 

Scott B. Shadrick and James W. Lussier 

U.S. Army Research Institute 

Robin Hinkle 

University of Louisville 

Consortium Research Fellows Program 

Armored Forces Research Unit 

Barbara A. Black, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences 

2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926 

June 2005 

Army Project Number Personnel Performance and 

622785A790 Training Technology 

Approved for public release; distribution unlimited

iii 
 

iv 
FOREWORD 

As new technologies, concepts, and doctrine are introduced into the Army’s operational 

systems there is a corresponding change to the cognitive processes and procedures used to 

effectively adapt to the “new way of accomplishing a task.” A serious challenge to the U.S. 

Army transformation effort is the ability to develop realistic concepts, systems, and technologies 

for the future. Effective, systematic techniques of concept development and knowledge 

exploration for envisioning future systems and operations are currently not available. 

Understanding the dynamic interactions between people, technology, and tasks; 

realistically being able to predict new technologies and their impact on cognitive processes; and 

predicting and anticipating potential challenges and unintended side effects will be key enablers 

for the transformation process. All too frequently new technologies are introduced without even 

a basic understanding of these relationships and the resulting systems are too burdensome to the 

Soldier or far too cognitively demanding to use during the stress of battle. This report reflects 

ongoing work by the U.S. Army Research Institute for the Behavioral and Social Sciences (ARI) 

as part of its Science and Technology Objective: IV.SP.2002.02, Methods and Measures of 

Commander-Centric Training to develop new methods to enhance the U.S. Army’s ability to 

produce new concepts and training needed to support the transformation effort. The techniques 

outlined in this report are being applied to ongoing research efforts. 

The findings of this research have been briefed to representatives of the Unit of Action

Maneuver Battle Lab (UAMBL), to Fort Knox officers who serve as Future Force Unit of Action 

Command and Staff in UAMBL exercises, to the Unit of Action Experimental Element, and to 

members of the Lead Systems Integrator Training Integrated Project Team. 

 MICHELLE SAMS

 Technical Director 

v 
vi 
CONCEPT DEVELOPMENT FOR FUTURE DOMAINS: A NEW METHOD OF 

KNOWLEDGE ELICITATION 

EXECUTIVE SUMMARY 

Research Requirement: 

The U.S. Army is in the process of transforming to a Future Force with increased reliance 

on technology. The transformation requires the development of new concepts, processes, 

procedures, doctrine, knowledge, and ideas. During the development of operational concepts for 

the (FCS) Unit of Employment and Unit of Action it became clear that the Army needed a more 

effective and efficient method for envisioning the future. Traditional methods of concept 

development and knowledge exploration are limited in their ability to generate realistic and 

reliable concepts about the future. While an impressive list of candidate technologies and 

concepts can be conceived, the Army cannot afford to become too optimistic about their 

influence without having a method to appropriately evaluate their impact. There is a tremendous 

need for a powerful, yet flexible, approach to concept development and knowledge exploration 

that can cope with the unique difficulties of envisioning and studying future concepts. 

Procedure: 

U.S. Army Research Institute for the Behavioral and Social Sciences (ARI) examined 

methods to generate, refine, test, and validate new concepts related to doctrine, tactics, 

techniques, procedures, and technology integration. The present report reviews existing methods 

of concept development and knowledge exploration and analyzes the appropriateness of those 

methods for envisioning future concepts. A new method of concept development and knowledge 

elicitation that allows for the systematic investigation of future concepts is proposed. 

 

Findings: 

The current research concluded that existing methods of concept development and 

knowledge elicitation are not appropriate and are not effective for examining future issues. The 

inherent difficulty in envisioning new concepts is explored and existing methods are described. 

A new method, incorporating components of existing methods, was developed to provide a 

potentially powerful approach to systematically investigate future concepts. Information on 

implementing the method and understanding the impact of future concepts and technology is 

provided. 

Utilization and Dissemination of Findings: 

The findings of the research can benefit personnel in the U.S. Army Training and 

Doctrine Command, the Unit of Action Maneuver Battle Laboratory, and other agencies 

involved in concept development, the development of new doctrine, and the elicitation of 

vii 
realistic ideas for the Future Force. Further, the outcomes of the research have been applied to 

the development of new training methods for the U.S. Army and the U.S. Army National Guard.

viii 
CONCEPT DEVELOPMENT FOR FUTURE DOMAINS: A NEW METHOD OF 

KNOWLEDGE ELICITATION 

CONTENTS

 

 Page 

Introduction................................................................................................................................ 1 

Importance of Knowledge Elicitation........................................................................................ 3 

Current Army Methods for Concept Development and Knowledge Elicitation............ 3 

Systematic Methods of Analysis.................................................................................... 4 

Methods of Task Analysis ......................................................................................................... 4 

 

Knowledge Elicitation ............................................................................................................... 8 

 Communication Techniques and Settings...................................................................... 8 

 Elicitation Techniques ................................................................................................... 9 

 

Knowledge Creation .................................................................................................................. 11 

 Environmental Scanning................................................................................................ 11 

 Technological Forecasting............................................................................................. 11 

 Backcasting.................................................................................................................... 12 

 Enabling Knowledge Creation ...................................................................................... 13 

A New Method........................................................................................................................... 14 

 Phase 1: Domain and Problem Identification ............................................................... 15 

 Phase 2: Initial Review and Analysis............................................................................ 16 

 Phase 3: Refine Problem Space and Develop Initial Scenario ..................................... 16 

 Phase 4: Initial Knowledge Elicitation ......................................................................... 17 

 Phase 5: Data Reduction and Consensus Building ....................................................... 17 

 Phase 6: Knowledge Representation and Concept Documentation.............................. 17 

Using Experts for Knowledge Elicitation and Creation ............................................................ 18 

Conclusions................................................................................................................................ 19 

References.................................................................................................................................. 21 

Appendix A............................................................................................................................... A-1 

ix 
CONTENTS (Continued) 

 Page 

List of Figures 

Figure 1. Listing of Task Analysis Methods............................................................................. 6 

Figure 2. Listing of Cognitive Task Analysis Methods............................................................ 7 

Figure 3. Development Process for New Knowledge and Concepts........................................ 13 

Figure 4. Concept Development Process.................................................................................. 14 

Figure 5. Overview of the FLEX Process................................................................................. 16 

Figure 6. Task Oriented Question Construction Wheel Based on Bloom’s Taxonomy........... 20 

x 
CONCEPT DEVELOPMENT FOR FUTURE DOMAINS: 

A NEW METHOD OF KNOWLEDGE ELICITATION 

Introduction 

The U.S. Army is currently in the process of transforming into a strategically responsive 

force capable of rapid deployment and effective operations in all types of military missions. 

Under the transformation effort, the Army is changing the way it equips, organizes, deploys, and 

trains. The cornerstone of the transformation is the Future Force and the (FCS). The FCS is 

envisioned to be a distributed, network-centric, multi-mission combat system that will utilize 

advanced, state-of-the-art technologies. To support the transformation effort the Army is 

developing a family of 18 networked war-fighting systems which are more lethal, survivable, 

deployable and sustainable than the existing arsenal of Army systems. The FCS is envisioned to 

be equipped with a combination of manned, robotic, and mixed (man and robotic) human-in-theloop equipment. 

It is predicted that the network of systems will allow unprecedented information transfer. 

The extensive amount of information will attempt to compensate for the loss of armor afforded 

by current systems and is assumed to allow for superiority and synchronized operations to see, 

engage, and destroy the enemy before the enemy detects the Future Force. It is predicted that the 

network centric approach will allow for an improved integration of communication across and 

within echelons. Further, the FCS, and its accompanying Future Force, is envisioned to operate 

as part of a combined, joint, interagency, and multinational team. The key enablers of this vision 

for information superiority are the rapid technological advances and the integration of 

information across the military. One of the most challenging problems facing the Army is to 

determine how to integrate and capitalize on future capabilities without having a clear 

understanding of what those capabilities are or how they should be implemented. 

The standard belief that the introduction of new technology is a simple process that will 

result in improved organizational performance is an oversimplification fallacy (Feltovich, Spiro, 

& Coulson, 1997; Woods & Dekker, 2000). With regard to technological advancement in the 

U.S. Army, Wass de Czege and Biever (1998) highlight this point suggesting that “combat 

power is not simply the sum of machine performance; it requires individual and organizational 

competence and synergy” (p. 19). Thus, a change in technology produces a corresponding 

change in the operational and cognitive systems – resulting in the transformation of existing 

roles, processes, and procedures and the development of new ones. 

Any assertion concerning the impact of future technologies represents a hypothesis 

regarding the relationship and dynamics of the new technology to people and existing systems 

(Woods, 2002; Woods 1998). The hypotheses, once observed and evaluated, often prove to be 

incorrect once the second order effects of the system are realized (Sarter, Woods, & Billings, 

1997). Thus, as Woods (2002) suggested, “envisioning the future of operations, given the 

dynamic and adaptive nature of the process, is quite fragile” (p. 2). 

Even if the new technology is successfully integrated, the technology, in and of itself,

does not allow for improved performance. In fact, the insertion of new technology may increase 

1
cognitive demands (Howell & Cooke, 1989). The technology must be coupled with 

corresponding changes to all facets of the system. The point was made vividly clear by 

Cordesman and Wagner (1996) in their review of the Gulf War. The authors stated, 

Much of the equipment deployed in U.S., other Western, and Saudi forces was 

designed to ease the burden on the operator, reduce fatigue, and simplify the task 

involved in combat. Instead, these advances were used to demand more from the 

operator. 

Almost without exception, technology did not meet the goal of unencumbering 

the military personnel operating the equipment, due to the burden placed on them

by combat. As a result, weapons and support systems often required exceptional 

human expertise, commitment, and endurance. The Gulf War shows that there is 

a natural synergy between tactics, technology, and human factors and effective 

military leaders will exploit every new advance to the limit. As a result, virtually 

every advance in ergonomics was exploited to ask military personnel to do more, 

do it faster, and do it in more complex ways. 

This stress occurred whether the issue was flying and sustaining higher sortie 

rates, conducting faster armored maneuvers over longer periods of time, targeting 

and firing at longer ranges, or increasing the speed of logistic support. Wars may 

occur in which one side is so superior that it can rely on tactics and technology 

without a matching improvement in manpower, and without stressing human 

factors to their limit. Such wars, however, are unlikely to involve significant 

levels of conflict, and the idea that technology can reduce the stress of combat or 

the need for manpower quality and readiness is almost certain to be a myth. 

While there is no way to prove such a conclusion, one very real lesson of the Gulf 

War is that new tactics and technology simply result in altering the pattern of 

human stress to achieve a new intensity and tempo of combat. (p. 25) 

The future of U.S. Army dominance will not continue to be as dependent on the 

Department of Defense (DoD) designing, building, and protecting unique technologies. Instead, 

dominance will be maintained by designing, implementing and protecting the world's best 

technology integration process (Carter & White, 2000). This is a different way of doing 

business. It is predicated on the notion that the U.S. will be the fastest integrator of new 

technologies – into the hardware, into the software, into our doctrinal and organizational 

systems, and into our training and leader development process. A key to success is a solid Army

process for concept development – a means to generate, elaborate, refine, describe, test, and 

validate new Future Force concepts relating to doctrine, tactics, techniques, procedures, unit and 

team organization, job allocation, training, leader development, and other aspects of technology 

integration. 

How do we best use the total Army team – warfighter, scientist, theorist, analyst, and 

engineer – in the development process of future concepts? There is a critical need to create the 

methods needed to systematically extract critical information (knowledge elicitation) about the 

future issues in order to more effectively and efficiently manage the concept development 

2
(knowledge creation) and integration process. This report addresses that need by reviewing 

existing methods and documents a new, systematic method of knowledge elicitation to address 

the Army’s need for a formalized process for concept development. 

Importance of Knowledge Elicitation 

Expert knowledge involves the reasoning, judgment, and performance of individuals who 

are unusually accurate, skillful, and reliable in their domain. The knowledge is mostly heuristic, 

based on experience, and inaccessible through standard observational methods. Knowledge 

elicitation describes a process in which a researcher or analyst captures knowledge, procedures, 

and strategies through systematic interaction with domain experts for the purposes of building 

expert models and developing expert reasoning strategies. The focus on expert knowledge has 

three functions: (a) to improve and support human performance in decision making tasks; (b) to 

design instruction to efficiently and efficiently train individuals; and (c) to automate task 

functions that consist of human logic processes. 

Traditional methods of knowledge elicitation often rely on direct observation of expert 

behaviors. The outcomes of these elicitation efforts can be decomposed to describe knowledge 

that is linked to observable behaviors within a specific domain and context. In addition, other 

methods have been developed to capture internal cognitive processes that are not readily 

accessible by observing behavior. While much research has been conducted on concept 

development and knowledge elicitation, the acquisition of expert knowledge still remains one of 

the most time-consuming, tedious, and essential tasks in designing systems (Chervinskaya & 

Wasserman, 2000). 

 

Concept development for the Army Future Force involves future systems and developing 

technological capabilities. It is concerned with future operational capabilities, training, and 

doctrine; it addresses how those systems should be developed and how they will be employed. 

When the goal of knowledge elicitation is future concept development, experts are required to 

generate best-guess estimates based on an anticipated set of new capabilities. In other words, 

while the participants in the knowledge elicitation exercise may be experts in their current 

domain, they will be asked to theorize and reason on how things might be different under various 

projected future conditions. How does knowledge generalize to future systems where detailed 

specifications of developing technology are not yet known? How does expertise in a current 

domain inform the development of concepts for future products and practices? 

Current Army Methods for Concept Development and Knowledge Elicitation 

Generally, the Army has used two methods to elicit new knowledge and ideas from

experts. The first is to have a strong theorist or analyst develop and present new concepts, i.e., 

an Army thinker thinks and then writes. A good example of this is Conceptual Foundations of a 

Transformed U.S. Army by Wass de Czege and Sinnreich, (2002). While strong creative thinkers 

are clearly a vital part of the effort, some drawbacks are evident in the ‘just let thinkers think’ 

method of concept development. The results are often very general, have not benefited from

concrete attempts to implement the ideas, have not uncovered difficulties through instantiation, 

can be hard to communicate, and do not easily avail a team of specialized contributors working 

3
jointly. Several efforts less successful than the above-cited example could be mentioned with 

regard to Future Force concepts, producing documents that are poorly thought out, too general, 

impractical, unrealistic, fragmented, disjointed and contradictory ‘cut-and-paste’ efforts. 

A second Army method is to stand up a replica of the new system and conduct a unit 

exercise in simulation, such as a focused Combat Training Center (CTC) rotation or Advanced 

Warfighter Experiment (AWE). Such full-scale events can be effective demonstrations of 

moderate-to well-developed concepts but have drawbacks, especially in the earlier stages of

concept formation. They are expensive and require great effort and coordination to conduct, do 

not allow flexible manipulation of variables, present great measurement difficulties, and lack the 

repetition necessary to reach well-founded conclusions. Further, the complexity of conducting 

the event tends to overwhelm the experimental intent. 

A limitation of both approaches is that they often do not account for the realization that

the new concept, technology, or process will likely transform what it means to be an expert in the 

field. As Woods and Dekker (2000) suggest, “introducing new technology is not manipulating a 

single variable, but a change that reverberates throughout a system transforming judgments, 

roles, relationships, and weightings on different goals” (p. 5). There is a need to investigate 

intermediate concept development and exploration methods to augment the two extremes 

described above, methods that provide structured activities to measure, assess, and guide the 

concept development process yet are flexible enough to respond rapidly to a wide range of 

conceptual constructions. 

Systematic Methods of Analysis 

Systematic methods for knowledge acquisition and concept development are available. 

These methods have been developed based on cognitive and applied behavioral science 

principles and provide structured processes by which critical information about task knowledge 

and performance can be identified. 

Methods of Task Analysis 

Task analyses are systematic approaches to understanding, describing, and defining a job 

in terms of the physical and cognitive processes needed to successfully perform particular tasks. 

That is, task analytic methods help to identify the knowledge, skills, abilities, and other 

characteristics required to perform a job. They address the frequency, difficulty, criticality, 

conditions, performance standards, and importance of specific tasks. This report makes a 

distinction between traditional behavioral approaches to task analysis and cognitive task analysis. 

 Traditional task analysis focuses on the way a task is performed – the task is directly 

observable with a clear beginning and ending point (Department of Defense, 2001). Several 

methods have been used to gather information in traditional task analytic investigations. The 

most frequently used method to capture task information is through the use of individual and/or 

group interviews with subject-matter experts. Task analysis information is also collected using 

questionnaires and direct observation of incumbent subject-matter experts. 

4
McCormick, Jeanneret, and Mecham (1972) make a distinction between job- and workeroriented methods stating, “The job-oriented concept typically would be reflected by the use of 

specific task statements . . . the worker-oriented concepts typically would be reflected by the use 

of descriptions of reasonably definitive human behaviors” (p. 348). 

- Job-oriented task analysis focuses on what the task involves regarding the activities, 

tools, products, and outcomes of the task. Job-oriented task analysis is a systematic process for 

breaking tasks down into discrete step-by-step segments. Typically, this is accomplished by 

interviewing experts and by observing as the tasks are completed. The outcome of a job-oriented 

task analysis is a list of tasks and subtasks such as can be found in an Army Training and 

Evaluation Program (ARTEP). For example, the task Control Tactical Operations is subdivided 

into eight subtasks: Manage CS/CSS Force Positioning, Manage Use and Assignment of 

Terrain, Maintain Synchronization, Control Tactical Airspace, Plan Actions, Sequels, and 

Branches, Make Adjustments to Resources, Concept of Operations, or Mission, and 

Coordinate Actions to Produce Maximum Effective Application of Military Power (ARTEP 3-

91, Draft). Techniques classified as job-oriented task analysis methods include the Functional 

Job Analysis (Fine & Wiley, 1971), Comprehensive Occupational Data Analysis Program

(Christal & Weissmuller, 1976), and the Task Inventory Analysis. 

Worker-oriented task analysis focuses on requisite characteristics of a subject-matter 

expert or job incumbent expressed in terms of knowledge, skills, abilities, and other 

characteristics. Worker-oriented task analysis techniques are typically used to identify the 

activities required to complete a task and does not specifically identify what the individual 

actually does (Burnett & McCracken 1982). Techniques classified as worker-oriented task 

analysis methods include the Position Analysis Questionnaire (McCormick, Jeanneret, & 

Mecham, 1972), Critical Incident Techniques, Occupational Analysis Inventory (Cunningham,

Boese, Neeb, & Pass, 1983), Common-Metric System (Harvey, 1993), and the Job Element 

Method (Primoff & Eyde, 1988). Figure 1 provides a more comprehensive list of task analytic 

methods. Please see Kirwan & Ainsworth (1992) and Jonassen, Hannum, & Tessmer (1989) for 

a more extensive review of methods. 

Job- and worker-oriented methods are very effective in specifying job requirements, 

characteristics, and system functions for observable jobs or sets of task. They are not, however, 

effective or efficient in systematically developing an understanding of cognitive processes or 

knowledge requirements. For example, the tasks and subtasks listed above from Army ARTEP 

3-91 do not indicate anything about the mental processes experts go through to perform the task 

well. As Gott (1998) stated, “Complex job environments require deeper cognitive analyses that 

can ferret out the conceptions and reasoning processes that lurk behind observable behaviors” (p. 

11). Cognitive task analysis provides a systematic process for identifying the cognitive elements 

and activities needed for task performance and is useful when there are few readily observable 

behaviors. The focus is on knowledge and cognitive processes as opposed to a breakdown of 

procedural steps. 

5
Ability Requirements Scales 

Activity Sampling 

Behavioral Task Analysis 

Checklist 

Common Metric Questionnaire 

Common-Metric Systems 

Critical Incident Technique 

Decision-Action Diagramming 

Decomposition Method 

Fleishman Job Analysis Survey 

Focus Groups 

Functional Analysis System Technique 

Functional Job Analysis Scales 

Goal Composition 

Groupware Task Analysis 

Heuristic Task Analysis Process 

Integrated Performance Modeling 

Environment 

Interface Survey 

Interviews 

Job Components Inventory 

Job Element Inventory 

Job Elements Method 

Link Analysis 

Multipurpose Occupational Systems 

 Analysis Inventory-Closed Ended 

Nominal Group Technique 

Observation

Occupational Analysis Inventory 

Operational Sequence Diagrams

Operator Function Model 

Petri-nets 

Position Analysis Questionnaire 

Procedural Task Analysis 

Questionnaires 

Scenario-Based Analysis 

Sequential Task Analysis 

Signal Flow Analysis 

Table Top Analysis 

Task Decomposition Techniques 

Task Inventory/Comprehensive 

 Occupational Data Analysis Programs

Threshold Traits Analysis (TTA) 

Timeline Analysis 

Trigger Analysis 

Verbal Protocols 

Walk-Through Analysis 

Work Profiling System (WPS) 

Workload Analysis 

Figure 1. Listing of Task Analysis Methods. 

There are many methods of cognitive task analysis, each with unique elements designed 

for various settings and objectives. All the methods typically involve three phases: a description 

of the task environment, identification of related tasks and processes, and the elicitation of 

knowledge (DuBois, Shalin, Levi, & Borman, 1995). One example is called the critical decision 

method. In this method a facilitator asks experts to recall a salient incident or cases. In most 

situations, the case identified represent non-routine incidents where expert performance can be 

highlighted. The incidents become the focus of semi-structured interviews where cognitive 

probes are used to understand the experts’ situation assessment and decision making processes. 

Appendix A provides a brief description of some major cognitive task analysis techniques. 

Appendix A is not intended to be an exhaustive presentation of such methods. Rather, the 

examples were chosen to represent the variety of methods currently employed. Figure 2 

provides a more complete list of methods and techniques. Please see Schraagen, Chipman, & 

Shalin (2000) for an extensive review of cognitive task analysis methods. 

 

6
Applied Cognitive Task Analysis 

Active Participation 

Activity Sampling 

Behavioral Descriptions 

Card Sorting 

Cloze Technique 

Cluster Analysis 

Cognitive Structure Analysis 

Collect Artifacts of Task Performance

Concept Mapping 

Conceptual Graph Construction 

Critical Decision Method

Critical Incident Method 

Critiquing 

Data Flow Modeling 

Decompose, Network, and Assess Method 

Decision Analysis 

Diagramming 

Discourse Analysis 

Distinguishing Goals 

Dividing the Domain 

Document Analysis 

Entity Life Modeling 

Entity-Relationship Modeling 

Ethnographic Methods 

Exploratory Sequential Data Analysis 

Field Observations 

Forward Scenario Simulation 

Goal Decomposition 

Goal Directed Task Analysis 

Goals, Operators, Methods, Selection Rules 

Group Interview 

Hierarchical Sort 

IDEF Modeling 

Influence Diagram Construction 

Interesting Cases 

Interruption Analysis 

Interviewing (Structured, Unstructured, 

Semi-Structured) 

Laddered Grid 

Likert Scale Elicitation 

Magnitude Estimation 

Minimal Scenario Technique 

Multidimensional Scaling 

Object Oriented Modeling 

Observation

On-Site Observation 

P Sort 

Petri Nets 

Precursor, Action, Results, & Interpretation 

Problem Analysis 

Problem Discussion 

Protocol Analysis 

Proximity Scaling 

Q Sort 

Questionnaire 

Rapid Prototyping 

Rating and Sorting Tasks 

Reclassification 

Repertory Grid 

Retrospective Case Description 

Role Playing 

Semantic Nets 

Shadowing (Self/Others) 

Simulators and Mockups 

Step Listing 

Storyboarding 

Structural Analysis Technique 

Stumbling Block Technique 

System Examination 

System Refinement 

System Validation 

Table-Top Analysis 

Task Action Mapping 

Task Knowledge Structures 

Teachback 

Think Aloud 

Tutorial Interview 

Twenty Questions 

Uncertain Information Elicitation 

User Needs Analysis 

Wizard of Oz Technique 

Figure 2. Listing of Cognitive Task Analysis Methods. 

7
With its concentration on non-observable cognitive processes, cognitive task analysis is 

more amenable to the investigation of future systems and processes. Specifically, those methods 

developed for the knowledge elicitation phase hold considerable promise for improving our 

ability to create new concepts and develop future systems. Components of various cognitive 

methods that extend traditional approaches for describing current systems, while not specifically 

designed for such purposes, can be used to systematically capture, analyze, test, and evaluate 

future constructs. The following section will provide more detail on knowledge elicitation 

techniques and methods. 

Knowledge Elicitation 

Knowledge elicitation methods are a key component of the information gathering stage of 

task or cognitive-task analysis – they provide the mechanism by which knowledge is transferred 

from the expert to the analyst. The methods attempt to provide a convenient way to accurately 

and easily allow experts to communicate their expertise. Despite the abundance of elicitation 

techniques available, capturing expert knowledge and understanding how the knowledge relates 

to problem solving capabilities remains an obstacle for expert system development (Moody, 

Blanton, & Will, 1998/1999). Often, the knowledge elicitation used to develop expert systems is 

vague or glossed over because it is largely ad hoc and non-scientific (Wright & Ayton, 1987). 

Klein, Calderwood, and MacGregor (1989a) point out a tendency to emphasize explicit 

knowledge (also called declarative knowledge) over implicit knowledge (also called tacit 

knowledge). Explicit knowledge is information that is easily accessed, expressed, stored, and 

applied. Implicit knowledge, on the other hand, is information that exists in the experts’ 

unconscious and is not easily accessible. Experts, through years of practice, develop automatic 

habits that are characteristic of expert performance. These unconscious chunks of knowledge 

usually go to the heart of what makes someone an expert. Effective knowledge elicitation 

techniques for determining the unique contributions of both implicit and explicit knowledge 

should be used. Knowledge elicitation methods that overemphasize explicit knowledge lead to 

“the mistaken conclusion that explicit knowledge is sufficient for performing a task well” (Klein, 

et al., 1989a, p. 463). 

The knowledge elicitation technique should provide the expert with an appropriate set of 

procedures for transferring information to the researcher or analyst. In planning knowledge 

elicitation, there are several decisions that must be made to ensure that the process is effective 

and the results are reliable and valid (Meyer & Booker, 1991). The researcher must determine 

the setting in which the expert judgment is to be gathered and select a form of communication 

for asking and answering questions. The elicitation technique chosen determines how the 

experts will respond and how their responses will be coded. Often it is important to allow future 

participants to ‘see’ the thought processes of previous experts. Finally, a method for aggregating 

responses from multiple experts into a single representation is needed for the researcher to 

communicate and describe the results. 

Communication Techniques and Settings 

The setting describes the arrangement of the meeting between researcher and experts. 

Several broad categories have been used to effectively elicit knowledge including private 

interviews, group interviews, interactive group discussions, and remote response reception. 

8
These settings can be combined or modified as appropriate. The use of multiple settings may be 

especially important when collecting observations about future concepts and situations. Detailed 

tacit knowledge can be collected from an expert from a specific domain area during a private 

interview. Group interviews, however, allow an expert from one domain area to interact with 

experts from other domains. Combining the perspective from different domain areas may be 

critical for the analyst to truly understand goals, technological relationships, interconnections, 

and ideas concerning the future. 

Private interviews should be semi-structured and can be used to develop initial concepts 

and ideas that are unimpeded by the inherent limitations of interactive groups. The results from

the private interviews feed the interactive group discussions; including face-to-face meetings

including two or more experts from multiple domain areas. The interactive group discussion 

technique originated in marketing as the focus group and can be either structured or unstructured. 

Meyer & Booker (1991) claim that interactive groups produce a larger amount of data with 

greater predictive value than do individual interviews or the Delphi method (described below) 

alone. Feedback from other participants stimulates conversation, resulting in deeper and richer 

data. A possible problem with this method is groupthink, a frequent tendency of groups to be 

less critical in their evaluation and not question poor decisions. Used in conjunction with 

personal interviews, the interactive group discussions allow expert participants to debate various 

topics and develop a consensus by aggregating results. 

Elicitation Techniques 

The selection of a knowledge elicitation method for a specific domain involves several 

considerations. Criteria for selection depend upon the researcher’s level of familiarity with the 

domain and the amount of time and money required for the analysis. Knowledge elicitation 

methods are typically divided into two broad categories, direct and indirect. Direct methods 

involve the researcher asking domain experts to tell how they perform their job. Examples 

include the various forms of interviewing, protocol analysis, simulation, concept mapping, and 

the Critical Decision Method (CDM). The effectiveness of direct methods depends upon the 

articulation and cooperation of the experts and their ability to verbalize the information. 

Typically experts are easily able to verbalize declarative knowledge. Declarative knowledge, 

also called explicit knowledge, is represented by classifications and relationships. It focuses on 

descriptions of facts, things, methods, or procedures. Although facts and concepts can be 

expressed verbally, the interview process may not always yield patterns and structures 

discernable by the researcher (Evans, 1988). 

Indirect methods are thought to be more suitable when knowledge is not easily expressed 

by the expert. Tacit knowledge, which has been learned implicitly through experience, and 

overlearned automatic procedural knowledge can often be difficult for experts to express. 

Indirect methods constrain the experts to state their knowledge with the help of predefined 

structures such as, repertory grids (Kelly, 1955; Gaines & Shaw, 1992), decision trees, card 

sorting (Geiwitz, Kornell, & McCloskey, 1990; Major, 1991), and laddering techniques 

(Geiwitz, et al., 1990; Rugg, Mcgeorge, & Shadbolt, 1990; Cordingley, 1989). It is worth 

noting, however, that direct methods such as CDM have also been used successfully to elicit tacit 

knowledge. In addition, indirect methods are capable of complementing direct methods. 

9
Due to the complexity of most domain areas and the difficulties associated with 

forecasting future capabilities and requirements a combination of several methods should be 

employed to collect information from knowledgeable experts. The next sections will briefly 

highlight three of the most popular methods of knowledge elicitation. As with the review of 

cognitive task analysis, the review is not intended to be exhaustive. The intent is to provide an 

overview of a few of the most widely used methods which contribute to the development of a 

new method of elicitation focused on the elicitation of knowledge for use with future situations 

and concept development. 

The Delphi Technique. The Delphi technique was originally developed by the RAND 

Corporation to provide a method to elicit expert knowledge and develop group consensus among 

participants (Dalkey, 1969). In traditional Delphi approaches, experts do not interact directly 

with one another and only interact with the researcher in a limited manner. The researcher 

gathers responses, makes them anonymous, and provides them to other expert participants. After 

reviewing the responses provided by other participants, each expert submits a revised response. 

The process is repeated as many times as necessary. The method was designed to avoid the 

groupthink bias of interactive groups. The objective of bias avoidance is met if, as experts revise 

their answers, they are converging on the correct answer. Difficulty arises in assessing the 

quality of the answer and, thus, whether groupthink existed in reaching consensus (Meyer & 

Booker, 1991). Traditionally, the Delphi method used the standard postal mail system to 

distribute and collect responses; now, however, electronic-mail is usually used instead. 

Interviews. Interviewing involves the researcher asking the expert a series of questions 

such as to recall and describe the steps taken in performing a task. Sometimes the expert is 

asked to differentiate between or give examples of terms, concepts, or events. Interviews can be 

unstructured, structured, or semi-structured. In unstructured interviews, the researcher asks 

open-ended questions about the expert’s reasoning in making decisions and records the 

responses. This interview method allows the researcher to become familiar with jargon and gain 

an overview of the domain. The major disadvantage of unstructured interviews is that disorder 

can ensue if the expert speaks off topic or erroneously assumes that the researcher is 

knowledgeable in the domain (Hoffman, Shadbolt, Burton, & Klein, 1995). Structured 

interviews are preplanned to reduce the time length of the interviewing process by focusing the 

expert only on specific questions. Planning structured interviews requires that the researcher

possess some knowledge of the domain. Semi-structured interviews are often the preferred style 

as they allow the researcher to add questions to clarify points and to omit questions that become

redundant or irrelevant as the interview evolves. The technique promotes more continuity in the 

data than unstructured interviews. Continuity assists in the comparison and aggregation of 

responses from various experts. By asking the expert to explain the rationale or assumptions of 

the response, researchers ensure that each expert is responding to the same question. Compared 

to structured and unstructured interviews, semi-structured interviews are more likely to produce 

data that is germane without the inefficiencies of collecting unnecessary data (Meyer & Booker, 

1991). 

Protocol Analysis. Protocol analysis involves the researcher recording the expert as the 

task is completed. Asking the expert to actually perform tasks will often generate more valid 

knowledge than asking experts to simply describe the steps required by asking probing questions 

10
(Wright & Ayton, 1987). In addition, the expert is required to ‘think aloud’ while working 

through the problem or situation. The purpose is to identify knowledge elements and steps 

required for problem solving in the given domain area. The method has the advantage of 

allowing the expert to perform the task in a work context while verbalizing the cognitive 

activities required for success. One limitation of the method is that it is based on introspection 

and bias to the problem-solving processes (Wright & Ayton, 1987). If the task is one that can be 

performed automatically, then experts are often able to introspect while they perform. If, 

however, the task has a substantial cognitive component, the think aloud requirement may 

interfere with task performance. However, this limitation can be negated by video recording the 

expert during task performance and replaying the video during an interview session. This 

process is called retrospective protocol analysis and improves the amount and quality of tacit 

knowledge elicited. 

Knowledge Creation 

To create reliable knowledge it is not sufficient to elicit information from a group of 

domain area experts, it is necessary to prod them to go beyond what they already know. When 

developing concepts about the future, researchers should conduct a thorough review of current 

trends. Three methods are useful in this regard: environmental scanning, backcasting, and 

technological forecasting. The methods produce insights that guide decision making and are 

invaluable to the development of realistic scenarios that can be used to elicit knowledge and 

develop new concepts. 

Environmental Scanning 

In order to reliably investigate future issues and scenarios, a firm understanding of the 

current environment is needed. Environmental scanning is a systematic method of 

accomplishing this. In this context, environmental scanning is defined as the acquisition, 

compilation, synthesis, analysis, and utilization of information, knowledge, trends, and 

relationships concerning the internal and external environment which would assist in selecting, 

planning, and implementing a realistic course of action (Aguilar, 1967). Coates (1985) describes 

the objectives of environmental scanning as: detecting scientific, technical, social, political, and 

economic trends relevant to the problem under investigation, defining potential threats, 

opportunities, and challenges implied by those trends, enhancing future-focused thinking, and 

identifying trends that are converging, diverging, impending, distant, or interacting. 

Technological Forecasting 

Technology is often a key driver in the development of any new product or system. This 

is particularly the case where military systems are concerned. The ability to accurately predict 

the availability of a given technology will have a critical impact on the success of a given 

program, project, or system. Technological forecasting, as the name suggests, is interested in 

forecasting the types of technologies that will be available in a future time period, the 

characteristics of those technologies, and a realistic estimate of their availability. 

There are two primary methods of technological forecasting: ontological view and 

teleological view (Frick, 1974). The ontological view assumes that technological change and 

innovation are the result of scientific and technical opportunities and advancement. The 

11
teleological view assumes that technology change and innovation are the result of environmental 

factors (i.e., social, economic, political, etc.) that demand a desired end result. Thus, 

technological forecasting methods can be classified as exploratory (ontological view) or 

normative (teleological view). Frick (1974) characterized the methods by stating that one could 

think of exploratory and normative forecasting as analysis according to the ‘push’ of 

opportunities versus the ‘pull’ of objectives. Exploratory methods include: economic analysis, 

extrapolation of time-series, learning curves, trend analysis, morphological systematic 

exploration, probabilistic exploratory forecasting, qualitative historical analogy, and timeindependent contextual mapping (Jantsch, 1967; Frick, 1974). Normative forecasting methods 

include: morphological analysis (schematic and matrix), relevance trees, mission flow, decision 

matrices, and systems analysis. 

Traditionally, technological forecasting was centered on the estimates by recognized 

experts. Those estimates may no longer be as appropriate since technological advancement and 

progress has become increasingly more dependent on the unique contributions of several diverse 

technologies (Meredith & Mantel, 2004). Meredith and Mantel (2004) also point out that the 

degree and speed of technological advancement and change is related to the funding of several 

unique technologies. Therefore, in order to reliably forecast the availability of future 

technological systems it is necessary to elicit knowledge from a range of experts on a number of 

relevant topics. 

One final point about technological forecasting is relevant to the current discussion. The 

fact that technological advancement results in a new capability does not mean that it will be or

should be put to immediate use. The introduction of new technology or technological systems 

does not affect a single variable – it affects and transforms the entire system, transforming roles, 

judgments, goals, and relationships (Carroll & Rosson, 1992). The successful application and 

implementation of a new technological innovation often does not occur until five to seven years 

after the discovery (Rosegger, 1986). It takes a fair amount of time to understand how to most 

effectively and efficiently implement a technology while understanding the effect on the overall 

system and organization. As Harville (2004) suggested, “it is also possible ... to pour resources 

into developing a new technology too soon, so that the effort is largely wasted and futile” (p. 1). 

Backcasting

Environmental scanning focuses on identifying information about the environment, 

including the current state-of-the-art and likely technological advancements. Backcasting is an 

extension of an environmental scan to identify what the desired future should look like. This is 

in contrast to forecasting, which attempts to predict the future. As Hojer & Mattson (1999) 

suggest, “forecasting, based more or less on prolonging existing trends, is of little value beyond 

the role of an alarm-clock” (p. 10). 

Backcasting is an extension of a thorough and reliable forecast that begins with a 

desirable future endstate (Robinson, 1982). Once the endstate or desirable future has been 

identified, the backcast works backwards to determine what reasonable steps must be 

accomplished to reach the endstate through visionary thinking. The backcast should include 

“what-if” analysis to identify conditions and risks to meeting the objective. Thus, the backcast 

12
can help identify the “technological leaps” required to reach the endstate. The backcasting 

approach consist of six steps: determine the objectives, specify the goals, constraints, and 

targets, describe current state, specify exogenous variable, complete scenario analysis, and 

complete impact analysis (Robinson, 1990). 

Enabling Knowledge Creation 

When looking at future issues, especially issues that are five to ten or more years out, it is 

necessary to move from elicitation of existing knowledge to the creation of new knowledge. 

While current techniques of cognitive task analysis and knowledge elicitation provide a 

mechanism for acquiring existing knowledge they are not designed for predicting and assessing 

realistic observations and concepts about the future. Effective methods are needed for capturing 

knowledge from available information sources and using that knowledge to create new ideas and 

concepts. 

Based on the distinction between explicit and tacit knowledge, Nonaka & Takeuchi 

(1995) present a model for knowledge creation and conversion. Figure 3 reproduces the four 

patterns of interaction between explicit and tacit knowledge outlined by Nonaka & Takeuchi, 

1995). The four quadrants characterize the ways in which knowledge can be converted into new 

concepts and ideas. The authors assert that tacit knowledge can be converted into new tacit 

knowledge through the process of socialization – the sharing of experiences, metal models, and 

technical skills. Tacit knowledge can be converted into new explicit knowledge through the 

process of externalization – where experts interact with novices and transfer information using 

metaphors, analogies, and models. Explicit knowledge is converted to tacit knowledge through 

the process of internalization, thought experimentation, and individual learning. Finally, explicit 

knowledge can be converted to new explicit knowledge through the process of combination. In 

combination, new knowledge is created by reconfiguring existing knowledge through sorting, 

recategorizing, or other methods to expand the understanding of existing concepts. 

 

Tacit 

Knowledge 

Explicit 

Knowledge 

 To

Tacit 

Knowledge Socialization Externalization

Explicit 

Knowledge 

From

Internalization Combination

Figure 3. Development Process for New Knowledge and Concepts. 

In sum, knowledge creation is a continuous process for transferring and sharing tacit and 

explicit knowledge with groups and individuals (Bloodgood & Salisbury, 2001). The knowledge 

creation process results in the development of principles, facts, concepts, procedures, and 

processes – these products must be extended to develop future concepts. The modes of 

13
knowledge creation described by Nonaka & Takeuchi seem most appropriate when transferring 

or transforming existing knowledge within a relatively stable system, but not necessarily 

development and synthesis of unique, unknown concepts and ideas for a U.S. Army future 

environment that is changing rapidly. Figure 4 illustrates an expansion of the process to include 

the development of new concepts. The critical step is expanding the methodology to create 

unique knowledge. 

Figure 4. Concept Development Process. 

A New Method 

While the methods described above provide a number of capabilities, they do not provide 

a useful or suitable process for the development and evaluation of new knowledge and concepts 

about the future. A new method is needed to address the unique aspects associated with the 

development of new concepts and prediction of future realities. This section will describe such a 

method, the Flexible Method of Cognitive Task Analysis (FLEX). 

Although the FLEX method shares many features with other methods of analysis it offers 

many components that differentiate it from traditional methods of cognitive task analysis and 

knowledge elicitation. Specifically, it focuses on the development and evaluation of future 

knowledge. The FLEX attempts to be more comprehensive than conventional cognitive task 

analysis methods by attempting to capture not only traditional task information but also to 

provide a methodology that can capture the conglomerate of interdependent and dynamic 

knowledge sources. The method allows researchers to capture existing knowledge and it 

facilitates the creation of new knowledge and concepts. The technique is similar to the 

information acceleration method used in the marketing domain. In the information acceleration 

14
method where companies try to forecast consumers’ response to new products by providing early 

models to focus groups. 

The FLEX method is an interview-based problem solving approach that systematically 

develops and explores future concepts. The approach uses a combination of the Critical 

Decision Method, Protocol Analysis, the Delphi technique, and the standard interview. Unlike 

existing methods, the FLEX method grounds the experts thinking in a futuristic setting. Thus, 

knowledge will be captured by employing a vignette-based scenario approach where experts are 

provided with a brief multimedia presentation that highlights a potential future situation and 

requires the expert to solve a complex problem using the anticipated capabilities. To do so, 

experts are required to figure out how to solve the issues presented in the scenario using the 

resources and capabilities provided. 

Similar to protocol analysis, participants are asked to verbalize their responses by 

thinking aloud. However, similar to the Delphi technique, responses from each participant are 

provided to subsequent participants. Therefore, subsequent participants are able to identify 

weaknesses, confirm strengths, and build upon the prior responses. Throughout the process, a 

semi-structured interview is used to probe expert knowledge and gain a deeper understanding of 

the expert’s reasoning. Before the completion of the process, responses from subsequent 

participants are fed back to the originator for additional input. Finally, a small group of experts 

are used for interactive group discussions allowing for consensus building and validation. 

The vignette-based, scenario approach is appropriate in this context because it is often 

difficult for individuals to speculate on how future capabilities might be employed. Further, 

many experts tend to over-estimate the impact of future technological advances and to ignore the 

difficulties. Thus, forcing the participants to solve a concrete problem helps ground their 

thinking so that more appropriate and realistic information can be captured. Figure 5 illustrates 

how the flex process produces future tactics, techniques, and procedures (TTP), concepts, and 

other products. The following section summarizes the steps involved in developing a FLEX 

scenario and conducting a knowledge elicitation and concept exercise. 

Phase 1: Domain and Problem Identification 

In the first phase of the FLEX approach, it is necessary to define the domain area and 

identify the significant problems of interest. Defining the domain will help specify the scope of

the information that will be needed and will assist in the identification of experts. Since a goal of

the method is not only to capture new knowledge but to systematically investigate the efficacy of 

the information collected, the identification of a significant number of appropriate experts is 

required. In most cases, a diverse set of experts will be required from a variety of technical 

fields – particularly in cases where technological advancements are expected to play a critical 

role in the development of new systems and processes. During the problem identification phase, 

an environmental scan, backcast, and/or technological forecast should be conducted to assist in 

understanding the potential for technological innovation and advancement. 

 

15
Figure 5. Overview of the FLEX Process. 

Phase 2: Initial Review and Analysis 

During the initial review and analysis it is important to capture information concerning 

the current state-of-the-art for the domain and problem space. This phase is similar to what 

would be expected during traditional task analysis – both procedural and cognitive. During the 

initial review it is necessary to interview experts, document current processes, and gather 

information about expert performance. Contrary to traditional task analysis, the goal is not to 

develop a complete list of tasks and duties in a given domain. The goal is to gain an 

understanding of domain expertise, task requirements, processes, and outcomes. During the 

review it is critical to identify areas where innovation may lead to improved performance and 

processes and to gain an understanding of current research and immediate technological 

innovation for the domain. 

Phase 3: Refine Problem Space and Develop Initial Scenario

During this phase, initial decisions regarding the knowledge elicitation scenario should be 

established. An initial group of domain experts should be used to develop one or more scenarios 

with appropriate “branches.” The branches should represent contentious areas where multiple 

realities are possible. The scenarios should be developed in an iterative fashion, allowing experts 

16
to develop a realistic situation capable of focusing the knowledge elicitation process. At this 

stage, it is not necessary to fully develop the situation – it is only necessary to establish the initial 

conditions for the knowledge elicitation process. 

Phase 4: Initial Knowledge Elicitation 

Based on the scenario, this phase should elicit knowledge and concepts from a series of 

experts from the relevant domain areas. The scenario should be used as a starting point to focus 

the experts on a particular problem area. The elicitation process should allow the experts to 

further refine the scenario by adding new information, challenging existing assumptions, 

anticipate unintended consequences, and predicting technological interventions. During the 

elicitation, experts should be asked to discover new ways to solve problems associated with the 

scenario given hypothesized future capabilities. Knowledge elicitation should focus on both 

individuals and small groups (e.g., dyads, triads) and should combine experts from different 

specialty areas. 

Phase 5: Data Reduction and Consensus Building 

After a sufficient sample of experts have had the opportunity to complete the knowledge 

elicitation phases, it is necessary to aggregate the data into common and meaningful responses. 

The data reduction phase should allow a small group of experts, a different group from phase 4, 

to develop a consensus on the efficacy of the new knowledge and concepts captured from phase 

4. The information can then be used to update the scenario to reflect the new knowledge stream

or create a new branch to highlight “what if” situations. The altering of the scenario allows the 

experts to “leave their fingerprints” on the scenario and allows an iterative process for 

continuous improvement. 

After revising the scenario based on the initial knowledge elicitation and data reduction 

phases, it is necessary to reexamine the scenario with a new set of experts and provide the 

revised materials to former participants. This process provides a systematic way to evaluate the 

realism of the new information. Thus, Phases 4 and 5 may be repeated several times until a 

tested solution is developed. The goal of the iterative process is to develop a well tested and 

documented solution for the purpose of developing new theories, principles, tools, techniques, 

and procedures. 

Phase 6: Knowledge Representation and Concept Documentation 

 Knowledge representation provides a mechanism for documenting and displaying 

information in a useable format. In this context, knowledge refers to organized concepts, 

theories, principles, descriptions, and mental models of descriptive, procedural, and metacognitive information. The goal is to present the results of the knowledge elicitation process in a 

meaningful way. Knowledge can be represented using a variety of methods, including: logic, 

semantic networks, production rules, frame-based representations, decision trees, graphs, 

diagrams, charts, and tables. 

17
Using Experts for Knowledge Elicitation and Creation 

The previous sections of this document have outlined the use of experts to elicit 

knowledge about a particular task or domain area. Expertise is highly specific to a particular 

domain. Cognitive performance is interrelated to domain content and context (Newell & Simon, 

1972). In other words, experts in one domain area may be novices in a different area. 

Experts are bound by their history and experience and not all experts are able to reason 

beyond the information and knowledge readily available in their environment. When confronted 

with a new problem, experts call upon their prior knowledge and experiences in order to solve 

the problem or predict future events. There is no guarantee that an expert in an existing domain 

area will continue to be an expert in a future realization of that domain. In research leading to 

the development of the FLEX method, some of the military subject matter experts who served as

participants were unable to free their thinking from current doctrine. Furthermore, some who 

have spent time studying the Army’s emerging future force doctrinal concepts had difficulty 

critically evaluating those concepts. Other participants, however, were stimulated by the FLEX 

method to provide extremely unique, creative, and well-reasoned responses (Gossman, Mauzy, 

Heiden, & Flynn, in preparation). It did not seem to the researchers that expertise in current 

operations was a good predictor of how successful the participant would be. Consider the 

difficulty administrative assistants, experts in using typewriters, had adapting to the introduction 

of computers to the workplace. Yet, these experts will ultimately contribute to and influence the 

decision making process (Karat, Karat, & Vergo, 2004) regarding future systems. 

Often it is assumed that a domain’s most experienced individuals are also the ones with

the most expertise. Several research studies have questioned the accuracy this assumption (e.g., 

Jacob, Lys, & Neale, 1997; Camerer & Johnson, 1991). Many of these studies led Ericsson 

(2000) to suggest, “. . . continued improvements (changes) in achievement are not automatic 

consequences of more experience and in those domains where performance consistently 

increases aspiring experts seek out particular kinds of experience, that is deliberate practice.” 

While experience is often used as an indicator of expertise (e.g., hours of flying time) especially 

when more objective measures are lacking, it tends to be inadequate. Experience may be 

necessary for expertise, but experience is not, in and of itself, sufficient in the development of

expertise (Rohrbaugh & Shanteau, 1999). 

Even if true experts are identified there is a concern that their knowledge may be biased. 

With specific attention to forecasting future technologies, Tichy (2002) stated, “top experts . . . 

demand a more active policy to promote their field of work and tend to be overoptimistic with 

regard to the realization of innovations” (p. 19). The over-optimism bias is most notably 

expressed in terms of innovativeness of the technology, the chance of realizing the technology, 

and the potential exploitation of the technology. These findings led the author to conclude that 

“foresight exercises should include not only top experts of the relevant field, but also experts 

with a broader range of interest as well as experts with widely differing backgrounds” (p. 4). 

The failure to select the right group of experts in concept development exercises “is likely to 

bring assessments that are too optimistic” (Tichy, 2002, p. 19). In the Army transformation 

planning effort, in particular the development of Future Combat Systems, initial plans were very 

optimistic and remain so despite several reductions in requirements and increases in timelines. 

18
The question remains then, how do we identify and select participants for an elicitation 

exercise? It is necessary to select experts that are capable of extending their knowledge, mental 

models, and schemas beyond their current state in a reliable and valid way. The research on 

identifying appropriate experts, much less identifying those experts capable of extending their 

knowledge to future concepts, is relatively absent. As Subramani, Peddibhotla, and Curley 

(2003) suggested, “Given the importance of the issue, it is surprising that it has received little 

attention in prior research” (p. 4). 

Bloom’s (1956) taxonomy provides a potential starting point to developing a systematic 

methodology for accurately selecting experts. The taxonomy consists of six categories 

(knowledge, comprehension, application, analysis, synthesis, and evaluation) of higher-order 

cognitive thinking skills. Figure 6 illustrates the taxonomy and highlights the abilities of experts 

in each category. Experts at the synthesis and evaluation level may have the required abilities to 

examine future issues and develop new knowledge and concepts. These abilities include the 

ability to role-play, invent, speculate, and develop information (synthesis). Additionally, these 

individuals are able to compare, judge, critique, and assess concepts, and can make conclusions 

and recommendations based on information. 

Clearly, identifying appropriate experts for an elicitation exercise is the most important 

component of the process. Future research efforts must focus on developing techniques and 

procedures for addressing the issue. 

 

Conclusions 

Potter, Roth, Woods, and Elm (2000) stated, “in performing cognitive task analysis, it is 

important to utilize a balanced suite of methods that enable both the demands of the domain and 

the knowledge and strategies of domain expertise to be captured in a way that enables a clear 

identification of opportunities for improved support” (p. 321). There is clearly a need for a 

systematic approach to eliciting information needed to develop future concepts, processes, 

systems, and procedures. The method described here provides a flexible method for eliciting and 

creating concepts and information for investigating, creating, testing, and understanding future 

issues. Ongoing efforts, such as the U.S Army Research Institute’s project to develop training 

for crisis action planning and execution (Beaubien, 2005), are using the method to evaluate its 

effectiveness. 

In concluding this analysis it is important to restate one of our original questions: How 

do we best use the total Army team – warfighter, scientist, theorist, analyst, and engineer – in the 

development process of future concepts? Diaper & Stanton (2004) stated that, “Task analysis 

without psychology is like peaches and cream without the peaches: thin stuff” (p. 567). The 

same can be stated about the warfighter, theorist, analysis, or engineer. The Army can no longer 

rely on haphazard or unsystematic methods for concept development that do not include the total 

Army team. Therefore, the critical need is to implement a process that systematically allows for 

effective and efficient management of the concept development process. The FLEX method 

represents an attempt to integrate and exploit the capabilities of the total Army team. 

19
Figure 6. Task Oriented Question Construction Wheel Based on Bloom’s Taxonomy, © 2004 

St. Edward’s University Center for Teaching Excellence. 

20
References 

Aguilar, F J. (1967). Scanning the business environment. New York, NY: Macmillan Co. 

Beaubien, J. (2005). Red Cape – Crisis action planning and execution: Research and 

Development Plan. Available from U.S Army Research Institute for the Behavioral and 

Social Sciences, Fort Knox, KY. 

Bloodgood, J. M., & Salisbury, W. D. (2001). Understanding the influence of organizational 

change strategies on information technology and knowledge management strategies. 

Decision Support Systems, 31, 55-69. 

Bloom, B. S. (Ed.) (1956). Taxonomy of educational objectives: The classification of 

educational goals: Handbook I, cognitive domain. New York; Toronto: Longmans, 

Green. 

Burnett, M. F., & McCracken, J. D. (1982, Summer). Characteristics, Procedures, and Results of 

Two Job Analysis Techniques. Journal of Vocational Education Research 7, no. 3, 1-10. 

Camerer, C. F., & E. J. Johnson (1991). The process-performance paradox in expert judgment: 

How can the experts know so much and predict so badly? In Towards a General Theory 

of Expertise: Prospects and Limits, K. A. Ericsson & J. Smith, (Eds.). Cambridge: 

Cambridge University Press, pp. 195-217. 

Card, S., Moran, T. P., & Newell, A. (1983). Applied information processing psychology. 

Hillsdale, NJ: Lawrence Erlbaum.

Carroll J. M., & Rosson, M. B. (1992). Getting around the task-artifact cycle: How to make 

claims and design by scenario. ACM Transactions on Information Systems 10, 181-212. 

Carter, A. B., & White, J. P. (Eds.) (2000). Keeping the Edge: Managing Defense for the 

Future. Preventive Defense Project, 2000. Cambridge, MA. 

Chervinskaya, K. R., & Wasserman, E. L. (2000). Some methodological aspects of tacit 

knowledge elicitation. Journal of Experimental & Theoretical Artificial Intelligence, 12, 

43-55. 

Christal, R. E., & Weissmuller, J. J. (1976). New Comprehensive Occupational Data Analysis 

Programs (CODAP) for Analyzing Task Factor Information, Interim Professional Paper 

no. TR76-3, Air Force Human Resources Laboratory, Lackland Air Force Base, TX. 

Coates, J. F. (1985). Issues identification and management: The state of the art of methods and 

techniques (Research Project 2345-28). Electric Power Research Institute, 3412 Hillview 

Ave., Palo Alto, CA 94304. 

21
Cordingley, E. S. (1989). Knowledge elicitation techniques for knowledge-based systems. In D. 

Diaper (Ed.), Knowledge elicitation: Principles, techniques and applications. 

Chichester, England: Ellis Horwood Ltd. 

Cordesman, A. H. & Wagner, A. R. (1996). The lessons of modern War: The Gulf War. 

Westview Press: Boulder, CO.

Cunningham, J. W., Boese, R. R., Neeb, R. W., & Pass, J. J. (1983). Systematically derived 

work dimensions: Factor analyses of the Occupational Analysis Inventory. Journal of 

Applied Psychology, 68, 232-252. 

Dalkey, N. C. (1969). The Delphi Method: An Experimental Study of Group Opinion. Santa 

Monica, CA: Rand Corporation. 

Department of the Army (Draft). Battle Command Staff Tasks for Unit of Employment (ARTEP 

3-91). Fort Leavenworth, KS. 

Department of Defense (2001). Department of Defense Handbook: Instructional systems 

development/Systems approach to training and education (MIL-HDBK-29612-2A). 

Department of Defense: Washington DC. 

Diaper, D. & Stanton, N. A. (Eds.) (2004). The handbook of task analysis for human-computer 

interaction. Mahwah, NJ: Lawrence Erlbaum Associates. 

DuBois, D. A., Shalin, V. Levi., & Borman, W. C. (1995). A Cognitively-Oriented Approach to

Task Analysis and Test Development. Office of Naval Research, Contract No. N00014-

93-C-0199 (December 1995). 

Endsley, M. R. (1993). A survey of situational awareness requirements in air-to-air combat 

fighters. International Journal of Aviation Psychology, 3(2), 507-512. 

Endsley, M. R. (1999). Situational Awareness and Human Error: Designing to support human 

performance. Proceedings of the High Consequences Surety Conference, Albuquerque, 

NM. 

Endsley, M. R. (2000). Direct Measurement of Situation Awareness: Validity and use of 

SAGAT. In Mica R. Endsley & Daniel J. Garland (Eds.), Situation Awareness Analysis 

and Measurement. Mahwah, New Jersey: Lawrence Erlbaum Associates. 

Ericsson, K. A. (2000). Expert Performance and Deliberate Practice: An updated excerpt from

Ericsson (2000). Retrieved October 10, 2004, from http://www.psy.fsu.edu/faculty/ 

ericsson/ericsson.exp.perf.html

Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (revised 

edition). Cambridge, MA: MIT Press. 

22
Evans, J. St. B. T. (1988). The knowledge elicitation problem: A psychological perspective. 

Behavior and Information Technology, 7, 111-130. 

Feltovich, P. J., Spiro, R. J., & Coulson, R. L. (1997). Issues of expert flexibility in contexts 

characterized by complexity and change. In P. J. Feltovich, K. M. Ford, & R. R. 

Hoffman (Eds.), Expertise in context: Human and machine. Menlo Park, CA: 

AAAI/Mit Press. 

Fine, S. A., & Wiley, W. W. (1971). An introduction to functional job analysis. Kalamazoo, 

MI: W. E. Upjohn Institute for Employment Research. 

Flanagan, J. C. (1954). The critical incident technique. Psychological Bulletin, 51, 327-358. 

Frick, R. K. (1974). Operations Research and technological forecasting. Air University Review, 

24(4). 

Gaines, B. R., & Shaw, M. L. (1992). Knowledge acquisition tools based on personal construct 

psychology. Knowledge Engineering Review, 8, (pp. 49-85). 

Geiwitz, J., Kornell, J., McCloskey, B. (1990). An Expert System for the Selection of 

Knowledge Acquisition Techniques. Technical Report 785-2, Contract No. DAAB07-

89-C-A044). Anacapa Sciences. California. 

Gossman, R. J., Mauzy, R., Heiden, C., & Flynn, M. (in preparation). Flexible methods for 

Future Force development. Arlington, VA: U.S. Army Research Institute for the 

Behavior and Social Sciences. 

Gott, S. P. (1998, April). Rediscovering Learning: Acquiring Expertise in Real World Problem

Solving Tasks. Armstrong Research Laboratory, (Report Number AL/HR-TP-1997-

0009), Brooks AFB, TX. 

Hall, E., Gott, S., & Pokomy, R. (1995). A procedural guide to cognitive task analysis: The 

PARI methodology (Technical Report AL/HR-TR-1995-0108). Armstrong Laboratory, 

Brooks AFB, TX. 

Hamilton, F. (1996). Predictive Evaluation Using Task Knowledge Structures. M. J., Tauber, 

V., Bellotti, R., Jeffries, J. D., Mackinlay, & J. Nielsen (Eds.). ACM conference on 

Human Factors in Computing Systems CHI’96 Conference Companion, Vancouver, 13-

18th April. ACM Press. pp. 261-262. 

Harvey, R. J. (1993). Research Monograph: The development of the Common-MetricQuestionnaire (CMQ). Personnel Systems & Technologies Corporation. Available 

online at http://www.pstc.com/documents/monograph.pdf 

23
Harville, M. (2004). Motivation for the Enhancement of Machine Perception. Retrieved on 

October 5, 2004 from http://www.hpl.hp.com/personal/ 

Michael_Harville/visionStatement.htm 

Hoffman, R. R., Shadbolt, N. R., Burton, A. M., & Klein, G. A. (1995). Eliciting knowledge 

from experts: A methodological analysis. Organizational Behavior and Human 

Decision Processes, 62, 129-158. 

Howell, W. C., & Cooke, N. J. (1989). Training the human information processor: A look at 

cognitive models. In I. L. Goldstein (Ed.), Training and development in work 

organizations: Frontiers of industrial and organizational psychology (pp. 121-1 82). San 

Francisco, CA: Jossey-Bass. 

Hojer, M., & Mattson L. G. (1999). Historical determinism and backcasting in future studies. 

Paper presented at the conference on Urban Transport Systems in Lund, June 7-8 1999, 

preliminary version. 

Jacob, J., Lys, L. T, & Neale, M. A. (1997). Experience, expertise and the forecasting

performance of security analysts. Social Science Research Network Electronic Library. 

(December 12, 1997). http://ssrn.com/abstract=54681 

Jantsch, E. (1967). Technological Forecasting in Perspective, OECD, Paris 1967, p.6. 

Johnson, P., Diaper, D., & Long, J. B. (1984). Tasks, skills and knowledge: task analysis for 

knowledge based descriptions. In B. Shackel (Ed.) Proceedings Interact'84, 499503. 

North-Holland, Amsterdam.

Johnson, H., & Johnson, P. (1991). Task Knowledge Structures: Psychological basis and 

integration into system design. Acta Psychologica, 78, pp 3-26. 

Jonassen, D. H., Hannum, W. H, & Tessmer, M. (1989). Handbook of task analysis procedures. 

New York: Praeger Publishers. 

Karat, J., Karat, C. M., & Vergo, J. (2004). Experienced people value: The new frontier of task 

analysis. In The Handbook of Task Analysis for Human-Computer Interaction. 

Dan Diaper & Neville Stanton. (Eds.) Lawrence Erlbaum Associates. 

Kelly, G. A. (1955). The psychology of personal constructs. New York: Norton. 

Kieras, D. E. (1988). Towards a practical GOMS model methodology for user interface design. 

In M. Helander (Ed.), Handbook of Human-Computer Interaction (pp. 135-157). New 

York: North-Holland Publishing.

Kirwan, B., & Ainsworth, L. K. (Eds.) (1992). Comprehensive review of task analysis 

techniques A Guide to Task Analysis. London: Taylor & Francis, Ltd. 

24
Klein, G. A. (1996). The development of knowledge elicitation methods for capturing military 

expertise. (ARI Research Note 96-14). Alexandria, VA: U.S. Army Research Institute 

for the Behavioral and Social Sciences. 

Klein, G. A., Calderwood, R., & MacGregor, D. (1989a). Critical Klein, G. A., Calderwood, R., 

& MacGregor, D. (1989). Critical McCormick, E. J., Jeanneret, P. R., & Mecham, R. C. 

(1972). A study of characteristics and job dimensions as based on the Position Analysis 

Questionnaire (PAQ). Journal of Applied Psychology, 56, 347-368. 

Klein, G., Calderwood, R., & Macgregor, D. (1989b). Critical decision method for eliciting 

knowledge. IEEE Transactions on Systems, Man, & Cybernetics, 19(3), 462-472. 

Major, N. P., (1991). CATO An Automated Card Sort Tool. In Proceedings of the Fifth 

European Knowledge Acquisition for Knowledge-Bases Systems Workshop, University of 

Strathclyde. 

Major, N. P., & Reichgelt, H. (1990). ALTO: An Automated Laddering Tool. In B. Wielinga, 

J. Boose, B. Gaines, G. Schreiber & M. Van Someren (Eds.). Current Trends in 

Knowledge Acquisition, (pp. 222-236), Amsterdam: IOS Press. 

McCormick, E. J., Jeanneret, P. R., & Mecham, R. C. (1972). A study of job dimensions as 

based on the Position Analysis Questionnaire. Journal of Applied Psychology, 56(4), 347-

368. 

Meredith, J. R., & Mantel, S. J. (2004). Technological Forecasting (Appendix). In Mantel, S. J., 

Meredith, J. R., Shafer, S. M., & Sutton, M. M. (2004). Core concepts: Project 

management in practice (2nd Ed.). John Wiley & Sons. 

Meyer, M. A., & Booker, J. M. (1991). Eliciting and analyzing expert judgment: A practical 

guide. San Diego, CA: Academic Press. 

Militello, L. G., Hutton, R. J. B., Pliske, R. M., Knight, B. J., & Klein, G. (1997). Applied 

Cognitive Task Analysis (ACTA) Methodology. Navy Personnel Research and 

Development Center. A detailed description of how to conduct the ACTA method. 

Moody, J. W., Blanton, J. E., & Will, R. P. (1998/1999). Capturing expertise from experts: The 

need to match knowledge elicitation techniques with expert system types. The Journal of 

Computer Information Systems, 1998/1999, Winter, 89-95. 

Newell, A., & Simon, H. (1972). Human Problem Solving. Englewood Cliffs, NJ: PrenticeHall. 

Nonaka, I., & Takeuchi, H. (1995). The Knowledge Creating Company. New York: Oxford 

University Press. 

25
Potter, S. S., Roth, E. M., Woods, D. D. & Elm, W. (2000). Bootstrapping multiple converging 

cognitive task analysis techniques for system design. In J. M. Schraagen, S. F. Chipman 

& V. L. Shalin (Eds.). Cognitive Task Analysis (pp. 317-340). Mahwah, NJ: Lawrence 

Erlbaum Associates, Inc. 

Primoff, E. S., & Eyde, L. D. (1988). Jon element analysis. In S. Gael (Ed.) The job analysis 

handbook for business, industry, and government (Vol. 2, pp. 807-824). New York: 

John Wiley. 

Robinson, J. (1982). Energy backcasting: A Proposed Method of Policy Analysis. Energy 

Policy 10(4), 337-344. 

Robinson, J. (1990). Futures Under Glass: A Recipe for People Who Hate to Predict Futures. 

Futures, 11, 820-842. 

Rohrbaugh, C. C., & Shanteau, J. (1999). Context, process, and experience: Research on 

applied judgment and decision making. In F. T. Durso, R. S. Nickerson, & M. T. H. Chi, 

(Eds.), Handbook of Applied Cognition (pp. 115-139). New York: John Wiley. 

Rosegger, G. (1986). The Economics of Production and Innovation, 2nd ed. New York: 

Pergamon Press. 

Rugg, G., Mcgeorge, P., & Shadbolt, N. (1990). On the use of laddered grids in knowledge 

elicitation. Artificial Intelligence Group, Technical Report, February 1, University of 

Nottingham.

Sarter, N., Woods, D. D., & Billings, C. E. (1997). Automation Surprises. In G. Salvendy (Ed.), 

Handbook of Human Factors/Ergonomics, 2nd Ed, New York: Wiley. 

Schraagen, J., Chipman, S., & Shalin, V. (Eds.) (2000). Cognitive Task Analysis. Mahwah, NJ: 

Lawrence Erlbaum Associates. 

Shute, V. J., & Torreano, L. A. (1995). Cognitive task analysis made simple: Decompose, 

network, assess (DNA). Unpublished manuscript. 

Shute, V. J., Torreano, L. A., & Willis, R. E. (1999). Exploratory test of an automated 

knowledge elicitation and organization tool. International Journal of Artificial 

Intelligence in Education, 10, 365-384. 

Shute, V., Sugrue, B., & Willis, R. E. (1997). Automating Cognitive Task Analysis. In 

Cognitive Technologies for Knowledge Assessment Symposium Proceedings, AERA, 

Chicago, IL: 1997. An overview of the DNA method. 

26
Subramani, M. R., Peddibhotla, N. B., & Curley, S. P. (2003). How do I know that you know? 

Cues used to infer another’s expertise and to signal one’s own expertise. Correspondence 

to Curley, S. P. (June 4, 2003). Retrieved from

http://ids.csom.umn.edu/faculty/mani/Homepage/Papers/Subramani&Peddibhotla&Curle

y_WP.pdf 

Tichy, G. (2002). Over-optimism among experts in assessment and foresight. Institute of 

Technology Assessment. Manuscript, ITA-02-05, Vienna: Available at 

http://www.oeaw.ac.at/ita/pdf/ita_02_05.pdf

Wass de Czega, H. (1998). Optimizing Future Battle Command Technologies. Military Review,

March/April. 

Wass de Czege, H. & Biever, J. (1998). Optimizing Future Battle Command Technologies. 

Military Review, 78 (2), 15-21. 

Wass de Czege, H., & Sinnreich, R. H. (2002). Conceptual foundations of a transformed U.S. 

Army (Land Warfare Paper No. 40). Arlington, VA: Association of the Unites States 

Army. 

Woods, D. (1998). Designs are hypotheses about how artifacts shape cognition and 

collaboration. Ergonomics, 41, 168-173. 

Woods, D. (2002). Steering the reverberations of technology change on fields of practice: Laws 

that govern cognitive work. Proceedings of the 24th annual meeting of the Cognitive 

Science Society. 

Woods, D., & Dekker, S. (2000). Anticipating the effects of technological change: A new era of 

dynamics for human factors. Theoretical Issues in Ergonomics Science, 2000, 1(3), 272-

282. 

Wright, G., & Ayton, P. (1987). Eliciting and modeling expert knowledge. Decision Support 

Systems, 3, 13-26. 

27
Appendix A 

Applied Cognitive Task Analysis 

A widely used approach to cognitive task analysis is Applied Cognitive Task Analysis

(ACTA). The ACTA method is designed to provide streamlined techniques to elicit and capture 

the cognitive aspects of expert performance (Militello, Hutton, Pliske, Knight, & Klein, 1997). 

The intent was to develop tools that were easier to use than the methods of cognitive task 

analysis available at the time the method was developed. The method consists of three 

complementary techniques (task diagram, knowledge audit, and simulation interview) used to 

elicit different aspects of cognitive performance for a given task. The objective of the task 

diagram is to provide the analyst with a general overview of the domain area and to highlight the 

cognitively demanding portions of the task. The knowledge audit is used to further investigate 

the specific expertise required for a particular task or subtask. Finally, the simulation interview 

uses a scenario to probe the cognitive processes involved in a specific situation. The results of 

each of the techniques are consolidated in a cognitive demands table used to synthesize the data 

so that it can be applied to a specific situation or project. The ACTA method was developed to 

assist instructional designers and training professionals in developing training that focuses on 

cognitive skills. 

 

Critical Decision Method

Using the Critical Incident Technique originally modeled by Flanagan (1954), Klein, 

Calderwood, and MacGregor (1989b) developed a technique called Critical Decision Method 

(CDM). The premise of the CDM is that useful information can be retrieved by having experts 

recall a salient incident or cases. In most situations, the case identified represents a non-routine 

incident where expert performance can be highlighted. The incident becomes the focus of a 

semi-structured interview where cognitive probes are used to understand the expert’s situation 

assessment and decision making. 

There are several advantages to CDM. The semi-structured format is flexible, allowing

the interviewer to switch to more relevant issues as the dialogue progresses. Focusing on critical 

incidents highlights the very capabilities that separate experts from novices allowing little time to 

be wasted on general tasks that do not require the specialized skill of the expert. Probed recall of

such tough cases is effective in revealing previously inaccessible knowledge of highly skilled 

personnel. As Klein, Calderwood, & Macgregor (1989b) state, “the focus on non-routine cases 

also makes the method most appropriate to eliciting tacit knowledge that is not part of the 

formalized procedures for a domain” (p. 465). The categorization of CDM as a specific type of 

technique is not definitive because the technique incorporates aspects of several other knowledge 

elicitation methods. Klein (1996) stated that, “the CDM is a protocol analysis method” (p. 4). 

However, CDM lacks the ‘think aloud’ aspect of standard protocol analysis and focuses on a 

prior event, rather than a current or future problem. It is similar to a case study in that a scenario 

is analyzed from a decision making perspective, but in CDM the expert actually experienced the 

case firsthand. 

A-1
Precursor, Action, Results, Interpretation (PARI) 

The PARI technique (Hall, Gott, & Pokomy, 1995) is a specialized interview technique that 

combines the technique of thinking aloud described by Ericsson and Simon (1993) with expert 

and novice problem solving activity (Gott, 1998). The Precursor, Action, Results, Interpretation 

(PARI) procedure utilizes a “situated problem-solving session” where experts demonstrate their 

knowledge based on a specific problem or context while being probed through structured 

questioning to elicit knowledge and expert skills. The approach attempts to identify the action a 

performer would take given a specific precursor event, the result the action would have initiated, 

and the performer’s interpretation of the result. The approach further defines the cognitive 

process by allowing the performers to document their internal mental models. 

Decompose, Network, and Assess Method (DNA) 

The DNA method was developed to provide a practical approach to cognitive task 

analysis that was capable of eliciting and capturing task information from experts and novices 

across a wide range of domain areas (Shute & Torreano, 1995). The DNA method is supported 

with software tools that attempt to automate the knowledge elicitation, knowledge organization, 

and management, and hierarchically structure knowledge for the given domain. Shute, Torreano, 

and Willis (1999) describe the DNA as a method to “elicit knowledge from subject-matter 

experts which will Decompose a domain, Network the knowledge into comprehensive structures, 

and employ other experts in a given domain to Assess the validity, completeness, and reliability 

of the knowledge structures” (p. 369). 

The method and software consist of four modules (Customize, Decompose, Network, and 

Assess). The customize module is used to provide the analyst with a means to provide specific 

information about the domain area. The decompose module operates as an interactive, semistructured interview and is used to elicit procedural and conceptual domain knowledge from

subject-matter experts. The intent of the network module is to transform expert knowledge into 

knowledge hierarchies, graphs, and rules. The assess module is used to validate the information 

acquired and products developed during the elicitation process. Here, multiple experts are 

“employed to review and edit one another’s conceptual graphs as a method of validating 

externalized knowledge structures” (Shute, Torreano, & Willis, 1999, p. 375). 

Task Knowledge Structures (TKS)

The TKS method was derived from work by Johnson, Diaper, and Long (1984) on Task 

Analysis for Knowledge Descriptions and attempts to identify the various aspects of knowledge 

and information requirements for a task. The TKS method characterizes the knowledge of 

subject-matter experts based on previous experiences. A major assumption of TKS is that during 

the development of expertise individuals develop specific knowledge structures (Hamilton, 

1996) that are represented in memory (Johnson & Johnson, 1991). The TKS method provides a 

mechanism for collecting information about how experts perform required tasks and consists of 

two separate components, a goal structure and a taxonomic structure. The goal structure 

identifies the task activities, goals, subgoals, and provides an understanding of the steps needed 

A-2
to complete the task. The taxonomic structure includes information about actions and objects, 

including their classification, attributes, relationships to one another, features, and typicality 

rating (Johnson & Johnson, 1991). The TKS method assumes that the task knowledge is 

structured in a meaningful way and can be analyzed, modeled, and predicted. 

Goals, Operators, Methods, Selection Rules (GOMS)

The GOMS method (Card, Moran, & Newell, 1983) is a cognitive task analysis and 

modeling method that has been used extensively in the analysis of human-computer-interactions 

to document expert performance models. The approach is composed of four parts: goals, 

operators, methods, and selection rules. Goals are considered to be the endstate of the task when 

successfully completed – what the user wants to achieve. Operators are the procedural, 

perceptual, psychomotor, and cognitive actions that must be taken to accomplish the various 

goals and subgoals. The operators are sequenced to develop the methods or the specific step-bystep procedures for accomplishing the tasks. Operators are assigned an execution time 

represented the average amount of time required to complete the steps. Selection rules consist of 

the “if-then” decision rules and conditions that experts used to determine which method should 

be used to complete a particular goal or subgoal. 

The GOMS method is based on the assumption that experts interacting with a system are 

involved in problem solving. By analyzing how experts interact with the system it is possible to 

decompose the task or problem into subproblems and an understanding of the goals and 

expectations can be developed allowing for a logical sequence of operations and timing values 

for each operation to be documented. The model also provides a mechanism for making 

predications about how experts will perform given a proposed design. The method is not well 

suited for complex cognitive skills or for use with novices. However, there have been several 

variants to the GOMS method to address many of the methods shortcomings. The Natural 

Goals, Operators, Methods, and Selection Rules Language (NGOMS, Kieras, 1988) method, for 

example, added a mechanism to deal with expert-novice differences. 

Goal-Directed Task Analysis (GDTA) 

The GDTA method is a cognitive task analysis technique that concentrates on identifying 

an expert’s situational awareness needs and requirements for a particular task (Endsley, 1993). 

Endsley (2000) defined situational awareness as those “dynamic information needs associated 

with the major goals or subgoals of the operator in performing his or her job (as opposed to more 

static knowledge such as rules, procedures, and general system knowledge” (p. 149). Situational 

awareness requirements can be described by three levels of awareness. Level 1 Situational 

Awareness – Perception, involves the perception and identification of significant factors in the 

environment. Level 2 Situational Awareness – Comprehension, requires the expert or operator 

to interpret, store, retain, integrate multiple pieces of information, and understand the relevance 

to obtaining a particular goal. Level 3 Situational Awareness – Projection requires the ability to 

predict future events – the highest level of awareness. 

Thus, the GDTA method systematically identifies the situational requirements needed for 

effective decision making. The actual analysis is conducted using a mixture of approaches, 

A-3
including: observation, verbal protocol analysis, questionnaires, analysis of written materials, 

and various knowledge elicitation techniques. The GDTA method has been successfully applied 

to a wide variety of domains (Endsley, 1999). 

A-4