U.S. Army Research Institute 

 for the Behavioral and Social Sciences 

 

Research Report 1839 

Flexible Methods for Future Force 

Concept Development 

 

J. R. Gossman, Rebecca P. Mauzy, 

Charles G. Heiden, and Charlotte H. Campbell

Human Resources Research Organization 

Michael R. Flynn

Northrop Grumman Information Technology 

James W. Lussier, and Scott B. Shadrick 

U.S. Army Research Institute 

August 2005 

Approved for public release; distribution is unlimited.
U.S. Army Research Institute 

for the Behavioral and Social Sciences 

A Directorate of the Department of the Army 

Deputy Chief of Staff, G1 

Authorized and approved for distribution: 

 

 MICHELLE SAMS ZITA M. SIMUTIS 

 Technical Director Director 

Research accomplished under contract 

for the Department of the Army 

Human Resources Research Organization 

Technical review by 

May H. Throne, U.S. Army Research Institute 

Paula Durlach, U.S. Army Research Institute 

NOTICES 

DISTRIBUTION: Primary distribution of this Research Report has been made by ARI. 

Please address correspondence concerning distribution of reports to: U.S. Army 

Research Institute for the Behavioral and Social Sciences, Attn: DAPE-ARI-MS, 

2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926 

FINAL DISPOSITION: This Research Report may be destroyed when it is no longer 

needed. Please do not return it to the U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

NOTE: The findings in this Research Report are not to be construed as an official 

Department of the Army position, unless so designated by other authorized documents.
i 

REPORT DOCUMENTATION PAGE

1. REPORT DATE (dd-mm-yy) 

 August 2005 

2. REPORT TYPE 

 Final 

3. DATES COVERED (from. . . to) 

 April 2002 – July 2005 

5a. CONTRACT OR GRANT NUMBER 

 DASW01-99-D-0012 DO #0015 

4. TITLE AND SUBTITLE 

Flexible Methods for Future Force Concept Development 5b. PROGAM ELEMENT NUMBER 

 622785 

5c. PROJECT NUMBER 

 A790 

5d. TASK NUMBER 

 211 

6. AUTHOR(S) 

J. R. Gossman, Rebecca P. Mauzy, Charles G. Heiden, Charlotte H. 

Campbell (Human Resources Research Organization), Michael R. 

Flynn (Northrop Grumman Information Technology), and James W. 

Lussier, and Scott B. Shadrick (U. S. Army Research Institute) 5e. WORK UNIT NUMBER 

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

Human Resources Research Northrop Grumman Information 

 Organization (HumRRO) Technology 

66 Canal Center Plaza, Suite 400 7575 Colshire Drive 

Alexandria, VA 22314 McLean, VA 22102 

8. PERFORMING ORGANIZATION REPORT NUMBER 

10. MONITOR ACRONYM 

 ARI 

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 

 U.S. Army Research Institute for the Behavioral and Social Sciences 

 ATTN: DAPE-ARI-IK 

 2511 Jefferson Davis Highway 

 Arlington, VA 22202-3926

11. MONITOR REPORT NUMBER 

 Research Report 1839 

12. DISTRIBUTION/AVAILABILITY STATEMENT 

 Approved for public release; distribution is unlimited. 

13. SUPPLEMENTARY NOTES 

Contracting Officer’s Representative and Subject Matter POC: James W. Lussier 

14. ABSTRACT (Maximum 200 words): 

One key to the Army’s success in transformation will be a solid process for concept development. The Army needs a means to 

generate, elaborate, refine, describe, test, and validate new concepts relating to doctrine, tactics, techniques, procedures, unit and team 

organization, job allocation, training, leader development, and other aspects of technology integration. One approach is to construct an 

environment that transforms the scale in which situations can be presented realistically and to develop, within that environment, a set 

of tools that can be used to explore selected command group functional performance issues in a methodical fashion. Two tools, a 

scaled-world tool and a concept-development tool, were designed and developed. In addition, six scaled-world events and 10 conceptdevelopment sessions were produced. The scaled-world events and concept-development sessions were formatively evaluated. Based 

on the evaluation results, it appears that both tools have value for concept identification and concept development. Feedback on both 

tools was very positive and generally met the project objectives. In addition, future directions for improvements to and use of the 

synthetic task components are discussed in terms of short-term and long-term requirements and possibilities. 

15. SUBJECT TERMS 

Concept development Concept exploration Future Force 

Scaled-world environment Training Transformation 

SECURITY CLASSIFICATION OF 

16. REPORT 

 Unclassified 

17. ABSTRACT 

 Unclassified 

18. THIS PAGE 

 Unclassified 

19. LIMITATION OF

 ABSTRACT 

 Unlimited

20. NUMBER 

 OF PAGES 

21. RESPONSIBLE PERSON 

 (Name and Telephone Number) 

Ellen Kinzer 

Technical Publications Specialist 

703-602-8047

NSN 7540-01-280-5500 Standard Form 298 
ii
Research Report 1839 

Flexible Methods for Future Force 

Concept Development 

 

J. R. Gossman, Rebecca P. Mauzy,

Charles G. Heiden, and Charlotte H. Campbell

Human Resources Research Organization 

Michael R. Flynn

Northrop Grumman Information Technology 

James W. Lussier, and Scott B. Shadrick 

U.S. Army Research Institute 

Armored Forces Research Unit 

Barbara A. Black, Chief 

U.S. Army Research Institute for the Behavioral and Social Sciences 

2511 Jefferson Davis Highway, Arlington, VA 22202-3926 

August 2005 

Army Project Number Personnel Performance 

622785A790 and Training Technology 

Approved for public release; distribution is unlimited.

iii
iv 
FLEXIBLE METHODS FOR FUTURE FORCE CONCEPT DEVELOPMENT 

EXECUTIVE SUMMARY 

Research Requirement: 

 

The U.S. Army must find methods to rapidly incorporate technology, must make the 

necessary systemic changes so that future Soldiers can use the assets efficiently and effectively, 

and must do so at a rate that outpaces our competitors. In short, technology must be introduced 

to enhance rather than impair operational effectiveness. 

A key to success is a solid process for concept exploration. There is a tremendous need 

for a powerful, yet flexible, approach to concept development and knowledge exploration that 

can cope with the unique difficulties of envisioning and studying future concepts. The Army

needs a means to generate, elaborate, refine, describe, test, and validate new concepts relating to 

doctrine, tactics, techniques, procedures, unit and team organization, job allocation, training, 

leader development, and other aspects of technology integration. 

Procedure: 

Two tools were developed to support elicitation of future tactics. A scaled-world tool 

and a concept-development tool were designed and developed to examine situations related to 

the See function as it might operate in the future. The tools provide a means to present 

individuals with situations that stimulate their thinking in the context of realistic situations or to 

present situations in which only a limited number of variables are operative in order to assess 

how individuals will perform. Six scaled-world events and 10 concept-development sessions 

were produced. The scaled-world events and concept-development sessions were formatively 

evaluated using reviews by military personnel. 

Findings: 

Based on formative evaluation (FE) results, it appears that both tools have value for 

concept identification and concept development. Feedback on both components was very 

positive and generally met the project objectives. One exception is that the components are not 

as easily modifiable as originally anticipated. 

Utilization of Findings: 

The results of this project were transitioned to the Unit of Action Maneuver Battle Lab 

(UAMBL). The tools benefit those involved in the identification, refinement, and testing of 

concepts related to the Army’s transformation to the Future Forces environment. Both tools 

serve an important function in the initial stages of Future Forces development by providing 

mechanisms that can be employed in relatively simple settings using existing equipment in order 

to identify concepts. 

v 
vi 
FLEXIBLE METHODS FOR FUTURE FORCE CONCEPT DEVELOPMENT 

CONTENTS

Page 

INTRODUCTION ..................................................................................................................... 1 

PURPOSE AND OBJECTIVES................................................................................................ 2 

THE SEE FUNCTION .............................................................................................................. 3 

 Scaled-World Tool............................................................................................................... 4 

Concept-Development Tool................................................................................................. 4

Comparison of Scaled-World and Concept-Development Tools ........................................ 5 

 Scaled-World Events ........................................................................................................... 5 

Concept-Development Sessions .......................................................................................... 9 

Formative Evaluation of the Concept-Development Sessions ............................................ 13 

 Modifications to the Tools................................................................................................... 14 

 Conclusions.......................................................................................................................... 15 

REFERENCES .......................................................................................................................... 17 

Appendix A List of Acronyms.................................................................................................. A-1 

Appendix B Scaled-World Event Specifications...................................................................... B-1 

Appendix C Scaled-World Environment User Instructions ..................................................... C-1 

Appendix D Concept-Development Session Specifications..................................................... D-1 

Appendix E Administration Guide for Concept-Development Sessions for 

the Future Force .............................................................................................. E-1 

List of Tables 

Table 1. Description of SWE Tasks...............................................................................................6 

Table E-1. Current Concept-Development Sessions ..................................................................E-5 

List of Figures 

Figure 1. Scaled-world tool interface showing the user panels.............................................. 7 

Figure 2. Scaled-world tool with sensor feed displayed......................................................... 8 

Figure 3. Typical instruction screen from a concept-development session............................ 9 

Figure 4. Typical information screen from a concept-development session with hot links 

 to bring up additional information about the scenario......................................... 11 

Figure 5. Typical supporting map from a concept-development session with hot links at 

 the bottom allowing access to additional maps and pictures................................ 12 

CONTENTS (continued)

vii
 Page

Figure 6. Aerial photo of the area of operations from a concept-development session 

 1 with hot links at the bottom allowing access to additional maps and pictures.. 12 

Figure C-1 Scenario selection dropdown menu .......................................................................C-2 

Figure C-2 Scenario version dropdown menu ........................................................................ C-2 

Figure C-3 Situation details button..........................................................................................C-2 

Figure C-4 Initial scaled-world tool screen layout ................................................................... C-3 

Figure C-5 Sensor properties control........................................................................................ C-5 

Figure C-6 Loitering sensor...................................................................................................... C-6 

Figure C-7 Audible sensor output............................................................................................. C-7 

Figure C-8 Audible sensor controls.......................................................................................... C-7 

Figure C-9 Visual sensor image................................................................................................ C-8

viii
FLEXIBLE METHODS FOR FUTURE FORCE CONCEPT DEVELOPMENT 

Introduction 

The U.S. Army is in the process of transforming to a Future Force following the 

guidelines laid out several years ago by the Joint Chiefs of Staff (JCS1

) in the Joint Vision 2020

(JCS, 2000). The transformation is based on the overarching concept that U.S. forces, acting 

jointly and in combination with forces from other nations, will have the capability to find and 

defeat any enemy before that enemy poses a direct threat. A key component of this capability 

involves the concept of information dominance which will depend, in large part, on 

technological advances in the areas of advanced sensors, artificial intelligence, and networkcentric command and control systems. However, it will be equally dependent on the ability of

future Soldiers to use these new assets efficiently and effectively. 

The transformation will most likely be dramatic, producing significant change in how the 

U.S. Army operates; it will be widespread, involving multiple interdependent effects across the 

range of Doctrine, Organization, Training, Materiel, Leadership and Education, Personnel, and 

Facilities (DOTMLPF); and it will be continual, taking advantage of technology that will itself 

be advancing rapidly throughout the century. In addition, the future of U.S. Army dominance 

will not be as dependent as it previously was upon the Department of Defense (DOD) designing, 

building, and protecting unique technologies. Instead, it will most likely be built on designing, 

implementing, and protecting the world's best technology integration process (Carter & White, 

2000). This is a different way of doing business. It is predicated on the notion that we as a 

nation will not be able to control the development or proliferation of new technologies in a 

global marketplace. Rather, we will need to become the best and fastest integrator of new 

technologies – into the hardware, into the software, into our doctrinal and organizational 

systems, and into our training and leader development processes. 

A key to success in the Army’s transformation is a solid process for concept 

development. The Army needs a means to generate, elaborate, refine, describe, test, and validate 

new concepts relating to doctrine, tactics, techniques, procedures, unit and team organization, 

job allocation, training, leader development, and other aspects of technology integration. How 

do we best use the total Army team – warfighter, scientist, theorist, analyst, and engineer – in the 

development process? Generally the Army has used two methods. The first is to have a strong 

theorist or analyst develop and present new concepts, that is, an Army thinker thinks and then 

writes. A good example of this is provided by Wass de Czege and Sinnreich (2002). While 

strong creative thinkers are clearly a vital part of the effort, some drawbacks are evident in the 

“just let thinkers think” method of concept development. The results are often very general, 

have not benefited from concrete attempts to implement the ideas, have not uncovered 

difficulties through implementation, can be hard to communicate, and do not easily benefit a 

team of specialized contributors working jointly. Several efforts less successful than the abovecited example could also be mentioned, resulting in “cut-and-paste” concept documents that 

were poorly thought out, too general, impractical, unrealistic, fragmented, disjointed and 

contradictory. 

 

1

 A list of all acronyms used in this report is included in Appendix A. 

1
A second Army method is to stand up a replica of the new system and conduct a unit 

exercise in simulation, such as a focused Combat Training Center (CTC) rotation or Advanced 

Warfighter Experiment (AWE). Such full-scale events can be effective demonstrations of 

moderate- to well-developed concepts but have disadvantages, especially in the earlier stages of 

concept formation. They are expensive and require great effort and coordination to conduct, do 

not allow flexible manipulation of variables, present great measurement difficulties, and lack the 

repetition necessary to reach well-founded conclusions. Further, the complexity of conducting 

the event tends to overwhelm the experimental intent. 

There is a need to investigate concept development related to using methods that provide 

valid and reliable information while avoiding the difficulties described above. One approach is 

to construct an environment that transforms the scale in which situations can be presented 

realistically. Lickteig et al. (2002) make a strong case for developing such an environment. 

They describe small-scale transformation environments as “empirical venues that afford users, 

researchers, and developers the ability to customize tasks and conditions in order to iteratively 

explore and transform concepts into viable, human-centric solutions” (p. 6). They further state 

such an environment can serve as a middle ground between the situational 

complexity inherent in field research that resists definite conclusions, and the 

situational paucity of laboratory research that defies useful conclusions (Ehret, 

Gray, & Kirschenbaum, 2000). By design, it preserves key functional 

relationships based on questions of interest to evaluators or trainers, while paring 

away other functions that might confound answering those questions (p. 8). 

A scaled environment, according to Lickteig et al. (2002), serves at least two important 

functions. It provides a breeding ground in which to identify and refine innovations for larger 

environments, and it provides a proving ground in which to assess and resolve key issues from

larger environments. 

The idea of the scaled environment serving as a breeding ground is consistent with the 

discussion included in the Joint Operations Concept – Full Spectrum Dominance Through Joint 

Integration (2003), which focuses on identification of enabling concepts or subordinate concepts 

that will need to be developed with sufficient detail to directly link to capabilities. Identification 

of such enabling concepts is a primary function of scaled transformation environments. Gold 

(1999) further supports this use of scaled transformation environments and warns against 

focusing on so-called “Superbowl” culminating events. Instead, the concept identification 

process should be built on a “broad-based iterative approach using a variety of tools, simulations, 

and venues that can lead to early discovery. Early on, searching for questions is more important 

than finding answers” (p. 26). 

Purpose and Objectives 

These thoughts and arguments about concept exploration, identification, and 

development come at a critical time for the U.S. Army as it engages in a major effort to 

transform its operations. The U. S. Army Research Institute for the Behavioral and Social 

Sciences (ARI), as part of its Science and Technology Objective on Methods and Measures of 

2
Commander-Centric Training, conducted an analysis to describe the C4ISR functional and task 

requirements for FCS (Lickteig, Sanders, Durlach, Lussier, & Carnahan, 2003). Some examples 

of the Command, Control, Communications, Computers, Intelligence, Surveillance and 

Reconnaissance (C4ISR) functions are: control and interpret input from a heterogeneous set of 

advanced sensors to mentally construct an accurate picture of enemy activity and intent (See), 

control the movement and activity of friendly manned and unmanned systems to maintain 

desired movement rates and formations (Move), distribute a variety of indirect and direct effects 

over a set of targets (Fire), assess the effectiveness of a plan and the combat instruction sets 

provided to robotic elements (Rehearse), and other functions such as Sustain, Train, 

Communicate, Adapt, and Plan. A set of tools is needed that can be used to explore selected 

command group functional performance issues in a methodical fashion. The tools must be 

flexible, that is, they must be capable of being readily tailored to respond to a wide range of 

conceptual constructions. The current project developed two tools – scaled world events and the 

concept development sessions. Scaled world events use simulation to perform a function in 

isolation. Concept development sessions use realistic vignettes without simulation to allow a 

participant to generate and evaluate ideas about future tactical issues. 

The See Function 

The tools were developed to address the “See” function. The application of emerging 

sensor capabilities will greatly affect how Army personnel see the battlefield in the near future. 

New technology should enhance collective, real-time common understanding of the battlefield 

situation over great distances and allow commanders to make rapid and accurate maneuvers in 

any operating environment. The enhanced capabilities will present information from robotic air 

and ground sensors, and from humans, using graphics and text. The deluge of information, 

however, will require users to ‘fuse’ information from these various sources into an 

understanding of the situation. This alone requires innovative methods of training, new concepts 

of presentation and succinct approaches to information presentation. Additionally, all other 

battlefield functions rely on input from the See function for successful execution; thus, concepts 

and methods developed relating to the See function will have broad effect and application within 

Future Forces. 

An initial statement of the See function was formulated as follows: 

The Command group controls and interprets information to mentally construct an 

accurate picture of the environment and of threat dispositions, activities, 

capabilities, and intent in relation to friendly dispositions, activities, capabilities, 

and desires. 

The function was analyzed into its component tasks and activities as projected for the 

environment, and the tasks and activities were then examined to identify those that would be 

included in a synthetic environment for concept exploration. Some components of the See 

function that were identified are: decide what to look for, decide where to look, decide what to 

look with, set filters, fuse information on screen(s), review information, interpret information 

(analyze and synthesize), evaluate information, track forces, verify information, share 

information, and identify information gaps. 

3
Scaled-World Tool 

The scaled-world tool allows participants to deploy and control various robotic sensors to 

perform surveillance and reconnaissance tasks in isolation, that is, separate from a full tactical 

context. Various types of robotic entities (aerial, mobile ground, or immobile ground), equipped 

with different types of sensors (e.g., visual, infrared, auditory), are represented. It supports the 

conduct of scaled-world events (SWE). Scaled-world events are preset mission scenarios in 

which an individual or small team performs a task or set of tasks that emulate and isolate 

selected components of the See function in a laboratory setting. Each scenario specifies the 

mission goal and the type of sensors that are available. For example, a SWE might require the 

participants to locate a camouflaged enemy unit. The participant can see the location of their 

tasked sensors on an electronic map. Simultaneously, in additional windows, they can view the 

sensor images and associated data. The primary purpose of the scaled-world tool is to present an 

accurate portrayal of the task requirements so that concept development is grounded by a 

realistic appreciation for the human performance difficulties associated with conducting the task. 

The scaled-world tool also allows researchers, by measuring and assessing the performance of 

participants, to draw conclusions that are applicable to future Army conditions. Issues such as 

human performance capabilities, training requirements, and task requirements can be 

investigated. 

Concept-Development Tool 

The concept-development tool supports the conduct of concept-development sessions

(CDS). The notion of a CDS is that of a snapshot, or a series of snapshots, or a short 

presentation that places a decision-making, problem solving, or evaluation requirement on the 

participants. A CDS does not respond dynamically to participant input. However, CDSs can 

portray highly realistic situations and entities that are often very difficult to represent well in 

tactical simulation, for example, underground caves, car bombs, cultural factors, and situations 

arising from support and stability operations. 

The CDSs and the concept-development tool, which are independent of the scaled-world 

tool, place the participant(s) in a situation in which a decision, evaluation, or other problem is 

posed that relates to the target functions and tasks represented in the scaled-world environment. 

Each CDS includes a specification or questionnaire for collecting participant thoughts, questions, 

conclusions, and recommendations. 

The CDSs can be used in a variety of ways, including the following broad areas: 

• To represent “What now?” instances, portraying a military situation and asking the 

participant(s) to generate and evaluate courses of action (COA). 

• As “What if?” situations that explore the range of employment. For example, if the 

enemy used x method of infiltrating, how should we adjust surveillance tactics? How 

would the sensor plan need to be changed under y terrain and weather conditions?

4
• As acquisition-oriented human factors tools. For example, if the Army were to 

develop z sensor capability, how should that be employed? 

• To assess the effect of infusions of new technologies by presenting realistic situations 

and obtaining participant reactions. 

Comparison of Scaled-World and Concept-Development Tools 

 

The scaled-world and concept-development tools complement each other in their 

strengths and weaknesses. An SWE represents a “scaled-down” version of a situation in which 

variables of interest are isolated and complicating variables are eliminated to the extent possible. 

In the SWE, the participant is able to manipulate sensors and perform the task in an authentic 

manner, allowing the participant to directly experience human performance requirements. A 

CDS presents a complex situation with a full range of tactical considerations. The participant is 

able to explore realistic situations but cannot directly interact with them. The CDS facilitates an 

environment in which new concepts can be explored, without being constrained by simulation 

implementation. It allows the researcher to define and focus emerging future issues and to 

accumulate the judgments of participants. Experienced together, concept development is 

enhanced as the case-based quality of the CDS stimulates exploration of ideas and the 

performance requirements of the SWE impose realistic evaluation on the feasibility of the ideas. 

Scaled-World Events 

Six SWEs were produced based on capabilities expected to exist in the future. Each 

SWE included a narrative and graphical presentation of the situation involved, the anticipated 

activities of the participants, and anticipated performance measures. For example, factors that 

are expected to affect performance in the scaled-world include threat type and size, number and 

types of sensors available, and rate of information being presented to participants. 

The first SWE is designed at the level of a Brigade Combat Team (BCT)2

. This event is 

very basic and serves the function of orienting participants to the scaled-world environment. 

The only variables that are manipulated are the number and types of sensors available. 

Participants begin the event with a terrain map showing their area of operations (AO). They 

receive a message from higher informing them that threat force movement in an adjacent AO 

was detected with the threat possibly moving into their AO. Participants attempt to locate and 

identify the threat forces by deploying sensors and obtaining information from sensors owned by 

higher or adjacent units. The threat forces consist of a stationary and camouflaged scout platoon. 

The event is conducted during daylight conditions. Skies are clear with no obstacles to visibility 

related to weather. The event concludes when the unit detects, locates, and identifies the threat 

force. A brief description of each SWE is shown in Table 1. Detailed descriptions of all six 

SWEs are in Appendix B. 

 

2

 In the original materials, the BCT is referred to as Unit of Action (UA). 

5
Table 1. Description of SWE tasks 

SWE 1: Detect stationary troops. 

SWE 2: Locate mobile-based biological threat. 

SWE 3: Determine enemy course of action. 

SWE 4: Determine whether shots fired are a party or a battle. 

SWE 5: Identify when a critical event occurs. 

SWE 6: Conduct battle damage assessment. 

Representative screens from the scaled-world tool are shown in Figures 1 and 2. The 

scaled-world tool user interface is composed of four panels, as shown in Figure 1. The first is 

the mode selection area panel, which displays in the upper left portion of the screen and provides 

the user with the capability to manipulate the map, deploy sensors, and select sensor feeds to 

view. Those functions are controlled by the “Map Navigation,” “New Sensors,” and “Current 

Sensors” buttons at the top of the screen. For aerial and mobile ground sensors, users can 

establish a route of travel as well as a loiter pattern that the sensor will follow. The second 

panel, on the right, is the map area panel, which takes up most of the screen. The map 

represents a portion of Fort Hood terrain. Following sensor deployment, the map displays the 

sensor’s location as well as its type. For example, in the center of the map area panel of Figure 

1, three deployed sensors are shown. The third panel, in the lower left is referred to as the 

situation area, and contains a text description of the SWE situation. The fourth panel, lower 

right, is the sensor output area that displays messages sent by non-visual sensors (e.g., seismic, 

audio, biological agent). A message is displayed in the sensor output panel whenever a sensor 

comes within range of a target it is capable of detecting. The user can then display the sensor 

output for additional information including playback of audio recordings made by the sensor. 

For each visual sensor deployed, a new window displays the corresponding sensor image as 

shown in Figure 2. Three types of visual images are available: a daylight visual image, an 

infrared image, and a thermal image. A set of instructions (See Appendix C) provides 

information on loading exercises, navigating the map, and deploying and monitoring sensors. 

6
Mode Selection Area 

Sensor Output Area

Situation

Area 

Map Area

Figure 1. Scaled-world tool interface showing the user panels. 

 The scaled-world tool can collect limited data in a Microsoft Access® database. It can 

record the sensors that were deployed and whether a sensor ever came within range to detect the 

target. It cannot, however, record whether the user actually detected the target. 

7
 

Figure 2. Scaled-world tool with sensor feed displayed. 

The scaled-world tool and events were developed using Java and open-source software 

including: 

• Gentoo Linux (Operating System). 

• Netbeans 3.4 – 3.5 (Integrated Development Environment). 

• OpenMap (Map Application Programming Interface). 

• IzPack (Installer). 

• Sun JDK 1.4.1-1.4.2 (Java Virtual Machine). 

• OpenOffice (Office Suite) 

In addition, terrain and visual imagery maps for Fort Hood were obtained from the National 

Imagery and Mapping Agency (NIMA). 

Developing new events or modifying existing ones requires some level of expertise with 

Java. Simple modifications such as changing the characteristics of a sensor or moving the 

location of a target involve relatively minor changes to lines of existing code. Adding new 

8
sensors or targets, making major modifications to an event, or creating a new event 

requires more extensive coding. 

Concept-Development Sessions 

A concept development session (CDS) is a facilitated discussion revolving around a 

question or issue. Each CDS presents a depiction of a situation designed to start the participants 

thinking about how the Army might function in the future. The goal is to generate concepts for 

future operations that can be used to develop ideas about tactics, techniques, and procedures 

(TTP), e.g., how a particular capability might best be used. Each participant is shown a problem

scenario relevant to an important Army issue and asked to think through the specific situation. 

Then, a broader discussion is initiated to include considerations beyond the scenario presented in 

order to test the generality of the solutions devised for the specific scenario. A succession of 

participants confronting the same CDS helps to refine the concepts. In total, ten CDSs were 

developed. 

Each session starts with an instruction screen that identifies the role the participant is to 

play, the tactical mission, and the goal of the session. Figure 3 shows the instruction screen for 

one of the CDSs. 

Figure 3. Typical instruction screen from a concept-development session. 

9
When the participants click the Begin link, they see additional screens such as those 

shown in Figures 4, 5 and 6. The CDSs are designed to run in a dual-screen environment. “Hot 

links” on each screen allow the participant to obtain additional information. The participants are 

given as much time as desired to examine the information and are encouraged to “think out loud” 

as they study the CDS scenario. After reading the introductory page on the computer screen, the 

participants work through the CDS by studying and analyzing the information provided, 

formulating questions and plans, and indicating the need for additional information. 

 

The first CDS, for example, presents a tactical scenario that includes a detailed written 

account of civilian activities in a town known to harbor threat personnel on existing blacklists. 

The participants view a series of photographs including groups of armed persons congregated 

near the town’s central hall. They have access to maps along with graphic displays showing 

their unit task organization and additional assets available. They are told that the unit’s task in 

this situation is to determine the nature of the threat in the town and make a report to the 

Battalion Commander. 

The structured interview elicits responses aimed at shaping concepts and tactics, 

techniques, and procedures related to the situation. The specification of each CDS contains a 

number of pre-designed questions for the facilitator to ask. However, in each session facilitators 

are free to ask additional clarifying questions and follow-up on the course of participants’ unique 

ideas. Some pre-designed questions used for the first CDS are: 

1. What steps would you take to complete this task? (Follow-up and clarify as necessary) 

2. What additional assets/capabilities, if any, would you liked to have had to complete this task, 

and how would you have employed those assets?

3. Is there information that you did not have that would have helped you complete this task? 

What information? How could it best be presented?

4. Is this an appropriate task for the unit indicated? Should it be a higher level task? Lower 

level? 

Appendix D contains descriptions of all 10 CDSs. 

 

10
Figure 4. Typical information screen from a concept-development session with hot links to bring 

up additional information about the scenario. 

11
Figure 5. Typical supporting map from a concept-development session with hot links at the 

bottom allowing access to additional maps and pictures. 

Figure 6. Aerial photo of the area of operations from a concept-development session 1 with hot 

links at the bottom allowing access to additional maps and pictures. 

12
One concept for use of a CDS is as follows. Initially, one person at a time participates. 

After several participants, responses and process observations are used to modify the CDS to 

allow it to become more realistic, more focused, and to incorporate participants’ ideas for 

additional trials. This process of refinement and presentation is repeated until, in the judgment 

of the researcher, no new information is being generated. At that point, final modifications are 

made and the CDS is presented to small groups. It is anticipated that additional information 

might be forthcoming from the group presentation. Again, the process is repeated until it 

reaches the point of diminishing returns. After analysis of data from multiple participants, a 

composite response to the issue is produced. Further exploration may use new CDSs or a SWE 

to systematically test the group solution. To facilitate administration of CDSs, an 

Administration Guide was developed (see Appendix E). This guide provides detailed 

information on presenting sessions, collecting data, modifying existing sessions, and creating 

new ones. (See Shadrick, Lussier, & Hinkle, 2005) for further discussion concept development 

in future environments.) 

The concept development tool was designed to allow for modular integration of new 

CDSs in a fashion that allows for easy substitution of new sessions and easy modification of

existing sessions. The CDSs themselves were developed using commercial off-the-shelf 

software including Bluesky ® RoboHTML HelpTM, Microsoft ® PowerPointTM, and Adobe ® 

Photoshop TM. 

 

Formative Evaluation of the Concept Development Sessions 

To evaluate whether the CDS concept was feasible, seven trials were conducted with nine 

participants. Each trial included two CDSs and was approximately two hours. The participants 

included one colonel, two majors, five captains, and one retired colonel. Five sessions were 

conducted with a single participant; two sessions were conducted with two participants. All the 

sessions were facilitated by a military subject-matter-expert and a research psychologist. 

 

A number of conclusions may be drawn from the formative evaluation. Encouraging 

results were obtained on the ability of the CDS format to elicit participant responses. Some

participants were stimulated by the CDSs to provide extremely unique, creative, and wellreasoned responses. There were, however, considerable individual differences. Some

participants – even some who had spent time studying the Army’s emerging future force 

doctrinal concepts – had difficulty critically evaluating those concepts. Some participants 

appeared to be able to transition into looking at the situation from the viewpoint of how things 

might be in the future (2015-2020) quite readily. Others required considerable, and even 

forceful, encouragement. Soldiers have a tendency to look at situations the way they have been 

trained (i.e., mission, enemy, terrain, troops, time, civilians) and they often tend to focus on a 

COA analysis. In a situation such as that provided in the CDS, it may require some training or 

encouragement to lead their thinking toward, “How else could we do things particularly if we 

had access to assets that don’t even exist today?” On the other hand, once they did start thinking 

futuristically, they were (for the most part) very creative. They came up with ideas for a variety 

of capabilities including sensors that could “see through things,” that could sense the presence of 

humans through exhaled gasses, and that could talk to people in their own language. If the 

13
function of the CDSs is to stimulate individuals to think creatively and futuristically, they appear 

to work. 

Concerning the CDS process itself, comments were generally positive as to its use as a 

concept development tool. Most participants either directly or indirectly indicated that the CDS 

tool would work. On the other hand, specific comments regarding each CDS tested were 

received, including some that conflicted (e.g., too much information; not enough information.) 

 

Modifications to the Tools 

This section discusses considerations for future modification, enhancement, and 

expansion of the SWE and the CDS tools. 

In the case of the SWE, automated data collection capabilities are limited. The system

can record when and where sensors were placed and whether they would ever have been capable 

of detecting a target. It does not actually record whether a target was detected; that would 

require direct observation or direct reporting. A feature could be added to the SWE that would 

allow the participants to indicate targets detected and their locations. The system would be able 

to record scenario events and certain aspects of human-computer interaction in a log file 

allowing the researcher to review the session in speeded up time and to pause and restart (e.g., 

VCR like controls). Replay of map screen and sensor information windows viewed by the 

player would be included. Time into the mission should be continuously displayed. The 

playback should also display target information on the map, annotations regarding the time of 

key events such as a target falling inside sensor range, time and content of operator reports, and 

whether these were operator initiated or prompted. Observers of the playback should be able to 

toggle sensor fans and UV routes on or off. Screenshots of the playback should be easily 

captured and saved for later analysis, e.g., analysis of the amount of area covered by sensors at a 

particular time. 

A number of suggestions would improve the sensor behavior in the SWE. Currently, the 

operator can set a route for each platform. When the route is entered, the platform immediately 

starts on its mission and the route disappears from the map. The fact that the route disappears 

makes it difficult for the operator to coordinate the routes of multiple platforms. The operator 

should be given the ability to display routes (toggle on or off) and to display sensor fans (toggle 

on or off) to show area of sensor coverage. The fact that the sensor deploys immediately after 

the route is set makes it difficult for the operator to synchronize multiple platforms. A planning 

mode would allow better control of sensor missions. Also, the SWE should be upgraded to make 

it easier to retask sensors mid-mission (speed, altitude, and route). 

In the SWE, participants can control multiple sensors and view the images from those 

sensors in separate windows. It can be difficult to associate the sensors on the map with their 

corresponding images. Information on the map display should be better linked to information in 

the individual sensor windows, for example, passing the mouse over a sensor window could 

highlight the location of the associated UV on the map. Conversely, clicking on a sensor on the 

map could cause some clear indication in associated sensor window, or open the sensor window 

if it is not already open. 

14
Additional improvements to the SWE include adding a feature to have the sensors 

automatically return “home” after a set time or after completing a pattern; incorporating 

additional loiter patterns; and having mobile sensors begin their routes from a point of departure 

determined by scenario and represented on the map when the scenario is initiated. 

Further suggestions for SWE improvement are converting the latitude/longitude process 

into a grid coordinate system more familiar to Soldiers and modifying the user interface to 

supply information on the coordinates for specific locations, e.g., a text window could display 

the coordinates of the mouse cursor. Also, the user interface should be modified to indicate the 

current zoom level. 

The CDS, because of its inherent flexibility, did not result in such an extensive list of 

desirable modifications. One goal for the CDSs was not fully met – the goal of making them

easily modifiable. One notion behind the approach was that as users started to identify concepts 

and procedures for a CDS, the CDS could be easily modified to incorporate their ideas. The 

current CDSs are modifiable, but only to a limited extent. As the computer-based CDSs were 

constructed, the developers became too tempted to add features to enhance presentation and ease 

of use and lost sight of the goal to make the CDSs easily modifiable. To modify the CDSs, a 

good working knowledge of PowerPoint and Compiled Help Files is needed. Future effort will 

need to be directed at how best to achieve the goal of ready modifiability, including the 

possibility of developing and presenting events and sessions in completely different ways. 

Conclusions 

This report described a project to develop tools for generating and testing concepts and 

ideas for situations or environments that are beyond the scope of current tactics, techniques, and 

procedures. It examined a process for exercising actions in a relatively controlled environment 

in which only a limited number of variables are allowed to have impact. It also examined 

another process for presenting situations or scenarios in a relatively detailed manner in order to 

stimulate the thinking of Soldiers and others who can provide valuable information related to the 

current transformation efforts underway in the U.S. Army. Both the scaled-world and the 

concept-development tools were formatively evaluated and, based on initial analysis, appear to 

be viable methods for concept identification, exploration, and testing. 

As far as future research directions are concerned, several possibilities could be pursued. 

Primarily, the current tools could benefit from additional testing. The effectiveness and 

efficiency of the approach has not been empirically demonstrated. Studies that compare the tools 

to more conventional methods are important. If the approach can be shown to be more flexible, 

faster, or more direct in providing reasoned conceptual information related to systems, 

operations, organizations, and doctrine, then it will establish for itself a position of advantage for 

future developers and decision-makers. 

 Testing should be conducted with a variety of people. In this initial trial, we found 

considerable variability among the nine participants. Some participants needed considerable 

prodding to get them started thinking futuristically. Many participants were not particularly 

aware of Future Force development directions. They are extremely involved in their day-to-day 

15
activities and have limited time to keep up with planning that is occurring for 10 to 15 years 

down the road. One might assume that these would be less productive participants, however, it 

may also be true that those most familiar with the specifics of Future Forces development efforts 

may be the least likely to provide useful information. Once some of the ‘naïve’ participants got 

started, some were highly capable of thinking beyond their own current assignments and view of 

the world. On the other hand, one of the participants directly involved with transition to the 

Future Force was unable to overcome the specific direction, and the specific details, of his 

knowledge of Future Force developments. He spent most of his time explaining the concepts 

described in current Army planning documents (i.e., the party line) and could not be induced to 

think critically about them. Although the tools were designed for concept elicitation, they may 

also have value in selecting people for assignments that require evaluation of future concepts. 

In the current project, researchers had a great deal of latitude to select scenarios – they 

simply had to relate to the See function. In the future, the Army will want to use these tools to 

answer specific questions that arise from different situations. For example, as the “future 

thinkers” attempt to predict how individual Soldiers and units will operate, they will raise 

questions that are best answered empirically. The tools provide at least one approach. Research 

is needed to show how well the methods work when applied to specific new issues. Some

answers to this question may be found in a related ongoing project in which similar tools and 

methods are being used to develop training for a task that lacks recognized experts – interagency 

crisis action planning and execution. 

How can we best use the total Army team – warfighter, scientist, theorist, analyst, and 

engineer – in the concept development process? The tools described here are a preliminary step 

in answering that question. They represent something between simple “brainstorming” and fullblown simulation – providing structure and control without requiring exorbitant resource 

commitments. What other effective tools fall somewhere in between those two extremes? What 

is the best way to use the SWE, CDS, and other new tools in the concept discovery, 

development, and testing process? These remain open questions and are the subject of 

continuing U.S. Army Research Institute for the Behavioral and Social Sciences research. 

16
References 

Carter, A. B., & White, J. P. (Eds.). (2000). Keeping the edge: Managing defense for the 

future. Cambridge, MA: The MIT Press. 

Ehret, B. D., Gray, W. D., & Kirschenbaum, S. S. (2000). Contending with complexity: 

Developing and using a scaled-world in applied cognitive research. Human Factors, 42, 

(1), 8-23. 

Gold, T. (1999). Report of the Defense Science Board task force on DOD warfighting 

transformation. Retrieved from http://www.acq.osd.mil/dsb/warfightingtrans.pdf 

Joint Chiefs of Staff. (2000). Joint Vision 2020. Retrieved from http://www.dtic.mil/ 

jv2020/jvpub2.htm

Joint operations concept: Full spectrum dominance through joint integration (Draft 

Version 4.8). (10 February 2003). 

Lickteig, C. W., Sanders, W. R., Durlach, P. J., Lussier, J. W., & Carnahan, T. J. (2003). Human 

performance essential to battle command: Report on four Future Combat Systems

command and control (FCS C2) experiments (ARI Research Report 1812). Alexandria, 

VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Lickteig, C. W., Sanders, W. R., Shadrick, S. B., Lussier, J. W., Holt, B. J., & Rainey, S. J. 

(2002). Human-system integration for future command and control: Identifying 

research issues and approaches (ARI Research Report 1792). Alexandria, VA: U.S. 

Army Research Institute for the Behavioral and Social Sciences. 

Shadrick, S. B., Lussier, J. W., & Hinkle, R. K. (2005). Concept development for Future

domains: A new method of knowledge elicitation (ARI Technical Report 1167). 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Wass de Czege, H. & Sinnreich, R. H. (2002). Conceptual foundations of a transformed U.S. 

Army (Land Warfare Paper No. 40). Arlington, VA: Association of the Unites States 

Army. 

17
Leave blank. Next page starts on an odd page number.

18
Appendix A 

List of Acronyms 

AO Area of Operation 

ARI U.S. Army Research Institute for the Behavioral and Social Sciences 

AWE Advanced Warfighter Experiment 

BCT Brigade Combat Team

BDA Battle Damage Assessment 

Bde Brigade 

Bn Battalion 

CA Combat Assessment 

CAB Combat Aviation Brigade 

CCIR Commander’s Critical Information Requirement 

CDS Concept-Development Sessions 

C4ISR Command, Control, Communications, Computers, Intelligence, Surveillance and 

Reconnaissance 

COA Course of Action 

COP Common Operational Picture 

CTC Combat Training Center 

DOD Department of Defense 

DOTMLPF Doctrine, Organization, Training, Material, Leadership and Education, Personnel, 

and Facilities (DOTMLPF) 

FCS Future Combat Systems 

FE Formative Evaluation 

GCM Graphic Control Measures 

HQ Headquarters 

IPB Intelligence Preparation of the Battlespace 

ISG Intuitive Summary Graphics 

ISR Intelligence, Surveillance, and Reconnaissance 

JCS Joint Chiefs of Staff

MI Military Intelligence 

MTW Major Theater War 

NIMA National Imagery and Mapping Agency 

O&O Operational and Organizational 

A-1
ORD Operational Requirements Document 

RSTA Reconnaissance, Surveillance, and Target Acquisition 

SME Subject Matter Expert 

SSC Small Scale Contingency 

SWE Scaled-World Events 

TTP Tactics, Techniques, and Procedures 

UA Unit of Action 

UAMBL Unit of Action Maneuver Battle Lab 

UAV Unmanned Aerial Vehicle 

UE Unit of Employment 

UGV Unmanned Ground Vehicle 

WMD Weapons of Mass Destruction 

A-2
Appendix B 

Scaled-World Event Specifications 

B-1 
SCALED-WORLD EVENT 1 SPECIFICATION 

Title Detect and identify threat forces 

General 

description 

In this event, participants attempt to locate and identify threat forces within 

their area of operations (AO) by deploying sensors and/or obtaining 

information from sensors owned by higher or adjacent units. The event is 

triggered by a message from higher stating that reports of possible threat 

forces in their AO have been received. 

This event is for the Unit of Action (UA) command group (with two members 

– the commander and the intelligence, surveillance, and reconnaissance [ISR] 

Coordinator – participating). 

This event is very basic and serves the function of orienting participants to the 

scaled-world environment as well as providing a research opportunity. The 

only variables that are manipulated in this event are the number and types of 

organic sensors available to the unit, and the numbers and types of nonorganic sensors from which the unit can obtain information. 

This event measures performance on the dependent variables as a function of 

manipulating the independent variables; however, it also examines process 

data. That is, it records and analyzes the specific activities, decisions, and 

other process data to the extent possible in order to provide a more complete 

understanding of the measured relationships between the independent and 

dependent variables. 

Participants begin the event with a terrain map showing their AO. They 

receive a message informing them that threat force movement in an adjacent 

AO was detected with the threat possibly moving into their AO. This 

provides the cue to conduct electronic surveillance to detect and identify the 

possible threat forces. 

Threat forces consist of a scout platoon. They are stationary and 

camouflaged. 

The event is conducted during daylight conditions. Skies are clear with no 

obstacles to visibility related to weather. 

The event concludes when the unit detects, locates and identifies the threat 

force. 

Tasks ▪ Decide what to look for. 

▪ Decide where to look. 

▪ Decide what to look with. 

▪ Review information. 

▪ Interpret information. 

▪ Evaluate information. 

B-2 
Independent 

variables 

▪ Number and type of organic sensors. 

▪ Number and type of non-organic sensors. 

Dependent 

variables 

▪ Time to detect threat forces. 

▪ Accuracy of detection and identification. 

Process data ▪ Observations/data of participant performance/actions during 

completion of the event, collected automatically. 

▪ Participant responses/reactions, collected during post-event 

debriefing. 

Measurement 

plan 

The two dependent variables (time required to locate and identify threat 

forces and accuracy of location and identification) are measured by 

observation. Two observers monitor each event and record start-time and 

end-time. The observers also record the accuracy of the location and 

identification of the threat as either correct or incorrect. These measures 

provide the primary basis for comparing the effects of manipulating the 

independent variables of types and numbers of organic and non-organic 

sensors available to the participants. In addition, the scaled-world 

environment is constructed to collect this data to the extent possible. 

In addition to these measures, the complete event session is videotaped in 

order to capture process data. The observers and the scaled-world 

environment also record process data as the event proceeds. They record the 

steps/actions the participants used in deploying sensors, in processing 

information received from the sensors, data received from the sensors, 

decisions made, interactions between participants, and other process 

information as appropriate. The observers employ an observation checklist. 

However, it will not be used restrictively. The tapes of the session along with 

the data collected by the scaled-world environment are used to 

verify/supplement observer recorded process data. 

Following the event, the participants are debriefed by the observers in a 

structured interview setting. The observers use a structured interview form

for conducting the debrief session and for recording responses. These 

sessions are also videotaped to allow verification of observer-recorded data. 

Data analysis Primary data analysis consists of (1) comparing the time to locate and identify 

the threat force as a function of numbers and types of organic and non-organic 

sensors; and (2) comparing the accuracy of location and identification of the 

threat force as a function of numbers and types of organic and non-organic 

sensors. 

Secondary data analysis consists of performing qualitative analysis of the 

process data recorded during the event and post-event debriefing. This 

analysis is conducted to provide further explanation of results observed in the 

primary analysis. 

B-3 
SCALED-WORLD EVENT 2 SPECIFICATION 

Title Locate a mobile based biological weapon threat 

General 

description 

The enemy has a biological weapon of mass destruction (WMD) that is 

contained on a mobile platform (probably a medium to large truck). The 

objective of the event is to locate the truck and the weapon. The unit has two 

types of sensors that are capable of detecting biological weapons: a fully 

robotic ground-based sensor capable of detecting a biological weapon from a 

distance less than 100 meters; and a manned sensor capable of detecting a 

biological weapon from a distance less than 10 meters. Both sensors have 

error rates of 25% false positives and 1% false negatives. The unit has 

organic ground and air sensors capable of providing visual (video, infrared, 

radar) images and access to information from non-organic sensors. They also 

have ground sensors capable of providing audio and seismic data. 

This event is for the UA command group (with two members – the 

commander and the ISR Coordinator – participating). 

This event is process oriented. The primary interest is in observing the 

strategy that the group employs in locating the biological weapon. Because 

of the limited range of the biological weapons sensors, the group will need to 

develop a strategy for using its other reconnaissance and surveillance assets to 

isolate probable locations of the platform containing the biological weapon. 

Participants begin the event with a terrain map showing their AO. They 

receive a message informing them that intelligence reports indicate that 

enemy forces are thought to have a biological weapon which has been placed 

on an unknown mobile platform. The location of the platform and weapon is 

unknown; however, its detection and identification are highest priority. 

The event is conducted during daylight conditions. Skies are clear with no 

obstacles to visibility related to weather. 

The event concludes when the biological weapon has been located.

Tasks ▪ Decide where to look. 

▪ Decide what to look with. 

▪ Fuse information on screens. 

▪ Review information. 

▪ Interpret information. 

▪ Evaluate information. 

▪ Verify information. 

Independent 

variables 

None (Note: Future variations of this event could include manipulation of 

variables such as weather conditions, time of day/night, range of biologicalsensors) 

B-4 
Dependent 

variables 

▪ Time to locate the biological weapon. 

▪ Accuracy of detection. 

Process data ▪ Observations/data of participant performance/actions during 

completion of the event, collected automatically. 

▪ Participant responses/reactions, collected during post-event 

debriefing. 

Measurement 

plan 

The two dependent variables (time required to locate and accuracy of 

detection) are measured by observation. Two observers monitor each event 

and record start-time and end-time. The observers also record the accuracy of 

the participants’ location and identification as either correct or incorrect. In 

addition, the scaled-world environment is constructed to collect this data to 

the extent possible. 

In addition to these measures, the complete event session is videotaped in 

order to capture process data which, as stated above, are the date of primary 

interest for this event. The observers also record process data as the event 

proceeds. They record the steps/actions the participants used in deploying 

sensors, in processing information received from the sensors, data received 

from the sensors, decisions made, interactions between participants, and other 

process information as appropriate. The observers employ an observation 

checklist; however, it is not to be used restrictively. The tapes of the session 

along with the data collected by the scaled-world environment are used to 

verify/supplement observer recorded process data. 

Following the event, the participants are debriefed by the observers in a 

structured interview setting. The observers use a structured interview form

for conducting the debrief session and for recording responses. These 

sessions are also videotaped to allow verification of observer-recorded data. 

Data analysis Primary data analysis consists of performing qualitative analysis of the 

process data recorded during the event and post-event debriefing. This 

analysis attempts to identify factors in the participants performance related to 

effective and ineffective results (as indicated by successful location of the 

biological weapon in a timely manner). The analysis also seeks to identify 

factors/variables which would have facilitated performance by examining 

results obtained from the participant debriefing session. 

B-5 
SCALED-WORLD EVENT 3 SPECIFICATION 

Title What course of action is the enemy taking? 

General 

description 

The unit has received a message from UA Battalion showing likely courses of 

action (COA) the enemy may take. A number of these COAs fall within the 

unit’s AO. The unit is directed to initiate reconnaissance and surveillance to 

determine which COAs the enemy is actually taking and to report the 

information to Battalion. 

This event is for the UA command group (with two members – the 

commander and the ISR Coordinator – participating). 

This event measures performance on the dependent variables as a function of 

manipulating the independent variables; however, it also examines process 

data. That is, it records and analyzes the specific activities, decisions, and 

other process data to the extent possible in order to provide a more complete 

understanding of the measured relationships between the independent and 

dependent variables. 

This event provides an opportunity to exercise more of the “SEE” tasks than 

does the “Detect and identify threat forces” event, and also introduces 

additional dependent and independent variables. Independent variables 

include numbers and types of sensors, numbers of COAs lying within the 

Company’s AO, number of COAs the enemy actually follows, and 

unanticipated COAs. Dependent variables include time required to determine 

enemy COAs and accuracy of determining enemy COAs. 

Participants begin the event with a terrain map showing their AO. They 

receive a message from Battalion Headquarters (HQ) detailing selected 

intelligence preparation of the battlespace (IPB) products including potential 

enemy COAs and directing them to conduct reconnaissance and surveillance 

to identify which COAs the enemy is following. 

The event is conducted during daylight conditions. Skies are clear with no 

obstacles to visibility related to weather. 

The event concludes when the Company command group reports enemy 

COA(s) to Battalion HQ. 

Tasks ▪ Decide where to look. 

▪ Decide what to look with. 

▪ Fuse information on screens. 

▪ Review information. 

▪ Interpret information. 

▪ Evaluate information. 

▪ Verify information. 

▪ Track forces. 

▪ Share information. 

B-6 
Independent 

variables 

▪ Number and type of organic sensors. 

▪ Number and type of non-organic sensors. 

▪ Number of COAs lying within the Company’s AO. 

▪ Number of COAs the enemy follows. 

▪ Unanticipated COAs the enemy follows. 

Dependent 

variables 

▪ Time to identify enemy COA after enemy initiates activity. 

▪ Accuracy of identification. 

▪ Timeliness and accuracy of reporting to Battalion HQ. 

Process data ▪ Observations/data of participant performance/actions during 

completion of the event, collected automatically. 

▪ Participant responses/reactions, collected during post-event 

debriefing. 

Measurement 

plan 

The dependent variables are measured by observation. Two observers 

monitor each event and record start-time, time each enemy COA was 

identified, time reports were made to Battalion HQ, and end-time. The 

observers also record the accuracy of the participants’ identification as either 

correct or incorrect. In addition, the scaled-world environment is constructed 

to collect this data to the extent possible. 

In addition to these measures, the complete event session is videotaped in 

order to capture process data. The observers also record process data as the 

event proceeds. They record the steps/actions the participants used in 

deploying sensors, in processing information received from the sensors, data 

received from the sensors, decisions made, interactions between participants, 

communication with Battalion HQ, and other process information as 

appropriate. The observers employ an observation checklist; however, it is 

not used restrictively. The tapes of the session along with the data collected 

by the scaled-world environment are used to verify and supplement observer 

recorded process data. 

Following the event, the participants are debriefed by the observers in a 

structured interview setting. The observers use a structured interview form

for conducting the debrief session and for recording responses. These 

sessions are also videotaped to allow verification of observer-recorded data. 

B-7 
Data analysis Primary data analysis consists of (1) comparing the accuracy of COA 

identification (i.e., the COA(s) the enemy actually follows) as a function of 

numbers and types of organic and non-organic sensors, numbers of COAs 

falling within the AO, number of COAs actually followed, and unanticipated 

COA(s) followed; (2) comparing the time required to identify COA(s) 

followed as a function of these variables; and (3) accuracy and timeliness of 

reporting to Battalion HQ as a function of these variables. 

Secondary data analysis consists of qualitative analysis of the process data 

recorded during the event and post-event debriefing. This analysis is 

conducted to provide further explanation of results observed in the primary 

analysis. 

B-8 
SCALED-WORLD EVENT 4 SPECIFICATION 

Title Party or battle?

General 

description 

U.S. forces are in Alkastan as part of an allied peace keeping operation. 

Alkastan has been engaged in a civil war based, for the most part, upon ethnic 

divisions within the country. Neither faction is particularly friendly toward 

allied forces, and there continue to be numerous skirmishes between factions. 

Allied forces routinely conduct reconnaissance and surveillance using air and 

ground assets, both manned and robotic, and on occasion, these forces come

under fire. On the other hand, there have been two recent incidents in which 

allied forces incorrectly identified actions of civilian populations as 

threatening, firing on them and producing casualties. Allied forces have 

concluded that, from a distance, civilian celebrations are difficult to 

distinguish from threatening actions since both involve small-arms fire, and a 

lot of loud vocalizations. In the present instance, a reconnaissance, 

surveillance, and target acquisition (RSTA) troop has been cued by a brigade 

sensor to investigate a group of about 100 people shouting and firing 

weapons. The troop commander must determine whether this represents a 

threat to his unit. 

This event is for the troop commander. 

The troop has sensors and scouts capable of sending live video to the troop 

commander as well as to higher. It also has organic unmanned aerial vehicles 

(UAVs) capable of providing video and infrared images. It has unmanned 

ground vehicles (UGVs) that can provide seismic information. The unit 

leader has no access to any additional sensor information. 

Two independent variables are manipulated in this event: the time of day 

(daylight or dark) and the threat level (no threat – it’s a celebration; threat to 

other ethnic group; threat to the unit). The dependent variable is 

identification (correct or incorrect) of the threat level. This event measures 

performance on the dependent variable as a function of manipulating the 

independent variables; however, it also examines process data. That is, it 

records and analyzes the specific activities, decisions, and other process data 

to the extent possible in order to provide a more complete understanding of 

the measured relationships between the independent and dependent variables. 

Participants begin the event upon being cued by sensor feed. The troop 

commander must determine where the fire is originating and the probable 

intent of the group initiating fire. 

B-9 
Tasks ▪ Position sensors/soldiers. 

▪ Review information. 

▪ Interpret information. 

▪ Evaluate information. 

▪ Verify information. 

▪ Share information. 

Independent 

variables 

▪ Time of day (daylight or night). 

▪ Threat condition (no threat; threat against another force; threat against 

the platoon). 

Dependent 

variable 

▪ Accuracy of identification of threat condition. 

Process data ▪ Observations/data of participant performance/actions during 

completion of the event, collected automatically. 

▪ Participant responses/reactions, collected during post-event 

debriefing. 

Measurement 

plan 

The dependent variable is measured by observation. Two observers monitor 

each event and record the threat condition identified. The observers also 

record the accuracy of the participants’ identification as either correct or 

incorrect. In addition, the scaled-world environment is constructed to collect 

data to the extent possible. 

In addition to these measures, the complete event session is videotaped in 

order to capture process data. The observers also record process data as the 

event proceeds. They record the steps/actions the participants used in 

deploying personnel and sensors, in processing information received from the 

personnel and sensors, data received from the personnel and sensors, 

decisions made, interactions between participants, communication with 

higher HQ, and other process information as appropriate. The observers 

employ an observation checklist; however, it is not used restrictively. The 

tapes of the session along with the data collected by the scaled-world 

environment are used to verify/supplement observer recorded process data. 

Following the event, the participants are debriefed by the observers in a 

structured interview setting. The observers use a structured interview form

for conducting the debrief session and for recording responses. These 

sessions are also videotaped to allow verification of observer-recorded data. 

B-10 
Data analysis Primary data analysis consists of comparing the accuracy of the threat 

condition identification as a function of the actual threat condition and time of 

day. 

Secondary data analysis consists of performing qualitative analysis of the 

process data recorded during the event and post-event debriefing. This 

analysis is conducted to provide further explanation of results observed in the 

primary analysis. 

B-11 
SCALED-WORLD EVENT 5 SPECIFICATION 

Title Sensor overload?

General 

description 

The commander of the Military Intelligence (MI) Company has been directed 

by Brigade HQ to monitor sensor feeds and report when critical event XX 

occurs (e.g., a column of heavy vehicles passes a specific location.) The 

commander has feeds from sensors that detect and feed video, infrared, and 

seismic signals. The task becomes more difficult as the number and types of 

available feeds increases, and as the number of events the commander is 

monitoring for increases. 

This event is for the MI Company commander. 

This event is designed to examine the processing capabilities of an observer 

as a function of the number and type of information streams being observed, 

and of the number of targets the observer is attempting to detect. Three 

independent variables are manipulated in this event: the number of sensor 

feeds being monitored; the type of sensor feeds being monitored; and the 

number of critical events the participant is monitoring for. Two dependent 

variables are measured: was the critical event detected; and how long after it 

first appeared in a sensor feed was the event detected. This event measures 

performance on the dependent variables as a function of manipulating the 

independent variables; however, it also examines process data. That is, it 

records and analyzes the specific activities, decisions, and other process data 

to the extent possible in order to provide a more complete understanding of 

the measured relationships between the independent and dependent variables. 

Because of the likelihood of data confounding if a repeated measures design 

were employed, each participant participates in a single trial. Participants 

begin the event upon receiving an order from Brigade (Bde) HQ to observe 

sensor feeds 1 – n for the occurrence of critical events a - x. The number of 

sensor feeds varies from 1 to 9. The type varies among video, infrared, and 

seismic. The display size for each sensor remains constant regardless of the 

number of sensor feeds employed. The number of critical events varies from

1 to 3. (Note: the participant does not control sensor placement or type; he is 

monitoring sensor feeds. In addition, in situations where there is a single 

sensor feed, the critical event will be one appropriate to the type of sensor 

being employed). As soon as a critical event is detected, the participant 

reports to Battalion (Bn) HQ via radio contact. 

Tasks ▪ Filter sensor feeds. 

▪ Review information. 

▪ Interpret information. 

▪ Evaluate information. 

▪ Verify information. 

▪ Share information. 

B-12 
Independent 

variables 

▪ Number of sensor feeds. 

▪ Type of sensor feeds. 

▪ Number of critical events. 

Dependent 

variable 

▪ Accuracy of critical event detection. 

▪ Time required to detect critical event. 

Process data ▪ Observations/data of participant performance/actions during 

completion of the event, collected automatically. 

▪ Participant responses/reactions, collected during post-event 

debriefing. 

Measurement 

plan 

The dependent variables are measured by observation. Two observers 

monitor each event and record when the sensor feed indicating the occurrence 

of a critical event occurred. They also record if, and when, the participant 

observed the critical event and made his report to HQ. This process is 

repeated for each critical event included. 

In addition to these measures, the complete event session is videotaped in 

order to capture process data. The observers also record process data as the 

event proceeds. They record the steps/actions the participants used in 

deploying personnel and sensors, in processing information received from the 

personnel and sensors, data received from the personnel and sensors, 

decisions made, interactions between participants, communication with 

Company HQ, and other process information as appropriate. The observers 

employ an observation checklist; however, it will not be used restrictively. 

The tapes of the session along with the data collected by the scaled-world 

environment are used to verify/supplement observer recorded process data. 

Following the event, the participants are debriefed by the observers in a 

structured interview setting. The observers use a structured interview form

for conducting the debrief session and for recording responses. These 

sessions are also be videotaped to allow verification of observer-recorded 

data. 

Data analysis Primary data analysis consists of comparing the accuracy and latency of the 

critical event detection across participants as a function of the number of 

sensor feeds, type of sensor feeds, and number of critical events. 

Secondary data analysis consists of performing qualitative analysis of the 

process data recorded during the event and post-event debriefing. This 

analysis is conducted to provide further explanation of results observed in the 

primary analysis. 

B-13 
SCALED-WORLD EVENT 6 SPECIFICATION 

Title Determine battle damage assessment

General 

description 

Friendly forces have conducted an air assault in A Company’s AO. Aircraft 

will continue to be in a position to conduct additional strikes based on battle 

damage assessment (BDA) completed by A company. 

This event is for the UA command group (with two members – the 

commander and the ISR Coordinator – participating). 

This event is process oriented. The primary interest is in observing the 

strategy that the group employs in assessing the damage produced by the air 

strikes. Because the time to conduct BDA is limited, the group needs to 

develop a strategy for using its sensor assets as efficiently as possible. 

Participants begin the event with a terrain map showing their AO. They 

receive a message directing them to assess damage produced by air strikes at 

three locations in their AO. They are given a time limit to report BDA results 

based on how long the attack aircraft will be in position to initiate another 

attack. 

The event is conducted during daylight conditions. Skies are clear with no 

obstacles to visibility related to weather. 

The event concludes with the BDA report. 

Tasks ▪ Decide what to look with. 

▪ Fuse information on screens. 

▪ Review information. 

▪ Interpret information. 

▪ Evaluate information. 

▪ Verify information. 

Independent 

variables 

▪ Time attack aircraft will be in position to initiate a second attack (15 

or 30 minutes). 

Dependent 

variable 

▪ Accuracy of BDA. 

▪ Time to conduct BDA. 

Process data ▪ Observations/data of participant performance/actions during 

completion of the event, collected automatically. 

▪ Participant responses/reactions, collected during post-event 

debriefing. 

B-14 
Measurement 

plan 

The two dependent variables (accuracy of BDA and time to conduct BDA) 

are measured by observation. Two observers monitor each event and record 

start-time and end-time. The observers also record the accuracy of the 

participants’ assessment as either correct or incorrect. In addition, the scaledworld environment is constructed to collect this data to the extent possible. 

In addition to these measures, the complete event session is videotaped in 

order to capture process data which, as stated above, are the date of primary 

interest for this event. The observers also record process data as the event 

proceeds. They record the steps/actions the participants used in deploying 

sensors, in processing information received from the sensors, data received 

from the sensors, decisions made, interactions between participants, and other 

process information as appropriate. The observers employ an observation 

checklist; however, it will not be used restrictively. The tapes of the session 

along with the data collected by the scaled-world environment are used to 

verify/supplement observer recorded process data. 

Following the event, the participants are debriefed by the observers in a 

structured interview setting. The observers use a structured interview form

for conducting the debrief session and for recording responses. These 

sessions are also videotaped to allow verification of observer-recorded data. 

Data analysis Primary data analysis consists of qualitative analysis of the process data 

recorded during the event and post-event debriefing. This analysis attempts to 

identify factors in the participants performance related to effective and 

ineffective results (as indicated by successful location of the biological 

weapon in a timely manner). The analysis also seeks to identify 

factors/variables which would have facilitated performance by examining 

results obtained from the participant debriefing session. 

B-15 
Appendix C 

Scaled-World Environment 

User Instructions 

C-1 
Scaled-World Environment 

User Instructions 

I. Loading A Scenario 

1. After deciding which scenario you will execute, click the left mouse button on the 

Scenario menu and move the mouse over the selected scenario. As shown in Figure C-1. 

Next select the desired version of the scenario from the submenu, as shown in Figure C2. 

Figure C-1. Scenario selection dropdown menu. 

Figure C-2. Scenario version dropdown menu. 

 2. When the scenario is loaded, a description of the scenario situation is displayed. This 

 description provides details of the current situation as well as a list of the sensors you have 

 available. After reading the situation, you may close the window by clicking the Close 

 button. At any time during the scenario, you can view the details again by clicking on 

 Situation Details in the lower left hand corner of the screen. See Figure C-3 

C-2 
Figure C-3. Situation details button. 

II. Screen Layout – When the scenario loads, your screen will appear similar to the one shown 

in 

Figure C-4. The screen includes four main panels: 

Figure C-4. Initial scaled-world tool screen layout. 

1. Operations Panel – on the left side of the screen; changes depending upon which panel is 

selected (see below). The operations panel is used to interact with and obtain 

information from the map, deploy sensors, and select sensors to view. 

2. Panel Select Buttons – across the top of the screen; control the current operation panel. 

3. Map Area – the map area displays the current map view, along with any sensors that 

might be within the displayed region. 

4. Sensor Output Area – across the bottom of the screen; displays text messages sent by 

deployed sensors. 

C-3 
III. Navigating the Map 

1. The navigation panel is the default panel displayed in the Operations Panel. If you have 

selected the New Sensors or Current Sensors panel, you can return to the navigation 

panel by selecting the Map Navigation button. 

2. Map navigation consists of two functions: Pan and Zoom. 

• The pan buttons are the directional arrows in the upper part of the Navigation panel. 

To move the map, select the appropriate arrow. The amount of movement depends 

on the current area of the displayed map. In the center of the arrows is a square 

block. Clicking this block will return the center of the map to the center of the 

screen. 

• The zoom buttons are the magnifying glass images with the plus (+) and minus (-)

signs. When clicked, the plus image zooms in (displays a smaller area), and the 

minus image zooms out (displays a larger area). 

IV. Deploying Sensors 

Sensors are deployed using the New Sensors panel. Deployment involves two functions: 

selecting the type of sensor you want to deploy; and laying down (i.e., drawing) the sensor's

route of travel. However, before describing these functions, you need to know some basic 

sensor information. 

1. Basic Sensor Platform Information 

There are three main types of sensor platforms: Aerial, Ground, and Stationary. Each 

platform is capable of carrying a variety of sensor types. All sensor identification codes 

start with “U” indicating that they are unmanned. 

• Aerial – air based platforms for which you can set a speed and altitude. These 

sensors are identified by the “AV” in the sensor code (i.e., UAV). 

• Ground – land based platforms for which you can set a speed. These sensors are 

identified by the “GV” in the sensor code (i.e., UGV). 

• Stationary – land based platforms that have no movement. These sensors are 

identified by the “GS” in the sensor code (i.e., UGS). 

2. Basic Sensor Types 

There are four classes of sensors: visual, audio, seismic, and biological agent. In 

general, aerial platforms will carry visual sensors; land based platforms will carry audio, 

seismic, and biological agent. There is also a hand-held biological agent sensor. 

• Visual – sensors that display a visual image. There are three types of visual sensors: 

video (displays a basic daylight image), infrared (displays image filtered through 

infrared filters), and thermal (displays visual image of heat emissions given off by 

objects). 

C-4 
• Audio – sensors that detect and record sounds. 

• Seismic – sensors that detect ground vibrations such as those produced by tracked 

vehicles, trucks, etc. 

• Biological agent – sensors that detect the presence of biological agents such as 

anthrax. 

3. Selecting a Sensor for Deployment 

• When you select the NewSensors panel, it will display a list of the sensor types 

(platform and sensor) that are available for the scenario. Click on the sensor you 

wish to deploy. 

• When you select a UAV or UGV type sensor, a sensor properties box shown in 

Figure C-5, lets you set the speed and (for UAVs) altitude of the sensor. After setting 

the sensor parameters, Click the Done button. You can now lay down the route (see 

below). 

• If you select a stationary sensor, you will not see a sensor control box. Stationary 

sensors do not move and, therefore, all you need to do is place them. 

Figure C-5. Sensor properties control. 

4. Laying down the sensor's route of travel. 

• If you have selected a UAV or UGV for deployment, you must specify its route of 

travel after you have specified its properties. You will do this on the map itself by 

specifying a starting point, interim waypoints, and a final loitering pattern. 

• To select a starting point, move the mouse pointer to the desired location on the map 

and Left Click. 

• Move the mouse pointer to the first desired waypoint and Left Click. 

• Repeat step 3 for each desired waypoint. 

• You can now specify a circular loitering pattern (i.e., a pattern the sensor will 

C-5 
continue to move in for the remainder of the scenario or until you recall it.) Move the 

mouse pointer to the location of the center of the loitering pattern and Right Click. 

• Move the mouse pointer and a circular track will appear on the map as shown in 

Figure C-6. When you have laid down the desired loitering pattern, Left Click. A 

green square will appear on the map with an appropriate sensor ID. 

• If you want to cancel a sensor before you finish its deployment, just Double Click

the left mouse button in the Map Area. 

Loitering pattern

Figure C-6. Loitering sensor. 

V. Viewing Sensor Output 

When you select the Current Sensors panel, you will be able to access all of the sensors 

you have deployed in addition to any other sensor inputs you may have access to. For 

example, you may be able to access the feed from a satellite controlled by a Unit of 

Employment (UE). Each of the sensor types has its own output. In addition, a sensor may 

display a text message in the Sensor Output panel. You can scroll through all of the 

messages that have been sent by the various sensors during the course of the scenario. 

There are three types of sensor output: audible, visual, and textual. Different scenarios 

require the use of different types of sensors. Each is available in the appropriate scenario. 

1. Seismic sensors will alert the user of activity via a text message in the sensor output 

area. See Figure C-7. Sensor output details can also be seen by selecting the sensor 

from the current sensors panel. The Bring In button can be used to return the sensor. 

C-6 

Figure C-7. Audible sensor output. 
 

2. Audible sensors will alert the user of activity in the sensor output area via a text message 

in the sensor output area. To listen to the sound recorded, bring up the current sensors 

panel and click the Audible Sensor. Then click the Play button. The Bring In button 

can be used to return the sensor. See Figure C-8. 

Figure C-8. Audible sensor controls. 

3. Visual sensors provide visual images that are only accessible through the current sensor 

panel. See Figure C-9. To view the images, click on the sensor in the current sensor 

panel. While the sensor output window is open, the image will update based on the 

position of the sensor in the map. The Bring In button can be used to return the sensor. 

 

Figure C-9. Visual sensor image. 

C-7 
4. Biological agent sensors will alert the user to agents detected via a text message in the 

sensor output area. You can select the sensor from the available sensors panel to 

view specific details. The Bring In button can be used to return the sensor. 

C-8 
Appendix D 

Concept-Development Session Specifications 

D-1 
CONCEPT-DEVELOPMENT SESSION 1 SPECIFICATION 

Title What do you want to know, how do you want to learn it?

General 

description 

This concept-development session provides an opportunity to explore 

elements of the “SEE” function in relation to a civilian cultural situation and 

allow for a wide range of responses from the participants. This conceptdevelopment session is for participants playing the role of the UA 

Reconnaissance Squadron Commander. 

The participants are presented with a tactical scenario, which includes a 

detailed written account of civilian activities in a town known to harbor threat 

personnel on existing blacklists. They are also able to view a series of 

photographs including groups of armed persons congregated near the town’s 

central hall. They have access to maps along with graphic displays showing 

their unit task organization and additional assets available. They are told that 

the unit’s task in this situation is to determine the nature of the threat in the 

town and make a report to the Battalion Commander. 

Data collection The structured interview, conducted by moderator(s), elicits responses aimed 

at shaping concepts and tactics, techniques, and procedures (TTP) related to 

this situation, including items such as determining what types of information 

they needed, how that information could best be presented, what steps they 

would take to accomplish the task, and what, if any, additional assets might 

have enhanced their performance. 

The moderators collect additional process data as they observe the participant 

completing the concept-development session. They record actions taken by 

the participant such as what information was reviewed and for how long, 

whether or not the mapping tool was used, etc., that might be helpful in 

conducting the interview or in interpreting the participants responses to the 

interview. 

Data analysis Data collected during the structured interview(s) will be analyzed by a panel 

familiar with Future Force UA concepts developed thus far as presented in the 

UA Operational and Organizational (O&O) Plan and the operational 

requirements document (ORD) for Future Combat Systems (FCS). 

Qualitative analysis procedures and techniques will be used and, as stated 

previously, the results will be used to modify the concept-development 

session for additional presentations. At some point determined by the panel, 

the concept-development session will no longer be administered and the 

concepts, TTP, lessons learned, etc., from all administrations of the conceptdevelopment session will be documented. 

D-2 
Structured 

interview

questions 

1. The unit’s task in this concept-development session was to determine the 

nature of the threat and report to the Battalion Commander. What steps 

would you take to complete this task? (Follow-up and clarify as 

necessary). 

2. What additional assets/capabilities, if any, would you liked to have had to 

complete this task, and how would you have employed those assets?

3. Is there information that you did not have that would have helped you 

complete this task? What information? How could it best be presented?

4. Is this an appropriate task for the unit indicated? Should it be a higher 

level task? Lower level? 

5. Follow-up questions based upon observations of the moderators. 

D-3 
CONCEPT-DEVELOPMENT SESSION 2 SPECIFICATION 

Title Design intuitive summary graphics for the Common Operational Picture 

(COP)

General 

description 

This concept-development session provides an opportunity for participants to 

design and create sample intuitive summary graphics (ISGs) for use on future 

planning maps and the COP. It focuses on ISGs for use at Bde level. 

Subsequent iterations may be conducted to design ISGs for Bn and 

Company/Troop level. 

An ISG is a symbol or icon displayed as part of a command and control 

system. They are simple to understand and informative, and provide the 

viewer with enough information to quickly grasp the nature of the event, unit, 

activity, terrain feature or effect being depicted. Examples of some current 

graphics that could be considered as ISGs include: military unit symbols and 

graphic control measures, international traffic/safety symbols, and the closed 

envelope representing email on many computers. In the near future it will be 

possible to clearly display multi-color, composite symbols using computers. 

The creation of a set of well-defined, easy to understand ISGs is required to 

ensure the military makes the best use of available computer capabilities and 

provides its personnel with one-look-to-understand graphics. 

This session includes two methods for the design/creation of intuitive 

summary graphics. One consists of the participant being presented with a 

series of spot reports in text. The participant is asked to design a graphic that 

represents the information in the reports. In the second method the 

participant is presented with a series of photographs and/or drawings of 

military activities and asked to design a graphic that represents the activities. 

In both methods the participant has a number of tools available to help with 

the design/creation of the graphics. These include but are not limited to 

drawing tools such as Microsoft PowerPoint®, a set of graphics to start with 

(such as boxes, lines, vehicles, personnel), a white board and markers, and 

current military symbology manuals. Additionally, there is graphic designer 

on hand to assist in developing the proposed ISGs in PowerPoint or a similar 

drawing application. This will support the recording and saving of the ISGs 

for future use. 

Data collection Data collected for this concept-development session consists primarily of the 

ISGs produced by the participants. These are collected either electronically 

or by drawing or photographing the white board used by participants. All 

ISGs produced are digitized for storage and retrieval. In addition, participants 

are observed as they complete the concept-development session and 

participate in a structured interview conducted by the moderator(s). The 

interview will primarily elicit responses to determine the decision-making 

process participants use to design ISGs. 

D-4 
Data analysis Data from this session will consist of graphic images. These will be 

presented to a panel familiar with Future Force concepts who will select those 

that clearly represent their underlying concept. 

Structured 

interview

questions 

1. Your task in this concept-development session was to develop or design 

intuitive summary graphics that reflected the situation presented either in 

text or graphically. Can you describe the thought process you went 

through for accomplishing this?

2. Was the description of the situation, either in text or graphics, complete 

enough for you to do your task? Explain. 

3. Follow-up questions. 

Note To use this concept-development session as a shell for developing other ISGs, 

it was developed so that the situation as presented in text or graphics can be 

easily changed. This allows a virtually unlimited range of situations to be 

developed and presented to participants resulting in the production of an 

unlimited number of ISGs. 

D-5 
CONCEPT-DEVELOPMENT SESSION 3 SPECIFICATION 

Title What information should a COP include? (Develop a COP)

General 

description 

This concept-development session provides an opportunity to explore 

elements of the “SEE” function related to the development of a brigade COP. 

It is designed for Future Force designers and developers?

In this concept-development session the participant(s) are shown a 

representation of a COP at brigade level that consists of multiple information 

layers with intuitive summary graphics representing operations graphics, the 

threat situation, terrain effects, etc. The participants are provided with a brief 

tactical scenario and associated operations graphics on a two dimensional 

map. They are then shown various other graphical overlays (threat, fires, 

obstacles, etc.), intuitive summary graphics, reports, messages, and sensor 

feeds related to the tactical situation and asked which of the additional pieces 

of information presented should be used to create the COP. They are also 

asked to recommend how the information should be displayed and what 

control the user or viewer of the COP should have in selecting what 

information is displayed. 

The COP represents a current tactical situation for a brigade conducting 

offensive operations in a major theater war (MTW) against a threat 

representing the new Operational Environment. The participant(s) are asked 

to evaluate the usefulness of each layer individually and in multiple 

combinations. The participant(s) are encouraged to request different 

combinations of layers during the concept-development session. The task is 

to provide suggestions for improving the clarity of the COP. 

This concept-development session is highly interactive between the 

participant and the moderator(s). This is necessitated by two factors: the 

moderator displays various preplanned information to the participant and asks 

for reactions; and the participant may wish to display unplanned information, 

but will probably not be familiar with the concept-development session 

software in order to create the display himself. Thus, during the conduct of 

the concept-development session, the moderator not only displays 

information to the participant, he/she is also recording the participants’ 

observations, reactions, and inputs on an on-going basis. The moderator uses 

a structured data-collection form to capture data. 

Data collection The primary data collected from this concept-development session are the 

“finished COPs,” i.e., the COPs that the participant determines as the best 

representation of the situation at the brigade level. The COPs are captured 

graphically with sufficient annotations to provide clarity for any subsequent 

analyses. Additional data are obtained from the interactive data collection 

that takes place as the concept-development session proceeds and follow-up 

interviews. 

D-6 
Data analysis The completed COPs, along with data collected during the conduct of the 

concept-development session and during subsequent interviews, are analyzed 

by a panel familiar with the Future Force UA concepts. Qualitative analysis 

procedures and techniques are used and, as stated previously, the results are 

used to modify the concept-development session for additional presentations. 

At some point determined by the panel, the concept-development session will 

no longer be administered and the concepts and recommendations for 

presenting the COP will be documented. 

Structured 

interview

questions 

1. Your task in this concept-development session was to develop a brigade 

level COP for a Future Force operation. What was your rationale for 

including the information you did? How well do you think the COP you 

produced will work? 

2. What, if any, important information is missing from the COP?

3. Who should be allowed to make changes to the COP? Explain. 

4. Is there a single COP that will serve all echelons? Explain. 

5. Follow-up questions based upon observations of the moderators. 

D-7 
CONCEPT-DEVELOPMENT SESSION 4 SPECIFICATION 

Title What is the process for updating COPs?

General 

description 

This concept-development session provides an opportunity to exercise 

portions of the “SEE” function in relation to the updating of the COP and 

allow for a wide range of responses from the participant(s). This conceptdevelopment session explores the preferences, requirements, and ideas of the 

participant(s) centered on what they see as the process for adding, deleting or 

modifying information appearing on the COP. 

The participant(s) are provided with a graphical representation of a current 

brigade COP (e.g., the product from concept-development session #3) and 

given time to review and understand the information on the COP. The 

participants are provided with a series of tactical reports and messages from

various units/personnel and are asked to decide which ones should be added 

to the brigade COP and whom they think should control the updates to the 

brigade COP. The participant(s) are then asked to describe what they see as a 

viable process for updating the brigade, battalion, or company COPs during 

combat operations. 

Data collection Participants are provided with a questionnaire which they fill out as they 

complete the concept-development session. The questionnaire prompts the 

participant to decide, for each of the reports and messages, which should be 

added to the COP and by whom. Participants are also observed by 

moderator(s) as they complete the concept-development session, after which 

they will participate in a structured interview. The interview elicits responses 

aimed at shaping concepts related to updating the COP. 

Data analysis Data collected from the participant-completed questionnaires, the 

observations and during the structured interview(s) is analyzed by a panel 

familiar with Future Force UA concepts developed thus far as presented in the 

UA O&O and the ORD for FCS. Qualitative analysis procedures and 

techniques are used and, as stated previously, the results are used to modify 

the concept-development session for additional presentations. At some point 

determined by the panel, the concept-development session will no longer be 

administered and the concepts, lessons learned, etc. from all administrations 

of the concept-development session will be documented. 

D-8 
Structured 

interview

questions 

1. Your task in this concept-development session was to determine what 

information should be added to a brigade COP and by whom. In general, 

are there some rules or procedures you can think of for deciding what 

information gets added? Explain. 

2. In general, are there some rules for deciding who can modify the COP? 

Explain. 

3. Do these rules and procedures apply to COPs at other echelons? Explain. 

4. Follow-up questions based upon observations of the moderators. 

D-9 
CONCEPT-DEVELOPMENT SESSION 5 SPECIFICATION 

Title Use Commander’s Critical Information Requirements (CCIR) to 

determine sensor requirements

General 

description 

This concept-development session provides an opportunity to explore 

elements of the “SEE” function related to the development of Future Force 

sensor requirements and usage. It allows for a wide range of responses from

the participants. The concept-development session is developed for 

participants playing the role of the UA Battalion Commander. 

The participants are provided with a brief tactical scenario at battalion level, 

associated operations graphics on a two dimensional map, a set of CCIR 

pertaining to the tactical situation. 

Data collection After participants work through the scenario, they will participate in a 

structured interview conducted by moderator(s). The interview elicits 

responses aimed at shaping concepts and TTP related to this situation 

including items such as determining what types of sensors they needed and 

how they would deploy and use those sensors. Participants are also 

encouraged to identify other (especially non-existent) sensor capabilities that 

might help answer the CCIR. The moderators collect additional process data 

as they observe the participant completing the concept-development session. 

They record actions taken by the participant such as what information was 

reviewed and for how long, etc. that might be helpful in conducting the 

interview or in interpreting the participant’s responses to the interview. 

Data analysis Data collected during the concept-development session and during the 

structured interview(s) are analyzed by a panel familiar with Future Force UA 

concepts developed thus far as presented in the UA O&O and the ORD for 

FCS. Qualitative analysis procedures and techniques are used and, as stated 

previously, the results are used to modify the concept-development session 

for additional presentations. At some point determined by the panel, the 

concept-development session will no longer be administered and the concepts, 

TTP, lessons learned, etc. from all administrations of the conceptdevelopment session will be documented. 

Structured 

interview

questions 

1. Your task in this concept-development session was to examine the tactical 

scenario and CCIRs and to decide what types of sensors would be 

important or useful in responding to the CCIRs. 

2. What additional sensor capabilities not included in the list would you 

want to have? How would you use them?

3. Does the idea of tying sensor capabilities to CCIRs make sense? Explain. 

4. Follow-up questions based upon observations of the moderators. 

D-10 
CONCEPT-DEVELOPMENT SESSION 6 SPECIFICATION 

Title Assess an urban area, what is the right sensor to human mix for ISR?

General 

description 

This concept-development session provides an opportunity to explore 

elements of the “SEE” function in relation to the assessment of an Urban 

Area. The participants will be playing the role of the UA Combat Aviation 

Brigade (CAB) RSTA Company commander. 

Participants are presented with a scenario that includes the requirement for 

the unit to conduct the initial assessment (ISR) activities of a town known to 

harbor threat personnel and equipment. Participants have access to maps and 

a series of photographs and sketches of the town. Additionally, they are 

provided with graphic displays showing their available assets. They are told 

that the unit’s task in this situation is to conduct an assessment of the town 

and make a report to the Battalion Commander. The participants are asked to 

develop a concept for employing their assets. This session focuses on the 

optimal mix of robotic sensors to human reconnaissance assets. 

Data collection After participants work through the scenario, they participate in a structured 

interview conducted by moderator(s). The interview elicits responses aimed 

at shaping concepts and TTP related to this situation including items such as 

determining what types of information they needed, how that information 

could best be presented, what steps they would take to accomplish the task, 

and what, if any, additional assets might have enhanced their performance. 

The moderators collect additional process data as they observe the participant 

completing the concept-development session. They record actions taken by 

the participant that might be helpful in conducting the interview or in 

interpreting the participant’s responses to the interview. 

Data analysis Data collected during the structured interview(s) are analyzed by a panel 

familiar with Future Force UA concepts developed thus far as presented in the 

UA O&O and the ORD for FCS. Qualitative analysis procedures and 

techniques are used and, as stated previously, the results are used to modify 

the concept-development session for additional presentations. At some point 

determined by the panel, the session will no longer be administered and the 

concepts, TTP, lessons learned, etc. from all administrations of the session are 

documented. 

D-11 
Structured 

interview

questions 

1. The unit’s task in this concept-development session was to conduct an 

assessment of the town and make a report to the Battalion Commander. 

What steps would you take to complete this task? (Follow-up and clarify 

as necessary.) 

2. You were asked to develop a concept for employing your assets – both 

human and robotic. Describe and explain your concept for employing 

assets. 

3. What do you think is the best mix of human and robotic assets for 

completing this task? 

4. Follow-up questions based upon observations of the moderators. 

D-12 
CONCEPT-DEVELOPMENT SESSION 7 SPECIFICATION 

Title Which reconnaissance asset(s) are needed to conduct reconnaissance in 

and around caves and cave-like structures?

General 

description 

This concept-development session provides an opportunity to exercise 

portions of the “SEE” function in relation to a tactical situation that requires 

the reconnaissance of a cave complex. This session allows for a wide range 

of responses from the participants by exploring the anticipated requirements 

for conducting reconnaissance of caves and cave-like structures (sewers) 

during combat operations. It is for participants playing the role of the UA 

Battalion Commander. 

The participants are provided with a brief tactical scenario that includes the 

requirement for their battalion to conduct reconnaissance of suspected threat 

locations in a mountainous area that includes a series of small and large 

caves. They are also provided with photos of the exterior and interior of the 

cave complex. They are then asked to identify and prioritize the sensor 

capabilities they expect will provide the best reconnaissance results. The 

outcome of this concept-development session is a list of desired sensor 

capabilities that the participants believe are best suited to use in caves and 

cave-like structures. 

Data collection After participants work through the scenario, they will participate in a 

structured interview conducted by moderator(s). The interview will elicit 

responses aimed at shaping concepts and TTP related to this situation 

including items such as determining what types of information they needed, 

how that information could best be obtained, and what steps they would take 

to accomplish the task. 

Data analysis Data collected during the structured interview(s) are analyzed by a panel 

familiar with Future Force UA concepts developed thus far as presented in the 

UA O&O and the ORD for FCS. Qualitative analysis procedures and 

techniques are used and, as stated previously, the results are used to modify 

the session for additional presentations. At some point determined by the 

panel, the concept-development session will no longer be administered and 

the concepts, TTP, lessons learned, etc. from all administrations of the 

concept-development session will be documented. 

D-13 
Structured 

interview

questions 

1. Your task in this concept-development session was to develop a list of 

desired sensor capabilities that you think are best suited to use in caves 

and cave-like structures. What information provided was important in 

helping you decide the sensor capabilities required?

2. Was there additional information that you would have liked to have had? 

How would it have assisted you in making your decisions?

3. What are the most important sensor capabilities you would need for the 

situation described?

4. Given that you had all of the sensor capabilities you have listed, describe 

a typical scenario for how you would deploy and use them in this 

situation. 

5. Follow-up questions. 

D-14 
CONCEPT-DEVELOPMENT SESSION 8 SPECIFICATION 

Title Fuse information from multiple sources

General 

description 

This concept-development session provides an opportunity to explore 

elements of the “SEE” function. The session is for participants playing the 

role of a UA combat assessment (CA) Battalion Commander. It will explore 

the cognitive processes associated with the fusion of information from various 

sources and allows for a wide range of responses from the participants. 

The participants are presented with 2-3 sets of information related to a tactical 

operation via a computer and in a variety of ways to include text, video, 

audio, and graphics. The information consists of tactical reports from recent 

threat activity that is relevant to the participant’s unit. The information is 

unfiltered and in some cases redundant or overlapping. In other instances, the 

information is incomplete when considered alone but relevant when fused 

with related information. The participants are allowed to review the situation 

and all supporting materials for as long as they want, after which they develop 

a situation report or verbal summary and ISG for each information set. 

Data collection At the completion of the concept-development session, participants will 

participate in a structured interview conducted by moderator(s). The 

interview will elicit responses aimed at identifying the cognitive processes 

involved in fusing information from a variety of sources into a commonly 

understood tactical situation. 

Data analysis Data collected during the structured interview(s) are analyzed by a panel 

familiar with Future Force UA concepts developed thus far as presented in the 

UA O&O and the ORD for FCS. Qualitative analysis procedures and 

techniques are used and, as stated previously, the results are used to modify 

the concept-development session for additional presentations. At some point 

determined by the panel, the concept-development session will no longer be 

administered and the concepts, TTP, lessons learned, etc. from all 

administrations of the concept-development session will be documented. 

D-15 
Structured 

interview

questions 

1. Your task in this concept-development session was to develop a picture of 

the tactical situation based on information from a variety of sources. Can 

you describe how you went about doing this (that is, can you describe the 

steps you took mentally in fusing the information from the various 

sources)?

2. Did the ways in which the information was presented to you help you 

with the fusion process? Did they hinder you? Did they make a 

difference one way or the other?

3. Are there different ways you would prefer to have the information 

presented to you? Explain. 

4. What information is critical to the fusion process?

5. Is there information that is routinely presented that is not important to the 

fusion process? Explain. 

6. Are there automated tools that you can think of that would assist you with 

the fusion process? Explain. 

7. Follow-up questions. 

D-16 
CONCEPT-DEVELOPMENT SESSION 9 SPECIFICATION 

Title Action – reaction – counteraction upon entering an immature AO

General 

description 

This concept-development session provides an opportunity to explore 

elements of the “SEE” function in relation to the use of organic unit sensors 

for terrain reconnaissance as an initial task upon entering an immature AO. 

The session is for participants playing the role of the UA CA Battalion 

Commander. 

One premise for FCS-equipped units is that they will be required to conduct 

terrain reconnaissance using organic sensors as one of their initial tasks upon 

entering an AO. The goal of the reconnaissance is to update data on the 

terrain in the AO since many maps and terrain databases will be outdated or 

incomplete. The question this raises is: if it is part of our doctrine to conduct 

such reconnaissance upon entering an AO, what will the threat’s reaction be 

and what should our counteraction be? For instance, if the threat knows we 

always do a sensor-based terrain reconnaissance at the outset of operations, 

how will they use this information? Will they sit tight and monitor our sensor 

use patterns and identify our tactics and procedures? If so, how will we 

counter this and maintain the advantage provided by our sensor capabilities? 

The participant(s) are presented with a short tactical scenario that has their 

unit entering an immature AO as the lead unit of a UA brigade. They are also 

provided with a task organization and system capabilities matrix for the 

battalion. The battalion is tasked with securing an entry point and conducting 

a detailed terrain reconnaissance of the brigade AO using its organic sensors 

and selected brigade sensors attached to the battalion. The scenario includes 

information concerning the threat’s composition, disposition and recent 

activities. The threat consists of a lightly equipped but sophisticated thinking 

enemy who uses its forces to strike at key government and military targets 

only. The threat also conducts extensive propaganda and recruiting activities, 

both of which have been successful. 

The task of the participants in this concept-development session is to do some

informed speculating about actions our forces and threat forces are likely to 

take in the described situation. They will be attempting to answer questions 

such as: “What actions should we take upon entry into the AO? If we do 

this, what do you think the threat will do? If the threat does this, what should 

our response be?”

D-17 
Data collection Participant responses will be captured by the moderator(s) throughout the 

concept-development session, moderators will record their responses/actions 

as a military subject matter expert (SME) leads them through the thought 

processes involved with completing it. In addition, they participate in a 

structured interview conducted by moderator(s) after they have completed the 

concept-development session. The interview elicits responses aimed at 

clarifying and/or expanding the data collected as the concept-development 

session was completed. 

Data analysis Data collected during the completion of the concept-development session and 

from the structured interview(s) are analyzed by a panel familiar with Future 

Force UA concepts developed thus far as presented in the UA O&O and the 

ORD for FCS. Qualitative analysis procedures and techniques are used and, 

as stated previously, the results are used to modify the concept-development 

session for additional presentations. At some point determined by the panel, 

the concept-development session is no longer to be administered, and the 

concepts, TTP, lessons learned, etc., from all administrations of the conceptdevelopment session are documented. 

Structured 

interview

questions 

Go through the actions and responses taken by the participant and ask any 

follow-up questions needed to clarify or add to the data you recorded during 

the completion of the concept-development session. 

D-18 
CONCEPT-DEVELOPMENT SESSION 10 SPECIFICATION 

Title Deploy a UA battalion in an AO to conduct reconnaissance operations

General 

description 

This concept-development session provides an opportunity to explore 

elements of the “SEE” function in relation to TTP associated with the 

deployment of a UA battalion and allow for a wide range of responses from

the participants. The concept-development session is for participants playing 

the role of the UA CA Battalion Commander. 

The participants are presented with a tactical scenario which requires them to 

decide how they will employ the battalion over a very large AO to conduct 

reconnaissance operations in a small scale contingency (SSC). They have 

access to maps along with graphic displays showing the threat situation, the 

battalion’s assets and various other information. They are told that the unit’s 

task in this situation is to deploy rapidly throughout the AO in a manner that 

supports conducting an area reconnaissance. 

Participants are allowed to review the situation and all supporting materials 

for as long as they want, and, when ready, they are asked to place their units 

on the map in initial positions and to add any graphic control measures 

(GCMs) needed. They are also asked to prepare short ‘task and purpose’ 

statements for each unit to support their concept of the operation. 

Data collection Once participants complete the session, they participate in a structured 

interview conducted by moderator(s). The interview elicits responses aimed 

at shaping concepts and TTP related to this situation including items such as 

determining what types of information they needed, how that information 

could best be obtained, and what steps they would take to accomplish the 

task. 

Data analysis Data collected during the structured interview(s) are analyzed by a panel 

familiar with Future Force UA concepts developed thus far as presented in the 

UA O&O and the ORD for FCS. Qualitative analysis procedures and 

techniques are used and, as stated previously, the results are used to modify 

the concept-development session for additional presentations. At some point 

determined by the panel, the concept-development session is longer to be 

administered, and the concepts, TTP, lessons learned, etc., from all 

administrations of the concept-development session will be documented. 

D-19 
Structured 

interview

questions 

1. Your unit’s task in this concept-development session was to deploy 

rapidly throughout the AO in a manner that supports conducting an area 

reconnaissance. Describe your rationale for your deployment plan. 

2. How was your plan influenced by the mix of live and robotic 

reconnaissance capabilities you had access to? 

3. Was the information you were provided sufficient to allow you to create 

your reconnaissance plan? Explain. 

4. Do you think the reconnaissance plan you produced adequately covers the 

area involved? Explain. 

5. Follow-up questions. 

D-20 
Appendix E 

Administration Guide for 

Concept-Development Sessions for the Future Force 

E-1 
Administration Guide for 

Concept-Development Sessions for the Future Force 

Introduction 

The Army, as it incorporates new technology such as robotics, advanced sensors, and 

network-centric Command, Control, Communications, Computers, Intelligence, Surveillance and 

Reconnaissance (C4ISR) systems, is beginning a transformation process on a wide scale. A key 

to success of this transformation is a solid Army process for concept development - a means to 

generate, elaborate, refine, describe, test, and validate new Future Force concepts relating to 

doctrine, tactics, techniques, procedures, unit and team organization, job allocation, training, 

leader development, and other aspects of technology integration. One method for concept 

development is provided in a tool for presenting concept-development sessions (CDS). 

The notion of a CDS is that of a snapshot, or a series of snapshots, or a short presentation 

that places a decision-making, problem-solving, or evaluation requirement on the participant(s). 

A CDS, unlike a simulation, is not required to respond dynamically to participant input. The 

CDS can portray highly realistic situations and entities that are often very difficult to model well 

in a simulation, e.g., underground caves, car bombs, cultural factors, and situations arising from

support and stability operations. The CDSs comprise the presentation of a detailed, realistic 

situation during which a participant provides concepts and ideas related to the situation 

portrayed. This may then be followed by “what now?” or “what if?” queries to the 

participant(s). By running a CDS multiple times, modifying it based on the information 

collected from participants, and repeating this run-modify process, it will be possible to develop 

and fine-tune ideas and concepts which can then be used as the basis for additional testing or 

development using other tools that may be developed. 

Structure of CDSs 

There are currently 10 CDSs which have been developed using RoboHTML and 

PowerPoint. They are designed for execution on a system that includes dual monitors. (It is 

possible to present a CDS on a single-monitor configuration; however, its effectiveness will be 

seriously degraded.) 

In a typical CDS, the component developed in RoboHTML is presented on the left screen, 

and the component developed in PowerPoint is presented on the right. The RoboHTML 

component provides descriptive information, often text-based although some is presented 

graphically as well, that sets the context for the session and provides the background and other 

details the participant needs to start the “thought” process that is at the heart of the CDS. The 

PowerPoint component includes the graphics (maps and overlays, pictures, etc.) that support the 

CDS and provide the richness of detail that helps guide the participant’s thought process. 

As is probably obvious by now, it is the participants’ “thoughts” that are the heart of the 

CDS. The CDS is designed to capture these thoughts both during and after the execution of the 

CDS. This is accomplished in two ways. First, the complete CDS session should be videotaped 

(or at least audiotaped) so that it can be analyzed for relevant concepts and ideas that may have 

E-2 
been elicited. In addition, the observers/moderators should take notes during the execution of 

the CDS, and should interview the participant after completing the CDS. Each of the CDSs 

provided includes a data collection instrument to assist the observers/moderators in collecting 

data during the CDS. A list of the ten CDSs currently developed, along with the echelon for 

which it was developed and its focus, is provided in Table E-1. 

“Running” a CDS 

Running a CDS is a straightforward process. The participant(s) should be seated at the 

display system; the observers/moderators should be seated in a position where they can easily 

observe the actions being taken without being intrusive. There must be at least one 

observer/moderator although two are recommended. It is strongly recommended that at least one 

observer/moderator be a military subject matter expert with a good knowledge of Future Force 

and FCS capabilities. 

Prior to starting the session, copies of the data collection form for the CDS should be 

provided to each observer/moderator. Use of this guide will be discussed shortly. 

At the start of a session, the participant(s) should be briefed on the CDS process. This 

should include a statement emphasizing that the CDS is not an exercise to be solved, but rather, 

it is a description of a situation designed to start the participant(s) thinking about how the Army

might function in the future. The briefing should also emphasize that the data the CDS is 

designed to collect are the “thoughts” of the participant(s). For this reason, participant(s) are 

encouraged to “think out loud” as they go through the CDS. Some participant(s) will be more 

comfortable doing this than others, and those who prefer not to do it should not be forced to do 

so. The participant(s) should be encouraged to think “future” and should be told that there are no 

correct or incorrect answers. The purpose is to “see what they think.” Following the initial 

brief, the observer/moderator should start the CDS. 

The initial page or screen of each CDS provides general instructions for completing it as 

well as describing the specific task the participant(s) is completing. After reading this page, the 

participant(s) will start the CDS. As the session proceeds, the observer/moderator should record 

significant actions the participant(s) takes as well as what he or she is saying, questions he or she 

asks, etc. using the data collection form provided for each CDS. If appropriate, and this is a 

judgment call, the observer/moderator can direct the participant(s) to certain features or aspects 

of the CDS or can even ask direct questions to attempt to lead the participant(s) in a specific 

direction. You need to be careful if you do this however, since you don’t want to unduly 

influence the thought processes of the participant(s) or influence them to think the way others 

have responded or the way you think. A good rule of thumb to remember is “you already know 

what you think; the purpose of the CDS is to find out what they think.” 

Another important point to keep in mind while observing is to be patient. If you do ask a 

question or otherwise prompt the participant(s), give them plenty of time to think about their 

response. Avoid the temptation to ask a quick follow-up question if they don’t respond 

immediately. 

E-3 
E-4 

Also keep in mind that you do not need to ask every question immediately as it occurs to 

you. If a question comes to mind that you can ask after the participant(s) has completed the 

CDS, write it down on the observation form. Your purpose is to observe, not to unduly influence 

– to allow the participant(s) to fully explore their line of thought without interrupting in a way 

that takes them into a different line of thought. 

It is anticipated that the CDS will go through a progressive life cycle. Specifically, it will 

be administered several times with single participants. The ideas gleaned from those 

administrations might then be incorporated into the CDS, and it will be administered with more 

participants, possibly two or more at a time. As more and more data are obtained the CDS is 

modified and administered until it reaches a point where no new information is forthcoming. At 

this point, the CDS has probably told you everything it is going to as far as concept development 

is concerned, and it may well become the basis for some kind of training exercise or other Future 

Force development tool. 

Analyzing Data from a CDS 

The final part of CDS administration involves analyzing the data obtained from the 

participant(s). Since each administration of a CDS will probably be different from all other 

administrations of that CDS, it is difficult to have a specific set of items that you can record and 

enter into a database. It is much more likely that you will go back over the responses produced 

in an administration of a CDS and “content analyze” them for important or critical points. It is 

these points that would be entered into a database to allow you to go back later and look for 

commonalities across participants, critical ideas, etc. Just keep in mind that if you change or 

modify the CDS substantially, you will need to be careful about comparing responses from the 

two different versions. 
Table E-1. Current Concept-Development Sessions. 

# Title Echelon Focus 1 What do you want to know, how do you want to learn it? Troop Use of assets to identify na

gathering in a potentially h

2 Design Intuitive Summary Graphics for the COP. Brigade Identifying characteristics 3 What information should a COP include? Brigade Identifying characteristics 4 What should the process be for updating the COP? Brigade, Battalion, Company Identifying processes for m

updating of COPs at variou

5 Use CCIR to determine sensor requirements. Battalion Determining how CCIR co

of future sensors. 

6 Assess an Urban area – what is the right sensor to human mix for ISR. Company Identifying concept for mi

with humans during ISR o

7 

Which reconnaissance asset(s) are needed to conduct reconnaissance in and around caves and cave-like structures? 

Battalion Identifying sensor capabili

8 Fuse information from multiple sources for display on the COP. Battalion Identify cognitive approac

information. 

9 Action – reaction – counteraction upon entering an immature AO. Battalion Identify possible threat rea

robotic sensors. 

10 Deploy a CAB in an AO to conduct reconnaissance operations. Battalion TTP associated with deplo

battalion in a very large A

E-5 
Modifying a CDS and Developing a New CDS 

It is not difficult to modify a CDS, although it does require knowledge of RoboHTML and 

PowerPoint. If you simply want to change the information on a given page, you can edit the 

appropriate file from the list of files that were developed to make up the compiled CDS. More 

extensive modifications could involve making new files using RoboHTML and/or new 

PowerPoint slides. 

Developing a new CDS involves starting from scratch with designing the CDS as the first 

step. It is important to develop design specifications that include a description of the CDS 

scenario, the focus of the CDS, the concepts it is intended to explore, and so forth. Once the 

design is complete, development can proceed using RoboHTML and PowerPoint. 

E-5 

Keep in mind that if you do significantly modify a CDS or create a new one, you will need 

to develop appropriate data collection instruments. 

E-6