U.S. Army Research Institute 

for the Behavioral and Social Sciences 

Research Report 1856 

Development of the Reactive Planning Strategies 

Simulation (REPSS) 

William R. Sanders 

U.S. Army Research Institute 

 Christopher V. Fultz 

 Western Kentucky University 

 Consortium Research Fellows Program 

N. Kelly Sharp 

University of Louisville 

Consortium Research Fellows Program 

July 2006 

Approved for public release; distribution is unlimited 
U.S. Army Research Institute 

for the Behavioral and Social Sciences 

A Directorate of the Department of the Army 

Deputy Chief of Staff, G1 

Authorized and approved for distribution: 

 

 STEPHEN GOLDBERG MICHELLE SAMS 

 Acting Technical Director Acting Director 

 

Technical review by 

Richard E. Christ, U.S. Army Research Institute 

Billy L. Burnside, U.S. Army Research Institute 

NOTICES

DISTRIBUTION: Primary distribution of this Research Report has been made by ARI. 

Please address correspondence concerning distribution of reports to: U.S. Army 

Research Institute for the Behavioral and Social Sciences, Attn: DAPE-ARI-MS, 

2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926 

FINAL DISPOSITION: This Research Report may be destroyed when it is no longer 

needed. Please do not return it to the U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

NOTE: The findings in this Research Report are not to be construed as an official 

Department of the Army position, unless so designated by other authorized documents.
REPORT DOCUMENTATION PAGE

1. REPORT DATE (dd-mm-yy)

 July 2006 

2. REPORT TYPE

Final 

3. DATES COVERED (from. . . to) 

October 2004 to July 2005 

4. TITLE AND SUBTITLE 5a. CONTRACT OR GRANT NUMBER

Development of the Reactive Planning Strategies Simulation (REPSS) 

5b. PROGRAM ELEMENT NUMBER 

622785

5c. PROJECT NUMBER

A790

5d. TASK NUMBER

211

6. AUTHOR(S) 

William R. Sanders (U.S. Army Research Institute); Christopher V. 

Fultz (Western Kentucky University); and N. Kelly Sharp (University 

of Louisville) 

5e. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

U.S. Army Research Institute for the Behavioral and Social Sciences 

ATTN: DAPE-ARI-IK 

121 Morande Street 

Fort Knox, KY 40121-4141 

8. PERFORMING ORGANIZATION REPORT NUMBER

10. MONITOR ACRONYM 

ARI

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 

U.S. Army Research Institute for the Behavioral and Social Sciences 

2511 Jefferson Davis Highway

Arlington, VA 22202-3926 11. MONITOR REPORT NUMBER 

Research Report 1856 

12. DISTRIBUTION/AVAILABILITY STATEMENT 

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES 

Subject Matter POC: William R. Sanders 

14. ABSTRACT (Maximum 200 words): 

The present report describes the development of the Reactive Planning Strategies Simulation (REPSS), and the results of initial

experimentation comparing the performance of distributed and co-located groups. The REPSS presents a simplified 

collaborative planning task where a commander and three teams organize relief mission convoys to four towns as part of a 

stability and reconstruction operation. The research supports the Leader Adaptability Army Training Objective (ATO) which 

seeks to provide prototype computer-based methods and tools to rapidly train and sustain fundamental leadership and battle 

command skills required as increasingly complex command and control technologies and networks become operational. With 

regard to the planning process, the comparison of co-located and distributed groups showed clear differences in verbal and textmessaging communications. Automated measures of team planning synchronization and quantity of goods delivered provided 

some evidence that the co-located groups were better at achieving the goal of maximizing the delivery of required supplies 

consistently across the four towns. A majority of participants (93%) indicated that the REPSS planning exercise could be 

useful in command group training. 

15. SUBJECT TERMS 

Collaborative Planning Future Force Future Combat Systems 

Mission Command Command and Control 

 

16. REPORT 

Unclassified

17. ABSTRACT 

Unclassified

18. THIS PAGE 

Unclassified

19. LIMITATION OF 

ABSTRACT 

Unlimited

20. NUMBER 

OF PAGES 

69 

21. RESPONSIBLE PERSON

(Name and Telephone Number) 

Ellen Kinzer 

Technical Publication Specialist 

703-602-8047 NSN 7540-01-280-5500 Standard Form 298 

i 
 ii
Research Report 1856 

Development of the Reactive Planning Strategies 

Simulation (REPSS) 

William R. Sanders 

U.S. Army Research Institute 

 Christopher V. Fultz 

 Western Kentucky University 

 Consortium Research Fellows Program 

N. Kelly Sharp 

University of Louisville 

Consortium Research Fellows Program 

 

Armored Forces Research Unit 

Barbara A. Black, Chief 

U.S. Army Research Institute for the Behavioral and Social Sciences 

2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926 

July 2006 

Army Project Number Personnel Performance 

622785A790 and Training Technology 

Approved for public release; distribution is unlimited.

iii 
iv
DEVELOPMENT OF THE REACTIVE PLANNING STRATEGIES SIMULATION (REPSS) 

 

EXECUTIVE SUMMARY 

Research Requirement: 

The U.S. Army’s transformation to the Future Force will require the leveraging of digital 

communications capabilities to support distributed battle command. The goals for the present 

research were to develop a collaborative group planning simulation and supporting measures, 

and to conduct experiments investigating group planning performance. The report documents 

the development of the Reactive Planning Strategies Simulation (REPSS) collaborative planning 

simulation, and the results of initial experimentation comparing the performance of distributed 

and co-located groups. The research supports the Army Training Objective (ATO) titled Leader 

Adaptability initiated in 2006. The Leader Adaptability ATO seeks to provide prototype 

computer-based methods and tools to rapidly train and sustain fundamental leadership and battle 

command skills required to lead and perform adaptively as increasingly complex command and 

control technologies and networks become operational. 

Procedure: 

The development of the REPSS proceeded through three phases: a) design and develop a 

group collaborative planning exercise which presents an iterative series of plan - execute - and 

adjust decision making cycles, b) develop associated performance process and outcome

measures, and c) conduct an experiment to test and refine the REPSS and performance measures.

The REPSS was designed to incorporate group collaborative planning processes consistent with 

the U.S. Army’s mission command method for the command and control of forces. Successful 

mission command rests on the commander’s intent, subordinates’ initiative, resource allocation, 

synchronization of plans, and the ability to effectively respond to unforeseen circumstances, 

opportunities, and threats. A simplified stability and reconstruction operation served as the basis 

for designing the exercise, with a commander and three two-person teams (seven members total) 

collaborating to allocate resources in support of a humanitarian support effort. A situational 

awareness manipulation was incorporated into the exercise requiring the group to gather 

information and share it in order to make the correct decisions in supplying a specific town. 

Measures of individual skills, team skills, the collaborative planning process, and 

planning performance outcomes were developed to support research investigating the 

performance associated with collaborative planning success. A primary objective was to 

develop a single “goodness of performance” metric for planning solution success, which was 

achieved by consistently applying a dollar value to team assets involved in the planning task. An 

emphasis was placed on developing automated performance measures, which could also serve as 

automated feedback if REPSS is used in a training role. The prototype REPSS system was 

employed in an experiment comparing the performance of two co-located groups and two 

distributed groups consisting of officers and non-commissioned officers at Fort Knox. 

Findings: 

v
 

The REPSS was successful in stimulating collaborative group planning activities, and 

providing automated estimates of performance success. From observations and written 

responses to post-experiment surveys, the pilot test participants accepted the stability and 

reconstruction humanitarian relief scenario, and actively worked to achieve the best solution 

possible. The collaborative planning process measures developed for REPSS provided valuable 

estimates of group planning performance. Findings for the comparison of co-located and 

distributed group conditions clearly showed differences in verbal and text-messaging 

communications for the two conditions. One limitation with the process performance measures 

was that they require labor-intensive communications transcript coding. Automated 

collaborative planning outcome measures developed in the effort provided useful estimates of 

team planning synchronization, as well as the desired “goodness of performance” metric. Results 

from the co-located vs. distributed group comparison provide some evidence that the co-located 

groups were better at achieving the goal of maximizing the delivery of required humanitarian 

supply unit (HSU) consistently across all four towns. Most of the participants (93%) indicated 

that the REPSS planning exercise could be useful in command group training. 

 

Utilization and Dissemination of Findings: 

The experiment findings suggest that the REPSS is a viable research tool for exploring 

Army group collaborative planning. Changes were made to eliminate redundancy in the selfpaced training materials, and to incorporate a training self-assessment feature. Three additional 

situational awareness manipulations were incorporated into the exercise in order to better assess 

each group’s ability to respond to changes in the environment. The automated performance 

assessment features were enhanced to provide a consolidated report on percentage of required 

supplies provided, and planning synchronization that is available immediately after each weekly 

plan is submitted. The REPSS software was successfully hosted on laptop computers which 

allows the REPSS to be used as a mobile research tool. The rapid train-up, simple interface 

tools, and automated feedback features make REPSS a likely candidate to support collaborative 

group planning skills training. 

vi
DEVELOPMENT OF THE REACTIVE PLANNING STRATEGIES SIMULATION (REPSS) 

 

CONTENTS 

 

Page

Introduction................................................................................................................................... 1 

 Research Goals ....................................................................................................................... 1 

 Background............................................................................................................................. 1 

 Considerations Guiding the REPSS Design ........................................................................... 3 

 Report Organization................................................................................................................ 6 

 

Method .......................................................................................................................................... 6 

Exercise Overview.................................................................................................................. 6 

Participants.............................................................................................................................. 7 

Apparatus................................................................................................................................ 8 

Measures Development...........................................................................................................13 

Procedure ................................................................................................................................16 

Results...........................................................................................................................................17 

Individual Characteristics.......................................................................................................17 

Group Roles Knowledge.........................................................................................................18 

Planning Processes..................................................................................................................18 

Planning Outcomes.................................................................................................................20 

Discussion.....................................................................................................................................26 

Summary.................................................................................................................................26 

The REPSS Refinements ........................................................................................................27

Future REPSS Research..........................................................................................................27 

The REPSS Training Potential ...............................................................................................28 

References.....................................................................................................................................29 

Appendix A Acronyms............................................................................................................. A-1 

Appendix B REPSS Mini Lab Room Configuration ................................................................ B-1

Appendix C Data Entry Screens for REPSS Teams................................................................. C-1

Appendix D Weekly Newspaper .............................................................................................. D-1 

Appendix E Demographic Survey ............................................................................................ E-1 

Appendix F Group Roles Knowledge Survey .......................................................................... F-1 

Appendix G Workload Survey.................................................................................................. G-1 

Appendix H Formative Evaluation Survey............................................................................... H-1 

Appendix I REPSS Exercise General Instructions................................................................... I-1 

Appendix J REPSS Exercise Team Instructions (Supply Team) .............................................. J-1

vii
CONTENTS (continued) 

 

 Page

List of Tables 

Table 1 Selected Demographic Survey Results..............................................................................17 

Table 2 Shared Collaboration Matrix for Display of Current Situation Information 

 and Team Plans..................................................................................................................19 

Table 3 Rank Ordering of Group Characteristics, Planning Process, and Planning Outcome 

 Measures Relative to HSU Delivered...............................................................................24 

List of Figures 

Figure 1 Participants working in the co-located group condition and in the 

 distributed group condition............................................................................................. 8 

Figure 2 The REPSS map showing a pop-up information window for Town India, describing 

 the current threat, population size, weather, and citizen morale level............................10 

Figure 3 Messages window showing time message sent, sender, and text of message.................11 

Figure 4 Transportation team resource request tool showing two routes to each town, 

 order blocks for armored and regular five-ton trucks, cost of vehicles for each 

 route, and total cost.........................................................................................................11 

Figure 5 Bar chart comparison of the frequency of verbal statements made by the 

 commander and each of the three teams during planning for distributed and 

 co-located groups.............................................................................................................18 

Figure 6 Bar chart comparison of the frequency of text messages sent by the 

 commander and each of the three teams during planning for distributed and 

 co-located groups............................................................................................................19 

Figure 7 Bar chart comparison of the planning workload levels reported by the 

 commander and each of the three teams for distributed and co-located groups.............20

Figure 8 Percent of total required HSU delivered by each group..................................................21 

Figure 9 Percentage of required HSU delivered each week, for each group.................................21 

Figure 10 Comparison of percentage of required HSU delivered to towns by each group.............22 

Figure 11 Level of plan synchronization for each group and each week ........................................23 

Figure 12 Comparison of average performance success ratings and actual percentage 

 of HSU delivered for distributed and co-located groups................................................24 

viii
DEVELOPMENT OF THE REACTIVE PLANNING STRATEGIES SIMULATION (REPSS) 

Introduction 

Research Goals 

A defining feature of Army transformation will be the leveraging of digital 

communications to support command and control of forces. While a great deal of attention is 

typically devoted to providing the communications tools required to support command and 

control, there is an equally important requirement to investigate the human performance 

necessary to support successful collaborative planning in the mission command environment 

through controlled human performance research. The goals for the present research were to 

develop a collaborative group planning simulation and supporting measures, and to conduct 

experiments investigating group planning performance. The report documents the development 

of the Reactive Planning Strategies Simulation (REPSS) collaborative planning task 

environment, and the results of initial experimentation comparing the performance of distributed 

and co-located groups. 

 

The present research effort supports the Army Training Objective (ATO) titled Leader 

Adaptability initiated in 2006 which seeks to “provide prototype computer-based methods and 

tools to rapidly train and sustain fundamental leadership skills (critical thinking, interpersonal 

skills, and self-awareness) and the battle command skills (visualize, describe, and direct) 

required to lead and perform adaptively as increasingly complex command and control 

technologies and networks become operational” (U.S. Army Research Institute for the 

Behavioral and Social Sciences [ARI], 2006). The Leader Adaptability ATO presents a number 

of training issues which can be addressed through experimentation using the REPSS 

collaborative planning task environment and it's supporting measurement tools. 

Background 

 Mission command. In conducting command and control research it is important to 

capture key collaborative planning processes, and to present these in a realistic and 

representative experimental exercise. Successful mission command as described in Field 

Manual (FM) 1, The Army (Department of the Army [DA], 2005) rests on the ability of 

commanders to convey the intent and concept of operations, provide resources adequate to 

accomplish the mission, and empower subordinates to make decisions while synchronizing their 

operations (DA, 2005, 3-33). 

Simulation features overview. The Leader Adaptability ATO states that the high 

operational tempo, volatile mission demands, and serious resource constraints present challenges 

to leaders in the contemporary operational environment (COE), and the future network-enabled 

environments, which need to be integrated into the way we train our leaders. The REPSS 

exercise presents a multi-stage resource allocation task, requiring a commander and three 

functional teams to send weekly relief supply convoys to four separate towns for four weeks in a 

COE scenario. The REPSS incorporates a number of features which support the Leader 

Adaptability ATO research goals to include: 

1
• High operational tempo: Planning groups have only 40 minutes to create and 

execute their plan for the week one, and 20 minutes to adjust and execute a plan 

for each of the three following weeks. 

• Resource constraints: The planning groups are given inadequate funds to fully 

support the relief effort, so that the task is one of identifying an optimal set of 

tradeoffs between each of the teams to maximize the group result. 

• Future networked environments: REPSS has been configured as both a traditional 

co-located command group planning environment and as a future networkenabled environment to investigate performance trends and training needs. 

• Leader training: REPSS incorporates self-paced automated training embedded 

within the system to enhance consistency of training delivery, and reduce training 

overhead costs. An automated performance assessment capability has been 

designed into the REPSS which could support the use of REPSS in a training role, 

allowing for immediate feedback on performance early and throughout an 

exercise. 

The REPSS was designed by the authors to stimulate collaborative planning within a 

group composed of a commander and three interdependent functional teams (supply, 

transportation, and security). A key design consideration is that command group planning skills 

involve more than a one-shot effort to develop a static plan. Instead, the command group 

planning process should foster a shared understanding of competing demands, and support the 

rapid and flexible adjustment of plans during mission execution in response to feedback gathered 

from the environment. The single REPSS exercise presents a simplified stability and 

reconstruction operation collaborative planning task which requires effective group 

communications, and the ability to adapt to change. With regard to performance outcomes, the 

REPSS exercise was designed to provide an overall estimate of the “goodness” of the group 

planning solution in terms of the quantity of humanitarian relief supplies delivered. Additional 

performance measures provide quantitative estimates of the groups’ ability to synchronize the 

competing requirements of the supply, transportation, and security teams. The REPSS planning 

tasks were intentionally designed so that they do not require doctrinal knowledge, and can be 

accomplished using a few simple spread-sheet interface tools. Several experiments have been 

conducted with the REPSS. Following the first REPSS experiment (REPSS 1) a number of 

changes were made to the interface design, the planning task was expanded to include multiple 

routes to each town, and much of the participant training was changed to a self-paced automated 

format. The report presents the REPSS design and findings from the second experiment (REPSS 

2), which are more representative of the ongoing REPSS research than the design and results 

from the REPSS 1 experiment. For the convenience of the reader, a list of all acronyms is 

provided as Appendix A. 

Co-located and distributed command groups. A key issue facing the Future Force is that 

Army command groups must be capable of transitioning from the traditional face-to-face (colocated group) tactical operations center environment to Future Force collaborative planning 

across networked communication systems (distributed group). For the present research 

2
“distributed groups” will be defined as groups whose members work as interdependent teams, 

separated by some degree of physical space, whose interactions are mediated through electronic 

communications technology. Previous research efforts conducted by the ARI have suggested 

that significant differences may exist in the pattern of collaborative planning for command group 

members based on whether they are operating as a co-located group, or as a distributed group. 

Results from a series of four Future Combat System – Command and Control (FCS C2) 

experiments conducted by the Defense Advanced Research Projects Agency (DARPA) revealed 

that the co-located command group engaged in a fast-paced and flexible verbal exchange almost 

90% of the time during simulated engagements (Lickteig, Sanders, Durlach, Lussier, & 

Carnahan, 2003). In sharp contrast, during experimentation with distributed command group 

members it was noted that communication was less frequent, and took the form of more 

traditional sequential staff briefings (Holden, Smith, Conzelman, Smith, Lickteig, & Sanders, 

2005). 

Considerations Guiding the REPSS Design 

Conceptual framework. Research addressing command group wargaming skill 

requirements has provided a useful conceptual framework for the development of the command 

group exercise tasks and performance measures based on a detailed task analysis of U.S. Army

institutional wargaming (Cianciolo & Sanders, 2006). The work points out the need to consider 

the contribution of individual knowledge and skills Soldiers bring to the task situation, and the 

need to identify how well the members of a collaborating group know each other’s roles. The 

assessment of group planning outcomes should include quantitative estimates of the “goodness” 

of the group solution, and also indicators of group planning synchronization. Another factor to 

consider in estimating group planning outcome success should be the group’s shared 

visualization of the problem space, or shared situational awareness, which is thought to underlie 

successful group problem solving (Endsley, Holder, Leibrecht, Garland, Wampler, & Matthews, 

2000). 

 

Doctrinal framework. Army doctrine prescribes the use of Mission, Enemy, Terrain, 

Troops, Time, Civilians (METT-TC) Factors in delineating the information essential to 

command and control, for commanders at all levels. Doctrine states that “All planning, whether 

it be deliberate, crisis, or campaign, must consider the mission, the enemy, the terrain and 

weather, the troops and support available, the time available and the civil considerations when 

planning for an operation.” (Field Manual 3.0, Operations, 2002). The METT-TC factors served 

as a doctrinal framework for creating the REPSS exercise task environment. Specific tasks and 

functional relationships were incorporated into the REPSS system to exercise the METT-TC 

factors, making the REPSS exercise applicable across a broad range of current and future 

command and control (C2) support systems. A summary of how METT-TC factors outlined in 

FM 3-0 were incorporated into the REPSS design is provided below: 

 Mission – Commanders determine the mission through analysis of the tasks to be 

 accomplished; who is to do them; and when, where, and why the tasks are to be 

 done. The REPSS design features include: 

• Higher commander’s guidance to maximize food delivery across towns and 

weeks. The success of the group’s planning effort is assessed against this criteria. 

3
• Reciprocally interdependent tasks for supply, transportation, and security teams 

which compete for limited funds. 

• Requirement for the group to plan, execute, and adjust their plans in response to 

changing conditions, and available assets. 

Enemy – Includes current information about strength, location, activity, and 

capabilities. In stability operations and support operations, this includes adversaries, 

potentially hostile parties, and other threats to success. Threats may include regional 

instabilities, and misinformation. The REPSS design features include: 

• A dynamic asymmetric threat to convoy and town operations which decreases in 

response to successful food deliveries, and increases when shortages occur. 

• A requirement for group members to actively “reach out” to access available 

information sources, to piece together and synthesize information on enemy 

activities. 

Terrain – Terrain considerations include key terrain, obstacles to movement, weather, 

and manmade features. Planners need to consider the effects of terrain on ground 

maneuver, and Combat Service Support operations. The REPSS design features that 

address the METT-TC Terrain factor include: 

• A broad range of convoy route terrain conditions, from hills to valleys, with 

different road trafficability conditions, and potential obstacles (bridges). 

• Changing weather conditions which require planners to match convoy equipment 

decisions to weather-induced road conditions. 

Troops and support available – Assess the quantity, training level, and psychological 

state of friendly forces, which can include contractors. The REPSS design features 

include: 

• Incorporation of both professional and less experienced civilian security guards, 

so planners must consider both quantity and training level when hiring guards. 

• Incorporation of up-armored and regular trucks, and both hardened and regular 

distribution centers, so that planners must consider both quantity and security 

level when renting trucks and distribution centers. 

Time available – Assess the time available for planning, preparing, and executing the 

mission. The REPSS design features include: 

• Very limited time for group members to share information, synchronize plans, 

execute, assess results, and adjust plans in a fast paced environment. 

4
Civil considerations – relate to civilian populations, culture, organizations, and 

leaders within the area of operations. Commanders must factor public opinion into 

their vision of the battlespace. The choice of a course of action and the allocation of 

resources impacts the protection and welfare of the local population, and public 

opinion. In stability operations and support operations, these people are a central 

feature of the area of operations. The REPSS design features include: 

• A requirement to factor public opinion for each town into decisions each week 

about the amount of food supplied, security of warehouses and transportation 

provided, as well as the number and experience level of contract security guards 

required. 

Exercise design features. The REPSS research environment was developed to provide a 

collaborative planning task environment in which groups must iteratively develop, execute, and 

refine their plans over time in response to a dynamic environment. A summary of how the 

REPSS exercise incorporates a number of design features necessary for the examination of 

collaborative planning is provided below: 

• Employ a non-doctrinal collaborative planning task, and user-friendly interface 

tools, to reduce the impact of individual skill differences, and minimize the trainup requirement. 

• Require participants to frame the problem space as part of their task. Provide a 

task with high face validity and immediate performance feedback that instills 

achievement motivation. 

• Employ interdependent task roles requiring shared understanding for success. 

• Design the exercise to present unforeseen circumstances, opportunities, and 

threats. 

 

• Employ a single “goodness of planning solution” quantitative metric, and assess 

adherence to higher commander’s guidance. 

• Employ measures of staff planning solution synchronization. 

• Employ quantitative estimates of group’s ability to share information and adapt 

plans in anticipation of future events. 

Report Organization 

5
The goals for the present research were to develop a collaborative group planning 

simulation and supporting measures, and to conduct experiments investigating group planning 

performance. The report is organized to address these goals as follows: 

• The Method section describes the development of the collaborative planning task, 

the development of measures, and the conduct of an experiment using REPSS. 

 

• The Results section describes the experiment results, with a focus on the 

comparison of co-located and distributed group collaborative planning 

performance. 

• The Discussion section describes the refinement of the REPSS simulation and 

supporting measures based on experiment results. 

Method 

Exercise Overview 

The REPSS exercise was designed to provide a simplified resource allocation task that 

presents a collaborative planning situation that has face validity for Soldiers. An exercise was 

developed depicting a hypothetical stability and reconstruction operation being conducted in 

Azerbaijan, consistent with recently developed FM 1 (The Army) guidelines (DA, 2005, 3-29). 

The simplified task does not require doctrinal knowledge. The train-up and conduct of the 

exercise requires approximately three hours to complete. Training includes an introductory 

briefing, followed by self-paced computer-delivered instruction. 

Previous ARI research with a group resource allocation task (Lussier, 1990, 1992) 

provided very useful guidance in developing an exercise for the REPSS that requires separate 

teams competing for resources to collaborate effectively in order to maximize the group-level 

payoff on a task. The REPSS exercise requires command group members to collaborate in 

optimizing the allocation of limited resources to acquire food, transportation, and security 

necessary to provide weekly food shipments to four towns, over a period of four weeks. The 

seven-member command group is configured as a commander and three two-person supply, 

transportation, and security teams. Funds for purchasing resources are extremely limited, 

requiring that the commander and teams synchronize their efforts through iterative collaborative 

planning to provide the most support possible with available funds. The population size and 

threat conditions differ for each town. Also, the threat conditions for each town are recalculated 

each week based on the proportion of required humanitarian relief supplies that were 

successfully delivered to each town. Estimated costs for food and medical supplies, 

transportation, and security personnel were derived from consulting Internet supplier sites, and 

sites providing information regarding the economy of Azerbaijan. It should be noted that food 

and medical supplies are shipped as a bulk package Humanitarian Supply Unit (HSU) that can 

meet the needs of one hundred people for one week. An automated performance assessment 

capability provides the total quantity of HSU delivered by each group, and estimates of plan 

synchronization. A summary of group member responsibilities is provided below: 

6
 Commander 

• Identify a strategy for providing supplies to towns across weeks. 

• Enforce time management. 

• Provide guidance and approve team spending plans. 

 Supply Team

• Calculate the requirement, and purchase food and medical supplies. 

• Calculate the requirement, and rent distribution centers. 

• Decide on the type of distribution centers to rent (normal or reinforced). 

 Transportation Team

• Calculate the requirement, and rent trucks to carry food. 

• Calculate the requirement, and rent trucks to carry security guards. 

• Decide on the type of trucks to rent (normal or up-armored). 

 Security Team

• Calculate the requirement, and hire convoy security guards. 

 

• Calculate the requirement, and hire town security guards. 

• Decide on the type of guards to hire (civilian or professional). 

Participants 

Four groups of U.S. Army officers and non-commissioned officers (NCOs) served as the 

seven-member command groups. Participants consisted of 28 U.S. Army officers and NCOs (4 

Captains, 3 Lieutenants, and 21 NCOs). All but one of the participants were male. Participants 

were assigned to their seven-person groups by the Fort Knox troop support office so that 

members of the same brigade or squadron would be in the same group. 

Apparatus 

 

7
Overview. The commander and each two-person team have a dual-display networked 

computer and hand-held voice radio with which to conduct their collaborative planning. The 

networked computers provide the capability to send and receive text messages, and provide two 

shared data matrices where the commander and teams can develop a weekly spending plan to 

support four separate towns across four weeks. The plan is entered via on-screen resource 

request forms. The reaction of the environment to the allocation of supplies each week comes to 

the command group members in the form of pre-written text messages indicating losses and 

damage proportional to the shortfall in supplies delivered. 

Co-located and distributed planning conditions. A central issue for the present research 

is the Army’s transition from the traditional face-to-face (co-located groups) tactical operations 

center environment to Future Force collaborative planning across networked communication 

systems (distributed groups). A “co-located group” condition was created by having all seven 

members perform their planning tasks within a common room. A “distributed planning group” 

condition was created by placing the commander, and each of the three two-man teams in 

separate rooms (four rooms total) which prevents any direct visual or verbal contact between the 

commander and teams. Figure 1 shows the REPSS command group configured as a co-located 

group (left) and a single team participating as part of a distributed command group (right). 

Figure 1. Participants working in the co-located group condition (left photo), and in the 

distributed group condition (right photo). 

The ease of communication differs for the co-located and distributed groups in several 

respects. The co-located group can engage in direct verbal communication, and can move about 

the room to view information and planning decisions presented on each teams’ computer 

monitors. There are also a number of visual cues available to co-located groups that enhance 

communication, such as facial expressions, and body language, which could convey information 

about agreement within the group, and workload levels. Also, the co-located group can engage 

in parallel “sidebar” conversations – while the distributed group must speak one at a time on a 

single channel radio net which all group members monitor. In contrast to the co-located group, 

the distributed group has only the radio, text-messaging, and the shared information matrices 

available to exchange information. 

8
Interface design. The REPSS computer interface allows the planning group to gather and 

share information, create a purchase request, and submit the request to send simulated convoys 

to four towns. The planning group gathers information by reading text messages and electronic 

town newspapers, and by clicking on digital map features to bring up information windows. 

Group members share information by face-to-face verbal communications, hand-held radio 

communications, text-messaging, and by entering their planning data into a shared collaboration 

matrix viewable by all members of the group. Each of the three teams create a purchase request 

using a pre-formatted spread sheet, and submit their request to send out the four convoys each 

week. The commander supervises the work of the three teams, and does not create a purchase 

request. The REPSS research environment is based on a software program developed in-house 

by ARI. The ARI Armored Forces Research Unit (AFRU) mini-lab provided the basic room

configuration and hardware setup necessary to conduct both co-located and distributed group 

planning exercises (see Appendix B). 

Map window. A dual monitor computer is provided for the commander, and for each of 

the three teams. The left display provides a map of the exercise area which can be accessed by 

selecting the “Map” tab on the menu tool bar (see Figure 2). The map depicts a fictional area 

constructed to provide a variety of terrain, road structures, and weather-related characteristics 

necessary to support the REPSS exercise goals. The map shows the point of departure for all 

convoys (Camp Puller), each of the four towns (Alpha, Echo, India, and Oscar) and the two 

routes for delivery of supplies to each town. The routes differ so that at points in the exercise 

one route is a better choice than another based on weather, terrain, and threat condition 

(THREATCON). Bridge icons have been placed on one route for each town to present decision 

making requirements for the exercise. The map does not include magnification or scrolling 

features which are not necessary for the REPSS exercise. By mouse clicking on a town, or a 

route, participants can bring up information windows. In Figure 2 a mouse click on the label 

“Town India” has pulled up the window displaying “Conditions” information regarding the town 

and current threat condition. The “Resources” tab on the town information window brings up 

information describing the food, security, and transportation resources delivered to the town the 

previous week. The map tool requires participants to “reach out” and pull up information to 

ascertain the success of each convoy, and the changing conditions in the environment. 

9
Figure 2. The REPSS map showing a pop-up information window for Town India describing the 

current threat, population size, weather, and citizen morale level. 

Messages window. The “Messages” tab on the left display brings up the Messages 

Window for sending and receiving text messages (see Figure 3). The “Send” field allows the 

commander and each team to compose and send a message to the commander, a specific team, or 

to the commander and all teams. The “Receive” field provides a list of all messages received, 

the time, and the sender. Separate messages are sent automatically to each team that provide 

information about changing conditions such as threat activities, weather, and important civilian 

events such as an election in one town. Teams see the threat reaction to each weekly convoy in 

the form of text messages describing damages incurred in transporting food to each town. 

10
Text message time sent, sender, and message window. 

Text entry and address line. 

Figure 3. Messages window showing time message sent, sender, and text of message. 

Allocations window. The “Allocations” tab on the left display brings up a resource 

request window. The spreadsheet design minimizes calculation requirements by allowing 

participants to simply type in the quantity of assets they need (either “armored five-ton” trucks, 

or unarmored “five-ton” trucks) (see Figure 4). The spreadsheet automatically calculates the 

costs of the assets for each town route and displays them in the “Total” column when the 

participant presses the “Calculate” button. This feature facilitates the iterative adjustment of 

plans, and fine tuning, based on group decisions. When a team has finished refining their plan 

for the week, they select the “Submit” button to enter their plan and start the REPSS reaction 

process. Resource request spreadsheet screens for all three teams are provided as Appendix C. 

Figure 4. Transportation team resource request tool showing two routes to each town, order 

blocks for armored and regular five-ton trucks, cost of vehicles for each route, and total cost. 

11
Collaboration matrices. The right display provides two blank 14 rows by 11 columns 

matrices which are accessed by selecting a tab on the screen tool bar. The shared matrices allow 

the group to create a structured display of information necessary for planning purposes, which 

can be modified or updated by any member of the group as the plan evolves. In practice the 

matrices were used to display and organize information entered by the commander and each 

team to help define the planning problem, share data and team estimates, and document the 

weekly planning solutions. It should be noted that the REPSS exercise directions to participants 

specifically stated that they were not required to use the matrices, and that their performance 

would not be evaluated in terms of their use of the matrices. 

Weekly town newspapers. The right side display screen also provides important 

information to participants in the form of a separate electronic weekly newspaper for each of the 

four towns. An example of the weekly newspaper is provided as Appendix D. The newspapers 

introduce an important functional requirement into the REPSS system, as participants must 

actively reach out to access the newspaper information, rather than passively wait for the 

information to be sent to them. The newspapers are accessed by a tab on the right screen. 

Newspaper content consists of the name of the town, the week of publication, a weather report, 

and descriptive accounts indicating citizen morale level. The newspaper information can be 

combined with text messages sent to the individual teams to create an understanding, or shared 

situational awareness, within the group that supports specific decisions that must be made. The 

weekly newspaper was designed as a dynamic document that is created “on-the-fly” to reflect 

changes in the citizen morale (CITMOR) for each town. Messages are automatically inserted 

into the newspaper from a data file of messages created to represent nine levels of CITMOR. 

The REPSS reactive plan response. The “Reactive” feature of the REPSS simulation 

assesses the adequacy of the food, transportation, and security assets in each weekly convoy to 

each town, and provides a reaction. Team purchase requests are automatically compared to a 

best-case solution where 100% of the recommended levels of food supplies, permanent 

distribution centers, armored trucks, and professional security guards would be provided. 

Where a team request falls short of the 100% solution, a deficiency score is calculated. The 

deficiency score is used to adjust CITMOR levels, and THREATCON levels (the likelihood of 

being attacked) for affected towns. Based on the deficiency score, the REPSS system generates 

message traffic tailored to each team describing citizen reactions, and humanitarian supply asset 

losses (HSU, transportation, security personnel). The cost of the asset losses is automatically 

deducted from the total funds that the group has available. Each message is composed of three 

elements, an event description, asset damage description, and the total dollar value of the 

damage. Examples of message traffic are as follows: 

• To security team: “Professional security guard: 1 suffered low damage. Total 

financial losses were $80.” 

• To transportation team: “Armored 5-ton truck. 1 suffered high damage, 1 

suffered medium damage. Total financial losses were $3,062.” 

Measures Development 

12
Overview. A variety of measures were developed to support the REPSS research. 

Estimates of individual characteristics, and knowledge of group roles were gathered through 

subjective self-report surveys. Planning process measures included the limited transcription of 

verbal, and text-messaging communications. The REPSS automatically created a record of each 

group’s weekly planning solution, identifying the total amount of HSU provided to the towns, 

and estimates of the groups ability to synchronize their purchases plans. The measures 

developed for REPSS, and the order in which they are presented in the report are as follows: 

 Individual Characteristics 

• Demographic Survey. 

 Group Roles Knowledge 

• Group Roles Knowledge Survey. 

 Planning Processes 

• Verbal communications frequency. 

• Text-messaging frequency. 

• Workload Survey. 

 Planning Outcomes 

• Shared Situational Awareness. 

• Quantity of HSU Delivered. 

• Plan Synchronization. 

• Performance Success (from Workload Survey). 

• Formative Evaluation Survey. 

Individual characteristics. All Soldiers completed a Demographic Survey which 

addressed: current duty assignment, level of military education, command and staff experience, 

experience with commercial computer applications, time since last wargaming a course of 

action (COA) in a command group, and experience in performing planning with the other 

members of the experimental group. The Demographics Survey is presented in Appendix E. 

Group roles knowledge. A Group Roles Knowledge Survey was developed to estimate 

each participant’s knowledge of the information requirements of the commander and three 

13
teams, which would support the efficient targeting of information to selected recipients. The 

survey requires that participants match information statements to the commander or team that 

would need this information the most (commander, supply, transportation, security). Correct 

responses were summed and a percentage correct score was recorded. The Group Roles 

Knowledge Survey is presented in Appendix F. 

Planning processes. Collaborative planning requires that the separate teams generate 

estimates of their team requirements, and that the group as a whole generate and iteratively 

refine a combined group solution. As noted previously, the process of collaborative planning 

will likely differ across groups in terms of the frequency of their verbal and text 

communications. It is also likely that the workload associated with collaborative planning will 

also differ across the commander and teams, and across planning environments. 

A transcript of commander and team radio and face-to-face verbal communications was 

prepared for one distributed and one co-located group exercise from the REPSS 1 experiment. 

Given labor-intensive nature of verbal recordings transcription the data reduction process was 

not performed for the REPSS 2 experiment. Each group exercise was approximately 140 

minutes in duration. From the transcripts the frequency of individual statements by group 

members was estimated. For the present research, the frequency of verbal communications was 

operationalized as the number of statements made by the members of the group, whether over 

the radio or in a face-to-face exchange. 

Text messages can potentially provide a viable alternative to voice channel 

communications. With REPSS, all text exchanges created in the Messages Screen are captured 

automatically in a data file which identifies both the sender and recipient of the communication. 

Transcripts of text message communications were prepared for one distributed and one colocated group exercise from the REPSS 2 experiment. The frequency of text message 

communications was operationalized as the number of text messages created by the members of 

the group. 

The examination of the collaborative planning process needs to consider the workload 

placed on the commander and each team, and on the group as a whole. Participants completed a 

brief survey addressing workload adapted from the Task Load Index (TLX) developed by the 

National Aeronautics and Space Administration (NASA-Ames Research Center, 1986). An 

estimate of participant workload was derived from the average of their ratings on five subscales 

representing workload dimensions: Mental, Physical, Temporal, Effort, and Frustration (1 = 

Low to 100 = High). The Workload Survey is presented in Appendix G. 

Planning outcomes. In the REPSS exercise the higher commander’s guidance 

specifically calls for the delivery of as much HSU as possible, delivered consistently across 

towns and weeks. Given this quantitative guidance, the total quantity of HSU delivered across 

all four weeks provides a single metric for estimating the “goodness” of each REPSS group’s 

planning. Additional assessments of the consistency of HSU delivery across weeks, and across 

towns, provide additional quantitative estimates of successful task performance. The REPSS 

program automatically calculates the amount of HSU delivered to each town each week, and also 

14
provides a total HSU delivered score. The information is available to the experimenter, and can 

be provided to participants immediately after the experiment as feedback. 

 

The REPSS weekly spending data provide estimates of how well the three teams were 

able to synchronize their individual plans in terms of purchasing only the assets necessary to 

transport the HSU purchased and security guards hired. The group’s plan is not synchronized 

when HSU or security guards cannot be transported due to lack of vehicles, or where 

transportation was rented in excess of that needed. Using data from the weekly request 

spreadsheet, the number of trucks required to transport HSU, and security guards, can be 

calculated, and compared to the number of trucks actually rented. A perfect match would yield a 

score of 100%, while any percentage of trucks under or over the required number represents a 

waste of assets, and a failure of the teams to synchronize their plans (10% too few, and 10% too 

many would both yield a synchronization score of 90%). One advantage of the synchronization 

estimate is that it reflects the ability of the group to coordinate each weekly plan, for each of the 

four towns, no matter how few resources might remain in the budget. As with the HSU delivery 

estimates, the synchronization estimate is automatically calculated by the REPSS and can be 

provided to participants as feedback immediately after each experiment. 

Another factor to consider in estimating group planning outcome success should be an 

assessment of the group’s shared visualization of the problem space, or shared situational 

awareness (SSA), which is thought to underlie successful group problem solving (Endsley et al., 

2000). The REPSS exercise incorporated an embedded problem which required the group to 

share information, and implement measures to adapt their planning to meet changing situation 

requirements. The embedded problem involved changing the situation for Town Oscar in Week 

2 to icy road conditions and a high threat environment. Under the icy road conditions the less 

expensive unarmored trucks are more effective than the more expensive armored trucks in 

delivering food and security personnel to the town. Successful planning was estimated as the 

percentage of unarmored trucks assigned to the Town Oscar in Week 2. 

The Workload Survey contains a subscale for perceived Performance Success which asks 

each participant to rate “How successful were you in accomplishing what you were asked to do?

 (1 = Failure to 100 = Perfect). The Performance Success sub-scale ratings were used to 

investigate how accurately participants could assess the success of their planning efforts in terms 

of delivering HSU to the towns. 

 A nine-item formative evaluation survey was developed to gather Soldier feedback on the 

adequacy of the REPSS exercise procedures, and to gather suggestions for improvements. 

Survey topics addressed the pace of the planning exercise, the adequacy of the researcher’s 

briefings and simulation train-up, and ease of using the interface. The Formative Evaluation 

Survey is presented in Appendix H. 

Procedure 

15
Schedule of experiment events. All Soldiers participated in a single three and one-half

hour REPSS exercise. The researchers first provided each participant with a three-ring binder 

containing an informed consent form, exercise overview briefing, REPSS collaboration tool 

orientation training materials, general instructions, commander and team instructions, and survey 

instruments. The researchers presented a short exercise overview briefing, and then had 

participants work through self-paced training with the REPSS team tasks and collaboration tools. 

Participants were given 40 minutes to perform the planning for Week 1, and 20 minutes to 

perform planning for each of the three subsequent weeks. At the conclusion of the REPSS 

exercise the participants completed the Demographic, Formative Evaluation, Workload, and 

Group Roles Knowledge Surveys. 

Training for participants. Training for participants consisted of a 10-minute overview 

briefing, followed by self-paced training at the commander and team workstations. The selfpaced training consisted of Power Point slides which presented general instructions common to 

all participants for the exercise, and also commander or team-specific instructions. The 

participant general instructions slides are provided as Appendix I. The commander and teamspecific instruction slides included a step-by-step orientation to REPSS interface actions, and an 

example of a data entry task. For purposes of illustration, the team-specific instruction slides for 

the supply team are provided as Appendix J. A researcher observed the participant training and 

provided assistance as needed. Total time required for the overview briefing and self-paced 

training was approximately one hour. After the train-up the group began the REPSS exercise. 

A typical exercise. A typical exercise would begin with the commander directing the 

teams to generate a plan for the Week 1 supply convoy. One way of developing an initial plan is 

to have the teams develop the 100% solution. The 100% solution is the most expensive option, 

where 100% of the recommended HSU is transported to each town, fortified distribution sites are 

rented for the HSU town storage, armored trucks are rented for HSU and security guards, and 

professional security guards are hired for both convoy security, and town security. The 100% 

solution is too expensive to sustain over four weeks, but this solution provides the group with an 

indication of how much they will have to cut back on spending. 

Some planning activities need to progress in a sequential fashion. The supply team

should first estimate the amount of HSU to transport to each town, and the number of 

distribution sites that need to be rented. The transportation team takes this information and 

determine how many trucks are needed for the HSU. The security team would then determine 

how many guards must be hired for the supply convoy and for town security, and provide this 

information to the transportation team so that they can determine the number of trucks needed to 

transport the supply guards and town guards. Each team would calculate the total cost for their 

contribution and provide this to the commander. The commander would then calculate the total 

cost of the Week 1 relief effort, and compare this cost to the total funds available for the fourweek effort. Where the planned spending level falls above or below the level of funds the 

commander wishes to expend, the teams will be asked to adjust their spending. A great deal of 

the collaborative planning process involves having the commander and the three teams develop a 

solution that matches expensive well-protected convoys to dangerous areas, and less expensive 

lightly protected convoys to safer areas. 

16
Results 

Individual Characteristics 

Demographic Survey. All participants completed a six-item Demographic Survey. The 

survey addressed experience with commercial computer applications, time since last wargaming 

a COA in a command group, and experience in performing planning with the other members of 

the experimental group. Table 1 provides selected results from the Demographic Survey which 

illustrate the types of comparisons that can be made with the measures from the REPSS 

experiment. The experiment was conducted with mixed groups composed of an officer serving 

as commander, and teams that could be composed of both officers and NCOs. Soldier responses 

indicated that on average they had between one to three years of experience with commercial 

computer applications. Participants in Group1 had the most experience in group planning, with 

57% having engaged in group planning in the last year, while Group 4 participants were the least 

experienced, with only 14% having recent group planning experience. Group 2 had the largest 

number of members who had experience planning together prior to the experiment, while Group 

4 members had the fewest with shared planning experience. The survey data reveal that Group 1 

differs from the other three in that the group could assign an officer to lead each team, while the 

other three groups could not. The analysis and results are limited to descriptive statistics. 

Inferential statistics were considered inappropriate due to the small (n = 4) sample of groups. 

Table 1 

Selected Demographic Survey Results 

Experiment 

Group 

Environment 

Condition 

Ranks Did Group

Planning in 

Last Year 

Group Members 

Planned With 

Before 

Years of 

Computer 

Experience 

Group 1 Distributed 1 CPT 

3 2LT 

3 SSG 

57.14% 2.1 2.93

Group 2 Distributed 1 CPT 

1 MSG 

5 SSG 

28.57% 4.0 3.29

Group 3 Co-located 1 CPT 

1 SFC 

5 SSG 

42.86% 1.8 1.43

Group 4 Co-located 1 CPT 

6 SSG 

14.29% 1.4 2.00

Group Roles Knowledge 

17
Group Roles Knowledge Survey. Participants were asked to indicate who should be the 

primary, and secondary recipient of each of the eleven messages presented on the Staff Roles 

Knowledge Survey. After a review of the results three of the eleven items were dropped after it 

was determined that the messages were relevant for the commander and all teams. The resulting 

assessment of eight survey items averaged across all members of the group yielded the following 

group scores: Group 1 = 7.67, Group 2 = 7.20, Group 3 = 7.17, and Group 4 = 7.14. The scores 

suggest that the members of each of the groups possessed a basic understanding of the roles and 

information requirements of the commander, and each of the teams. The assessment of each 

group’s knowledge of required commander and team roles is valuable, as it provides an estimate 

of whether group members possess the knowledge necessary to selectively provide information 

to the specific teams that needed it, and to request information from teams that have it. 

Planning Processes 

Frequency of verbal interactions. A transcript of verbal communications was created for 

 two groups who participated in the REPSS 1 experiment. Figure 5 shows the comparison of 

commander and team (TM) verbal statements for one distributed group and one co-located group 

from the REPSS 1 experiment. The distributed group exchanged a total of 262 verbal statements 

compared to 455 (42% more verbalizations) for the co-located group. The trend for more 

frequent verbal interactions in the co-located group was consistent across the commander and 

three teams. 

0

20

40

60

80

100

120

140

160

180

200

Commander Supply TM Transportation TM Security TM

Statement Frequency

Distributed

Co-located

Figure 5. Bar chart comparison of the frequency of verbal statements made by the commander 

and each of the three teams during planning for distributed and co-located groups. 

Frequency of text communications. A transcript of text messages was automatically 

recorded in the REPSS 2 experiment. The average frequency of text messages for the two colocated and two distributed groups are presented here (see Figure 6). Text messaging was not 

used frequently by either co-located or distributed groups. As might be expected, the co-located 

groups made almost no use of text messaging, a trend that appears consistent across the 

commander and team positions. 

18
0

2

4

6

8

10

12

14

16

Commander Supply TM Transportation TM Security TM

M

essag

e Frequenc

y

Distributed

Co-located

Figure 6. Bar chart comparison of the frequency of text messages sent by the commander and 

each of the three teams during planning for distributed and co-located groups. 

 Collaboration matrix communications. A screen capture was made to document the 

collaboration matrix entries when each weekly spending plan was submitted. For purposes of 

illustration, Table 2 presents the collaboration matrix developed by one co-located group. In 

future analyses the collaboration matrix data might provide estimates of the accuracy and 

timeliness of information exchanged within a group, and their ability to accurately form the 

problem space, laying out the critical relationships between the many variables. For the present 

research no assessment was conducted for the matrix data. 

Table 2 

Shared Collaboration Matrix for Display of Current Situation Information and Team Plans 

A B C D E F G H I J K

1 Town Population Threat 

Level 

CITMOR HSU HSU

Trucks 

Guard 

Trucks 

Cost 

HSU 

Ware

hous

e 

(min) 

Cost 

Ware

hous

e 

min

Cost 

Ware

house 

max

2 ALPH

A 

1600 Elevated Low 16 8 16 192k 1 55k 130k

3 ECHO 2400 Guarded Neutral 24 12 12 288k 2 110k 260k

4 INDIA 4000 Severe Very Low 40 20 80 480k 2 110k 260k

5 OSCA

R

6800 Elevated Low 68 34 68 816k 4 220k 520k

Subjective estimates of workload. Figure 7 presents a comparison of perceived workload 

levels reported by the commander and teams for two distributed and two co-located groups at the 

conclusion of each exercise. It was thought that distributed groups might find it harder to 

19
accomplish collaborative planning tasks through computer-mediated communications, compared 

to the face-to-face collaboration of co-located groups performing the same REPSS exercise. 

However, this preliminary look at workload does not reveal consistent differences between the 

distributed and co-located groups. On average, the commander and teams report experiencing 

workload at or below the mid-level range, suggesting low to moderate levels of workload. 

0

20

40

60

80

100

Commander Supply TM Transportation TM Security TM

Worklo

a

d

Distributed

Co-located

Figure 7. Bar chart comparison of the planning workload levels reported by the commander and 

each of the three teams for distributed and co-located groups. 

High 

Low 

Planning Outcomes 

Percentage of HSU delivered. The percentage of each town’s HSU requirement that was 

actually delivered by each group provides a single estimate of the “goodness” of the group’s 

planning solutions. Results from the present experiment do not provide evidence that any one of 

the four groups was more successful than another in delivering HSU to the towns (see Figure 8). 

 Group 3 provided the most HSU (55.43%), while Group 2 provided a nearly identical quantity 

(54.98%). The results do not provide evidence of a difference in performance when comparing 

the performance of distributed and co-located groups. 

20
0

10

20

30

40

50

60

Group 1

Distributed

Group 2

Distributed

Group 3 

Co-located

Group 4 

Co-located

P

ercent H

S

U D

elivered

Figure 8. Percent of total required HSU delivered by each group. 

Consistency of HSU delivery across weeks. Participants were instructed to try to achieve 

a balanced distribution of food across the four towns, and the four weeks. The percentage of 

required HSU delivered to each town across the four weeks provides an additional estimate of 

how well each group was able to achieve the goal of balanced distribution of food. Figure 9 

shows the percentage of required HSU supplied each week by the groups. In general, it can be 

seen that the groups had difficulty providing a consistent rate of food distribution across weeks. 

None of the four groups were able to provide any HSU to the four towns by Week 4. Group 1 

was able to provide nearly 100% of the required HSU for the first week, failed to send any HSU 

to the towns in the second week, and provided more than enough HSU to the towns in Week 3, 

and failed to provide any support to the towns in Week 4. Group 2 provided HSU above the 

recommended levels in Week 1, succeeded in providing about half the required amount for 

Weeks 2 and 3, and was unable to deliver any HSU to the towns in Week 4. 

0

20

40

60

80

100

120

140

Group 1

Distributed

Group 2

Distributed

Group 3

Co-located

Group 4

Co-located

Percent H

S

U

Delivered

Week 1

Week 2

Week 3

Week 4

Figure 9. Percentage of required HSU delivered each week, for each group. 

21
Consistency of HSU delivery across towns. The delivery of HSU should also be 

examined for consistency across each town. Figure 10 presents the percentage of recommended 

HSU actually delivered (undamaged) to each town across all four weeks. These results suggest 

how well the group was able to achieve the commander’s goal of providing supplies consistently 

across towns. The data suggest that Group 1 showed a great deal of inconsistency in allocating 

HSU to towns, providing a much greater percentage of HSU to Town Oscar, the largest town, 

compared to the other three towns. Group 2 was fairly consistent in the percentage of required 

HSU it was able to deliver to the four towns. Group 3 was able to deliver a high percentage of 

required HSU to three out of four towns, providing proportionally less HSU to the largest town, 

Oscar. Group 4 was also fairly consistent in its delivery of required HSU across the four towns. 

 These results provide evidence that co-located groups were better at achieving the goal of 

maximizing the delivery of required HSU consistently across all four towns. 

0

20

40

60

80

100

Group 1

Distributed

Group 2

Distributed

Group 3 

Co-located

Group 4 

Co-located

Percent HS

U Delivered

Alpha

Echo

India

Oscar

 Figure 10. Comparison of percentage of required HSU delivered to towns by each group. 

Spending synchronization. As previously noted, The REPSS weekly spending data can 

provide estimates of how well the three teams were able to synchronize their individual plans in 

terms of purchasing only the assets necessary to transport the HSU purchased and security 

guards hired. Any percentage of trucks under or over the required number represents a waste of 

assets, and a failure of the teams to synchronize their plans. Figure 11 presents the estimates of 

team plan synchronization for the four groups. The synchronization estimate is useful in 

revealing how well each group can create an efficient plan each week that does not waste assets. 

 One advantage of the synchronization score is that it yields an estimate of planning efficiency 

for each week that is not influenced by performance in previous weeks. Each week the group 

begins with some level of funds to work with, and the synchronization score reflects how well 

the group can create an efficient plan with these resources. 

22
0

20

40

60

80

100

Group 1

Distributed

Group 2

Distributed

Group 3 

Co-located

Group 4 

Co-located

Plan Synchronization

Week 1

Week 2

Week 3

Week 4

Figure 11. Level of plan synchronization for each group and each week. 

Shared situational awareness. The REPSS exercise incorporated an embedded SSA 

problem where icy road conditions created the situation where the less expensive unarmored 

trucks would be more successful than expensive up-armored trucks in transporting supplies to 

Town Oscar in Week 2. Information identifying this situation was provided in text message 

statements sent to each team, and in the Town Oscar newspaper. A group recognizing and 

responding to this situation would send 100% unarmored trucks to Town Oscar. The percentage 

of unarmored trucks sent in the Town Oscar convoys for each of the four experimental groups 

were: Group 1 (distributed) 26%, Group 2 (distributed) 67%, Group 3 (co-located) 100%, and 

Group 4 (co-located) 53%. These results provide evidence that Group 1 failed to attend to 

available information stating that unarmored vehicles should be assigned to the Town Oscar 

route. It is unclear whether Groups 2 and 4 modified their plans in response to the icy road 

conditions information, or whether their allocation of unarmored trucks to the Town Oscar route 

might reflect a 50/50 allocation strategy for up-armored and unarmored vehicles. The actions of 

co-located Group 3 are consistent with having shared information and made the correct choice in 

assigning 100% unarmored trucks to the Town Oscar route. 

Subjective estimates of performance success. The TLX survey Performance Success subscale scores can be compared to the actual percentage of required HSU delivered by each group 

to explore how accurately groups can assess the success of their planning efforts (see Figure 12). 

 In Figure 12 both subjective performance success ratings and percentage of HSU delivered are 

described on a 0 – 100% scale. The results show a fairly close relationship between the actual 

percentage of HSU delivered and each group’s subjective rating of their performance. The data 

show a slight difference between co-located and distributed groups, where distributed groups 

perceive a higher level of success in their subjective ratings compared to the distributed groups 

for similar levels of actual percentage of HSU delivered. 

23
0

20

40

60

80

100

Group 1

Distributed

Group 2

Distributed

Group 3 

Co-located

Group 4 

Co-located

P

erfor

m

ance Success

% HSU Delivered

Success Rating

Perfect 

Failure 

Figure 12. Comparison of average performance success ratings and actual percentage of HSU 

delivered for distributed and co-located groups. 

Contributors to collaborative planning success. The overall percentage of HSU 

delivered score can be compared to estimates of group characteristics, process performance, and 

additional performances success estimates to explore potential contributors to collaborative 

planning success. Cianciolo and Sanders (2006) have provided a rank-ordering approach for 

comparing group characteristics and planning process measures to estimates of planning success. 

The approach is particularly useful for presenting small sample data in an effort to identify 

trends. Applying the rank-ordering technique to the REPSS measures, an example of a 

framework for organizing and presenting REPSS data can be provided (See Table 3). The four 

groups are ranked in order of the percentage of HSU they delivered over the four week exercise, 

where Group 3 provided the most HSU (55.43%), and Group 4 delivered the least HSU 

(51.16%). 

Table 3 

Rank Ordering of Group Characteristics, Planning Process, and Planning Outcome Measures 

Relative to HSU Delivered (Groups are ranked in descending order by amount of HSU 

delivered) 

 

Group 

Characteristics 

Planning 

Process 

Group % HSU Planning Outcomes 

Delivered 

% Planned 

In Last 12 

Months 

 Group 

Roles 

Score 

Group 

Perceived 

Workload

SSA % 

Correct 

 Trucks 

% Plan 

Synchronization

Group 

Perceived 

Success 

Experimental

Condition 

3 55.43 42.86 7.17 37.25 100 53.01 62.75 Co-located

2 54.98 28.57 7.20 56.93 26 77.64 53.20 Distributed

1 51.25 57.14 7.67 47.95 53 71.90 42.75 Distributed

4 51.16 14.29 7.14 49.03 67 65.30 62.25 Co-located

24
The results presented in Table 3 show that for this particular experiment the four groups 

performed about the same with regard to HSU delivery. With regard to group characteristics, the 

groups differ considerably in terms or their recent experience in group planning, with three out 

of the seven members (42.86%) in the highest performing group (Group 3) indicating that they 

had performed group planning within the last year. The groups were very similar in their 

knowledge of group roles and information requirements. Looking at the planning process, the 

highest performing group reported experiencing a lower level of workload during the exercise 

compared to the other three groups. With regard to planning outcome indicators, the highest 

performing group was also the only group to successfully identify the “icy road conditions” 

manipulation in Week 2 of the experiment and correctly assign 100% unarmored trucks to the 

route. The highest performing group scored lowest on the plan synchronization estimate, and 

highest in their subjective assessment of their groups success. Group 3 assigned to the colocated condition scored slightly higher on the overall measure of performance, the total 

percentage of required HSU delivered across all four towns and all four weeks. 

 Co-located versus distributed group comparison summary. The key manipulation for the 

REPSS experiment was the comparison of performance for groups working under distributed 

versus co-located conditions. With regard to the collaborative planning process, in the 

comparison of one co-located and one distributed group, the co-located group demonstrated a 

greater frequency of verbal communications (see Table 5). In the comparison of two co-located 

and two distributed groups , the distributed groups exchanged a greater number of text messages 

compared to the co-located groups (see Table 6). With regard to performance outcomes, the 

results show no consistent performance trends that would indicate a difference between the 

distributed and co-located groups. Under both conditions the groups appear to have performed 

at roughly the same levels with regard to HSU delivery, and plan synchronization. The four 

groups report roughly equivalent levels of perceived workload, and perceived success in 

completing the planning tasks. 

The REPSS formative evaluation survey. Results provided useful guidance identifying 

where changes needed to be made, and confirming the adequacy of written instructions, interface 

training, and interface tools. Twenty five of the 28 participants (89%) indicated that the general 

instructions, and commander and team instructions adequately described the planning exercise, 

and all participants indicated that the data screen training adequately prepared them for their 

tasks. Likewise, 27 of the participants indicated that they were able to use the REPSS screens to 

complete tasks in a timely manner. One design goal for REPSS was to present a task that placed 

difficult but achievable time demands on participants. Participants rated the pace of the REPSS 

exercise as Too Fast (1), About Right (2), or Too Slow (3). With regard to time demands, 39% 

of the participants responded that the pace of the exercise was “Too Fast,” while 57% reported 

that the pace was “About Right,” and one indicated that the pace was “Too Slow.” The average 

rating for the distributed groups was 1.43, compared to 1.86 for the co-located groups, 

suggesting that the distributed groups experienced more time pressure than the co-located 

groups. One potential application for the REPSS is as a training tool for building command 

group skills. When asked about the training potential of REPSS, 26 of the 28 participants 

reported that the planning exercise could be useful in command group training (two participants 

did not respond to the question). 

Discussion 

25
Summary 

The goals for the present research were to develop a collaborative group planning 

simulation and supporting measures, and to conduct experiments investigating group planning 

performance. The REPSS was successful in stimulating collaborative group planning activities, 

and providing automated estimates of performance success. From observations and written 

responses to post-experiment surveys, the pilot test participants accepted the stability and 

reconstruction humanitarian relief scenario, and actively worked to achieve the best solution 

possible. The collaborative planning process measures developed for REPSS provided valuable 

estimates of group planning performance. Findings for the comparison of co-located and 

distributed group conditions clearly showed differences in verbal and text-messaging 

communications for the two conditions. One limitation with the process performance measures 

is that they require labor-intensive communications transcript coding. Automated collaborative 

planning outcome measures developed in this effort provide useful estimates of team planning 

synchronization, as well as the desired “goodness of performance” metric. Results from the colocated vs. distributed group comparison did not reveal differences in terms of percentage of 

required supplies delivered, and the synchronization of group plans. Most of the participants 

(93%) indicated that the REPSS planning exercise could be useful in command group training. 

Several performance problems were identified which could be the subject of future 

assessments, or lead to changes in the REPSS exercise: 

• Groups developed well synchronized plans, only to realize late in the planning 

process that the combined cost of the team plans was too high. Commanders would 

then impose an across-the-board percentage reduction in spending that lowered team

plan synchronization. Future research will investigate the benefits of training for 

early decisions in the planning process. 

• Groups would often run out of funds during the third week of the four week exercise. 

 Future research should incorporate an assessment based on patterns of spending as a 

performance outcome measure. 

• Groups would try to introduce a different way of performing the task to gain 

efficiencies. In response, the introductory briefing was changed to include a drawing 

that illustrated the requirements of the humanitarian relief task. 

• While the higher commander’s guidance was to provide as much food as possible to 

the towns, one group chose to adopt a contradictory strategy which focused on 

protecting the guards at the expense of not delivering food. The introductory briefing 

was revised to include a story line that emphasizes the need to focus on food delivery. 

 

• Occasionally one team within a group would fail to submit a purchase request, or 

request goods for only a single town. The introductory briefing was revised to stress 

that purchase requests must be for all four towns. The experimenter’s display was 

modified to indicate whether each team had submitted their request. 

26
The REPSS Refinements 

 

A number of enhancements were incorporated into the simulation following the REPSS 2 

experiment. Three additional situational Awareness (SA) manipulations were developed for the 

humanitarian relief scenario, so that one change to the operating environment is presented each 

week. An additional route was added for each town so that the group would have to choose the 

better of two routes in sending convoys to each town. The Power Point self-paced training 

program was revised to reduce the repetition of material. The REPSS program was installed in 

networked laptop computers providing a portable data collection capability. The portable lab 

arrangement supports data collection utilizing troop support at various posts. 

Future REPSS Research 

As additional data are gathered with the REPSS an effort will be made to investigate the 

relationship between demographic characteristics of participant groups and performance 

outcomes, such as the impact of rank and previous deployment experience on the quality and 

timeliness of planning solutions. Likewise, as additional data are gathered there will be an 

opportunity to correlate subjective measures such as perceived workload and performance 

success with an empirical metric of planning quality, such as plan synchronization and quantity 

of HSU successfully delivered. 

Future experiments will investigate the utility of specific collaborative planning 

techniques. Time management appears to be a critical, and difficult task for planning groups. 

Experiment observations suggest that the REPSS groups often failed to adequately monitor the 

time available for their planning, and allocated insufficient time to synchronize the separate team

requirements. Future experiments will examine whether the imposition of the Early Decision 

strategy planning method identified by Lussier (1992) can result in a significant improvement in 

planning performance outcomes. Future experiments might also investigate the utility of the 

sharing information using the synchronization matrix. Experimenter observations suggest that 

when groups did not use the matrix, they devoted considerable time to verbally exchanging the 

types of information that would likely be in the matrix. This appeared to be true for both 

distributed and co-located groups. 

A future enhancement desired for the REPSS is the development of equivalent forms of 

the task vignette. The development of an equivalent version of the REPSS task could greatly 

facilitate research efforts. Having each participant group perform two separate exercises under 

differing conditions would allow the employment of within-group comparisons. This could be 

very advantageous, given the increasing difficulties faced in obtaining troop support for group 

task research, and the desire to control for individual differences in analyses. 

The REPSS Training Potential 

27
Command group planning exercises can be time consuming, expensive, and infrequently 

performed. The REPSS exercise might be employed as a three hour long, inexpensive training 

event, that could be conducted as frequently as required to maintain basic skills in collaborative 

planning, and as a means to convey unit-specific tactics, techniques, and procedures for 

collaborative planning. The REPSS exercise address key skills that should be a prerequisite for 

success in any planning task. Time management, critical thinking, and communication skills 

should be essential to all planning tasks, and these skills might be developed with a simplified 

resource allocation task, rather than requiring a full fidelity exercise requiring extensive doctrinal 

and subject matter knowledge. On the other hand, mastery of doctrinal and subject matter 

knowledge would not guarantee success in collaborative planning if time management, critical 

thinking, and interpersonal communication skills have not been mastered. 

As noted earlier, the REPSS system was designed to incorporate underlying processes 

associated with mission command, as outlined in FM 1, The Army (DA, 2005, 3-33). The 

processes include sharing commander’s intent, resource allocation, and synchronization of plans. 

The REPSS exercise incorporates features that can reinforce the importance of gathering, 

exchanging, and integrating METT-TC information as a condition for mission success. The 

ability to host REPSS on standard desktop and laptop computers would facilitate the 

implementation of REPSS as a training tool for Army units. The built-in self-paced training, and 

short train-up time requirement, also supports the implementation of REPSS as a training tool. 

The performance assessment features built into the REPSS provides the basis for automated 

feedback reports which could be accessed by an instructor to pause an exercise and interject 

training points. 

28
References 

Cianciolo, A., & Sanders, W. R. (2006). Wargaming Effectiveness: Its Conceptualization and 

Assessment. (ARI Technical Report 1178). Arlington, VA: U.S. Army Research 

Institute for the Behavioral and Social Sciences. 

Department of the Army. (2002). Field Manual (FM) 3, Operations. Washington, DC: Author. 

Department of the Army. (2005). Field Manual (FM) 1, The Army. Washington, DC: Author. 

Endsley, M. R., Holder, L. D., Leibrecht, B. C., Garland, D. J., Wampler, R. L., & Matthews, 

M. D. (2000). Modeling and measuring situation awareness in the infantry operational 

 environment. (ARI Research Report 1753). Alexandria, VA: U.S. Army Research 

Institute for the Behavioral and Social Sciences. 

Holden, W. T., Smith, M. L., Conzelman, C. E., Smith, P.G., Lickteig, C. W, & Sanders, W. R. 

 (2005). Developing an environment for exploring distributed operations: A wargame 

 example. (ARI Research Report 1836). Arlington, VA: U.S. Army Research 

Institute for the Behavioral and Social Sciences. 

Lickteig, C. W., Sanders, W. R., Durlach, P. J., Lussier, J. W., & Carnahan, T. J. (2003). Human 

performance essential to battle command: Report on four future combat systems 

command and control (FCS C2) experiments. (ARI Research Report1812). 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Lussier, J.W. (1990). VARWARS: A Group Problem Solving Exercise (ARI Research Product 

90-09). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social 

Sciences. 

Lussier, J. W. (1992). Early Decisions and Concurrent Option Comparison in Problem Solving 

 Groups (ARI Research Report 1618). Alexandria, VA: U.S. Army Research Institute for 

the Behavioral and Social Sciences. 

NASA-Ames Research Center. (1986) Collecting NASA workload ratings: A paper and pencil 

package (Version 2.1). NASA-Ames Research Center, Human Performance Group. 

Moffet Field, California. 

U.S. Army Research Institute for the Behavioral and Social Sciences. (2006). Fiscal Year 2006 

Program. Arlington, VA: U.S. Army Research Institute for the Behavioral and Social 

Sciences. 

 

29
Appendix A 

Acronyms 

AFRU Armored Forces Research Unit 

ARI U.S. Army Research Institute for the Behavioral and Social Sciences 

ATO Army Training Objective 

C2 Command and Control 

CITMOR Citizen Morale 

COA Course Of Action 

COE Contemporary Operational Environment 

DA Department of the Army

DARPA Defense Advanced Research Projects Agency 

FCS C2 Future Combat System – Command and Control 

FM Field Manual 

HSU Humanitarian Supply Unit 

METT-TC Mission, Enemy, Terrain, Troops, Time, Civilian 

NASA National Aeronautics and Space Administration 

NCO Non-Commissioned Officer 

REPSS Reactive Planning Strategies Simulation 

SA Situational Awareness 

SSA Shared Situational Awareness 

THREATCON Threat Condition 

TLX Task Load Index 

TM Team

 

A-1
Appendix B 

REPSS Mini Lab Room Configuration 

REPSS Mini Lab Room Configuration 

Room 29 Room 27 

Desk 

Chair 

Camera 

Computer Screen 

*SECURITY 

 TEAM *COMMANDER 

 *SUPPLY 

 TEAM 

*TRANSPORTATION 

 TEAM 

EXPERIMENT 

CONTROLLER 

Room 28 Room 26 

Room 25 

* Distributed Command Positions 

 Collocated Teams were all located in RM 26 

B-1
Appendix C 

Data Entry Screens for REPSS Teams 

Data entry screen for supply team. 

C-1
Data entry screen for transportation team. 

C-2
Data entry screen for security team. 

C-3
Appendix D 

Weekly Newspaper 

The 

WEEK 

ONE India Ink

Local woman fears troubles are not over. Aziza Ustinov, 61, told sources that 

she fears that winter is not over just yet and the worst is yet to come. She states that 

Red Feather Grass, a species of plant that grows only at higher altitudes, always 

blooms in accordance with the weather. “The first buds appear when the winter is 

over.” However, if they begin to sprout, and then the buds fall off, it is due to a 

continuation of the winter season, as the self-described “superstitious old woman” 

stated. Aziza claims that this is the case and this warns that more snowfall and 

bitterly cold weather is just around the corner.

Hazardous 

blizzard 

conditions are a 

possibility by the 

end of this week. 

Be extremely 

careful if 

traveling and try 

to bundle up! 

Crime rates fall. Local police have issued a statement that this past week 

there was a significant decrease in insurgent organized or suspected crimes. Levels are 

nearing what they were before the war began, nearly one year ago this month. 

D-1
Appendix E 

Demographic Survey 

 Name _______________________ Experiment Duty Position _________________ 

Date ________________________ Branch/MOS ___________________________ 

Current Rank _________________ 

1. Current Duty Position (Months in Position) ____________________________________ 

2. Years of Regular Army Experience ___________________________________________ 

3. Highest Enlisted Rank Achieved (or N/A if not applicable) _________________________ 

4. Prior Leader/Staff Experience (and time in position) _____________________________ 

____________________________________________________________________________ 

5. Have you been deployed? ___________________________________________________ 

____________________________________________________________________________ 

6. How long has it been since you last participated in a group planning session? (Describe) 

____________________________________________________________________________ 

7. How much experience do you have in group planning tasks such as the REPSS exercise?

 (Circle a number please.) 

1 2 3 4 5

No 

Experience 

 Very Experienced 

8. How much experience do you have with commercial computer applications? 

(Circle a number please.) 

1 2 3 4 5

No 

Experience 

Less Than 

1 year 

More Than

1 year 

More Than

3 years 

5 Years 

or More 

9. Have you used computer systems to conduct military planning? Yes _____ No _____ 

10. How many members of your experiment group have you worked with before on planning tasks? 

_______________ 

E-1
Appendix F 

Group Roles Knowledge Survey 

Group Roles Knowledge Survey 

Team __________ Date __________ 

For each message below please indicate who should be the primary recipient by putting a “1” in their column. 

Indicate who should be the next recipient, if any, by placing a “2” in the appropriate box. Even if you think a 

message should go to more than two teams, only provide two answers, one “1” and one “2.” 

Check box to indicate all who the information should be forwarded to. 

MESSAGE Commander Supply Trans Security

Coalition allies have donated an extra ten million dollars in 

efforts to assist you in your mission. 

The price of HSU has gone up by 10%. 

A safer, although longer, route is now available to transport

HSU. 

S2 has reported that there may be insurgents intending to

sneak into town due to the lack of security. 

Town Oscar has not been receiving the expected minimum

amount of HSU for nearly three weeks now! Re-check your 

calculations. 

Security Guards will no longer be receiving food and water 

from headquarters battalion. You will now have to purchase 

additional HSU for the Security Guards you employ. 

We observed a group of young men behaving suspiciously

around the warehouse since early yesterday. They run away

into the crowd when they see that we have noticed them. The 

Town Security Guards on hand informed us they could use 

extra hands if they wanted to make a good town-wide search 

for the men. 

Due to security concerns an extra 30 Town Security Guards 

will be needed to be allocated in Alpha. 

The money your Team requested has been allocated by the 

Commander.

An attack in Town Alpha during the unloading of HSU has 

left many Security Guards, Five-Tons and Distribution 

Warehouses either destroyed or severely damaged. 

Locals are not talking to coalition forces much. 

F-1
Appendix G 

Workload Survey 

Team __________ Date __________ 

Task Load Index Rating Scales 

Please rate the exercise by putting a mark on each of the six scales at the point which matches your experience. 

Mental 

Demand 

 Very Low Very High

(HOW MENTALLY DEMANDING WAS THE EXERCISE?)

Physical 

Demand

 Very Low Very High

(HOW PHYSICALLY DEMANDING WAS THE EXERCISE?)

Temporal 

Demand

 Very Low Very High

(HOW HURRIED OR RUSHED WAS THE PACE OF THE EXERCISE?)

Performance

 Failure Perfect

(HOW SUCCESSFUL WERE YOU IN ACCOMPLISHING WHAT YOU WERE ASKED TO DO?) 

Effort

 Very Low Very High

(HOW HARD DID YOU HAVE TO WORK TO ACCOMPLISH YOUR LEVEL OF PERFORMANCE?) 

Frustration

 Very Low Very High

(HOW DISCOURAGED, IRRITATED OR ANNOYED WERE YOU 

G-1
Appendix H 

Formative Evaluation Survey 

REPSS Formative Evaluation Survey Team __________ Date __________ 

1. How was the 20 minutes per week pace of the exercise for your Team

_____ Too Fast _____ About Right _____ Too Slow 

 Comments: ________________________________________________________________________

_____________________________________________________________________________________

2. The General and Team written instructions adequately described the planning exercise 

_____ True _____ False 

 Comments : _______________________________________________________________________

_____________________________________________________________________________________

3. The introduction to the data screens and tools adequately prepared me for my team tasks 

_____ True _____ False 

 Comments : _______________________________________________________________________

_____________________________________________________________________________________

4. I was able to use the workstation data screens to complete tasks in a timely manner 

_____ True _____ False 

 Comments : _______________________________________________________________________

_____________________________________________________________________________________

5. Please describe how you divided tasks within your team:

_____________________________________________________________________________________

_____________________________________________________________________________________

6. This planning exercise could be useful in command group training 

_____ True _____ False 

 Comments : _______________________________________________________________________

_____________________________________________________________________________________

7. Describe any tools that could be included to help you perform your tasks 

_____________________________________________________________________________________

_____________________________________________________________________________________

8. What worked best about this planning exercise is ___________________________________________

_____________________________________________________________________________________

_____________________________________________________________________________________

9. What I would change in this exercise is __________________________________________________

_____________________________________________________________________________________

_____________________________________________________________________________________

H-1
Appendix I 

REPSS Exercise General Instructions 

I-1
I-2
I-3
I-4
I-5
Appendix J 

REPSS Exercise Team Instructions (Supply Team) 

J-1
J-2
J-3
J-4
J-5
J-6
J-7
J-8
J-9
J-10
J-11
J-12
J-13
J-14
J-15
J-16
J-17