Innovative Prototyping and

Rigorous Experimentation (iP&rE): 

A One Week Course to Build

Culture and a Cadre

Albert Sciarretta, Steven Ramberg, Joseph Lawrence III, Andrew Gravatt,

Paulette Robinson, and Mitchell Armbruster

Center for Technology and National Security Policy 

Institute for National Strategic Studies 

National Defense University 

iii

About the Authors 

Mr. Mitch Armbruster

Mitch Armbruster is a Student Trainee (Research) at the Center for Technology and National Security 

Policy (CTNSP), Institute for National Strategic Studies (INSS), National Defense University (NDU). 

His work supports futures research and U.S. national security policy. He is a PhD candidate at the 

Catholic University of America, focusing on the security studies, alliance politics, and politics in the 

Middle East. Mitch’s dissertation examines how small members of an alliance can persuade their great 

power allies regarding threats. He holds an M.A. in politics from Catholic University and a B.A. in 

political science from the University of California, Los Angeles. 

Mr. Andrew Gravatt

Andrew Gravatt joined CTNSP as a Senior Research Fellow, where his in-depth knowledge of 

information and communication technologies is being used to conduct research in the areas of 

social media, big data, the deep web, and simulations. He began his career at NASA’s Goddard 

Space Flight Center as a systems engineer. He spent two years during the “dotcom boom” as a 

consultant managing and developing commercial web sites and applications. After serving as an 

Associate Professor at NDU’s iCollege for one year, Mr. Gravatt was appointed as the Chair of 

the Systems and Technology Department where he was responsible for the IT Project 

Management, the Enterprise Architecture, and the Chief Technology Officer curriculum. 

Dr. Joseph P. Lawrence III

Joseph P. Lawrence III, PhD, is a Distinguished Research Fellow at CTNSP on assignment from 

Pennsylvania State University via the InterGovernmental Personnel Act. At CTNSP, Dr. Lawrence 

occupies the Chief of Naval Research Chair and participates in advisory panels for the DOD, the DON, 

and the National Academies. Prior to his current position, he was a Federal Senior Executive and served 

as the Associate Technical Director and the Director of Transition at the Office of Naval Research from 

2005 to 2011. He first joined ONR in 2001 as the Director of Surveillance, Communication, and 

Electronic Combat. The greater part of his career was spent at the Naval Research Laboratory where he 

was engaged in research and then management of a large number of classified projects. 

Dr. Steven Ramberg

Steven Ramberg, PhD, is a Distinguished Research Fellow at CTNSP on assignment from Penn State 

University. At CTNSP, he provides analysis and advice on science and technology (S&T) topics and 

policies, primarily in areas of naval relevance. During his career, he served as a Fellow and as Vice 

President for Arete Associates during 2007 to 2010; as the Director of the NATO Undersea Research 

Centre (NURC) in LaSpezia, Italy from 2003 to 2007; and as Executive Director and Chief Scientist for 

ONR during 2001-2003 after joining ONR in 1988. Earlier, he worked at the Naval Research Laboratory 

where he published over 60 unclassified technical papers in the archival literature 

Dr. Paulette Robinson

Paulette Robinson, PhD is a Senior Research Fellow at CTNSP. Before joining CTNSP in 2013, she 

served as the Associate Dean for Teaching and Learning and Technology for the iCollege at NDU. Dr. 

Robinson is the leader for the Federal Consortium for Virtual Worlds, a group of over 1,600 individuals 

from government, industry, and academia who are interested in the use of virtual worlds in government. 

She is co-leading a project with the U.S. Department of Agriculture to provide secure access to virtual 
iv

worlds for government. Before joining the Federal Government, Dr. Robinson was the Assistant Director 

for Academic Support in the Office of Information Technology at the University of Maryland. 

Mr. Albert Sciarretta

Albert A. Sciarretta (U.S. Army Lieutenant Colonel, retired) is a Senior Research Fellow at CTNSP, 

where he assesses Army S&T efforts; as well as broader DOD technology and analytical needs. Mr. 

Sciarretta is also president of CNS Technologies, Inc., for which he supports DOD efforts related to 

assessing advanced military technologies, developing S&T investment strategies, and designing and 

executing tactical through operational demonstrations and experiment. His military service included 

operational assignments, instructing at the U.S. Military Academy, acting as a technology officer on 

armored vehicle task forces, and serving as Assistant to the Chief Scientist, U.S. Army Materiel 

Command. 

Disclaimer: The views expressed in this article are those of the authors and do not necessarily reflect the 

official policy or position of the National Defense University, the Department of Defense, or the U.S. 

Government. All information and sources for this paper were drawn from unclassified materials.
v 

Acknowledgments 

We were pleased to be able to contribute to the effort sponsored by the Rapid Reaction Technology 

Office (RRTO) of the Deputy Assistant Secretary of Defense for Emerging Capability and Prototyping 

(DASD[EC&P]) and were grateful for their intellectual and financial support from inception through 

fielding of the pilot courses. In testimony to the initial foresight and dedication of DASD(EC&P) and 

RRTO leadership, we offer a few selected quotes from student evaluations:

“Top down is good, but culture change, as you know, will be hard.”

“The real usefulness of the Capstone Project was the interaction between the members

[students] and getting a better understanding of different organizational cultures.”

“Application of any concept into practice is the key. S&T (6.3) transition pressure is real. 

Lack of [an acceptable] failure culture is alive and ‘not well’.” 

“The course and Capstone Project provided a repeatable framework for conceiving, 

designing and executing experiments. I thought I knew how to do that going into the 

course, but didn't. I feel much more able to do this now.” 

The authors are especially indebted to the Case Study presenters who gave of their own time and effort to 

substantially enrich the course. Their biographies are given in Appendix H. We owe a special thanks to 

Mr. Alan Brown whose Case Study was given in all three iterations of the pilot course with his unique 

knowledge and style to the acclaim of the students.

Finally, we also gratefully acknowledge and thank the Chief of Naval Research for providing and 

supporting two of the Center for Technology and National Security Policy (CTNSP) principals (Dr. 

Joseph Lawrence III and Dr. Steven Ramberg) in this effort. 

vii

Executive Summary 

This DTP contains all of the course planning and curriculum content provided in the Final Report 

delivered to the sponsor of a pilot, one week course developed at NDU for the Rapid Reaction 

Technology Office (RRTO) in the Office of the Deputy Assistant Secretary of Defense for Emerging 

Capability and Prototyping (DASD[EC&P]). The purpose for publishing this Defense and Technology 

Paper (DTP) is to make the material readily available to others who may benefit from our experience and 

wish to utilize the materials developed.

The RRTO sponsored the Prototyping and Experimentation to Improve Acquisition (PEIA) course 

development as part of its efforts to model a more agile technology organization and meet the goals of the 

Defense Department’s Better Buying Power 3.0 (BBP 3.0) initiative. The pilot course focused on these 

ends: 

(1) Experimentation with a variety of prototypes as a means to explore new capabilities in a

learning mode to inform future acquisition, but without necessarily leading to an acquisition

program of record in all instances;

(2) Achieve agile and affordable acquisition, based on the use of experimentation with

prototypes, to define more robust requirements and to achieve greater technological maturity at

the outset of formal procurement;

(3) Develop a cadre of midgrade officers and civilians familiar with prototyping, experimentation,

and critical thinking in these areas before arrival in positions of relevant responsibility within the

Department; and

(4) A growing and networked community of former students with online resources to support

their future prototyping and experimentation efforts.

This CTNSP DTP has been prepared primarily with the intent to enable future implementations of a short 

course on Innovative Prototyping and Rigorous Experimentation (iP&rE). The DTP provides an 

overview of the course, with a discussion of the educational approach, the outline for a one week 

presentation, an outline of each of the sessions in the course, and a discussion of student assessments. It 

also includes a proposed plan for the transition of the course and for its future offerings. The appendices 

include detailed information of the specific course materials as well as short biographies for the speakers.

ix

CONTENTS

I. BACKGROUND OF THE PEIA EFFORT...........................................................................................1

II. PEIA COURSE OVERVIEW................................................................................................................1

A. Overall Instructional Approach and Learning Outcomes..................................................................1

B. Assembling the Main Elements into a One Week Course.................................................................4

C. Short Descriptions of the Main Course Elements..............................................................................5

1. “Setting the Stage”.........................................................................................................................5

2. Best Practices and Lessons Learned in Rapid Technology Development.....................................6

3. Capstone Project ............................................................................................................................6

4. Definitions and Military Examples of Prototyping and Experimentation (P&E)..........................7

5. Design of Experiments...................................................................................................................8

6. Case Studies...................................................................................................................................9

7. Capstone Project Evaluation ........................................................................................................14

8. Online Community Portal............................................................................................................15

D. Student Assessment .........................................................................................................................15

III. COURSE TRANSITION PLAN .....................................................................................................16

IV. SUMMARY.....................................................................................................................................17

V. APPENDICES .....................................................................................................................................20

A. Course Management and Logistics..................................................................................................20

B. Capstone Project materials provided to the students .......................................................................23

C. Pre- and Post-Course Student Survey Questions – PEIA Course....................................................29

D. Student Survey Summary Reports (Courses #2 and #3)..................................................................32

E. Pilot Course 3 Schedule ...................................................................................................................49

F. Example of Student Capstone Project Outbrief...............................................................................51

G. Description of Course Files available online at http://ctnsp.dodlive.mil/........................................58

H. Speaker Biographies ........................................................................................................................59

I. Online PEIA Community of Information Prototype........................................................................65

1 

I. BACKGROUND OF THE PEIA EFFORT

Mr. Benjamin P. Riley, former Principal Deputy for the Deputy Assistant Secretary of Defense for 

Emerging Capability and Prototyping (DASD[EC&P]), envisioned a course to help build a cadre of DOD

military and civilian leaders familiar with the roles of innovative prototyping and rigorous 

experimentation to properly equip and prepare the U.S. military for an uncertain future. He began a 

dialog with faculty and staff of the Center for Technology and National Security Policy (CTNSP) at the 

National Defense University (NDU) to explore ways to offer a pilot course to these ends:

• Experimentation with a variety of prototypes as a means to explore new capabilities in a learning

mode to inform future acquisition, but without necessarily leading to an acquisition program of

record in all instances.

• Achieve agile and affordable acquisition based on more robust requirements and greater

technological maturity at the outset of formal procurement.

• Develop a cadre of midgrade officers and civilians familiar with prototyping and experimentation

processes and critical thinking before arrival in positions of relevant responsibility within the

Department.

• A growing and networked community of former students with online resources to support their

future prototyping and experimentation efforts.

Through this effort, the NDU/CTNSP staff fielded three iterations of a one-week pilot course while also 

searching for a permanent home for this educational offering and creating a prototype of an online 

Community of Interest (COI). 

The NDU/CTNSP staff, drawing upon diverse experiences in education, science and technology (S&T), 

acquisition, and testing, developed a pilot, one-week course in 2015 and iterated on the course in three 

offerings during 2015: February, June, and September. To exemplify teaching points, guest speakers 

were invited to discuss relevant DOD prototyping and experiment case studies. Their biographies can be 

found in Appendix H. This report summarizes the outcomes of the effort and provides documentation of 

the final course content.

II. PEIA COURSE OVERVIEW

A. Overall Instructional Approach and Learning Outcomes

The overall instructional approach was problem-based, using case studies to model approaches to 

prototyping and experimentation for the students. Thus, it was both pedagogical (i.e., lectures with 

presentations) and andragogical (i.e., case studies, student projects, and discussions) whenever we could 

make it so. This approach also gave the students vicarious experience through the experiences of others. 

The central curriculum focus of the one week course was on the Design of Experiments (DOE) using a 

variety of prototypes ranging from conceptual elements to operational elements across a range of 

experimental venues or types. The students were required to do an experimental design project 

(Capstone) within several small teams. Their findings and recommendations were briefed on the last day 

of the course to a board of senior DOD officials who would ostensibly be making programmatic decisions 

based upon the team’s output. This allowed the students to directly apply what they learned in the course 

to solving a problem. To illustrate the variety of prototypes and the ranges of experimental venues, a 

simple 3x3 matrix was developed as a mnemonic for the students as depicted pictorially in Figure 1 and in 

more detail in Figure 4. It was referred to as the 3x3 PEIA matrix and its use was required in the 

students’ briefings of their Capstone Project. 
2

Figure 1. A pictorial representation of the 3x3 PEIA matrix in Figure 4

The five Learning Outcomes (goals) established for the course are identified below in Table 1.

Table 1. Desired Student Learning Outcomes

Students will demonstrate an understanding of the:

1) Strategic importance of prototyping and of experimentation for:

a) Exploring potential new capabilities

b) Refining requirements

c) Maturing technologies

2) Approaches to conceive, design, conduct, and evaluate experiments

3) Value of appropriate risks in the various levels of prototyping and 

experimentation

4) Value of negative results (e.g., prototype failure) as an acceptable

outcome of experimentation

5) Critical importance within any acquisition Program of Record of 

stable requirements and adequately matured technologies
3

The pre-course survey also provided information about the students’ backgrounds, so better role 

assignments for the Capstone Project teams could be made. As part of the Capstone Project setup, the 

following key roles were defined as critical in any successful experimentation program. The roles were 

described to the students as:

Military User – You are the person who might use the new capability to solve your problem or to 

create an advantage.

System Buyer - You are the person who will manage the system integration and procurement of any 

new capability to meet stated requirements within cost and schedule.

R&D Supplier (researcher/technologist) – You are the person who assembled the technology and are 

its expert and (possibly naive?) advocate.

Requirements Setter – You will set the military requirements for the potential new system(s) to meet 

in procurement.

Tester and Evaluator – You will be asked to validate the results of any assessments against the 

stated requirements.

For the student Capstone projects, we assigned the students within each team to assume the role of one of 

the above expected participants in an acquisition program. Although not achieved for the pilot courses, 

Figure 2 provides notional goals for the demographics of the student target audience.

Figure 2. Notional demographics of student target audience

This breakout in Figure 2 includes a preference of somewhere between 20% and 50% of the student body 

with any acquisition training or experience. We did not enroll the desired fraction of Military O4/O5 

students and had a larger civilian and higher military rank enrollment. More details for all of the above

course information are provided in the following sections of the report and the appendices.

To support the assessment of the pilot courses, pre- and post-course questionnaires were developed and 

provided to the students. Additional discussion is in Section D, as well as Appendices C and D.

An example of a student Capstone Project presentation (outbrief) is in Appendix F.
4

B. Assembling the Main Elements into a One Week Course

It was a challenge to strike an effective balance between the need to provide the students with sufficient 

background information, tools, and multiple case studies with the practical student need for time to work 

as a team on their Capstone project. To mitigate this challenge, we used one case study (Smart Sensor 

Web) as a means to also illustrate principles of experimental design by a knowledgeable instructor and we 

chose case studies that had some synergy with the content of the Capstone Project. The resulting 

efficiencies seemed critical for all three pilot classes. With these critical efficiencies, the flow of the 

course became as follows:

Day 1 Welcome and Course Overview*

Setting the Stage**

Introduce Capstone Team Project***

Working Lunch (Teams Work on Capstone Project)

Best Practices and Lessons Learned in Rapid Technology Development

Definitions and Military Examples of Prototyping and Experimentation

Design of Experiments with Case Study

Day 2 Short Review and Discussion of Day 1 

Design of Experiments with Case Study (continued)

Working Lunch (Teams Work on Capstone Project)

Case Study 1

Day 3 Short Review and Discussion of Day 2

Case Study 2

Working Lunch (Teams Work on Capstone Project)

Teams Work on Capstone Project 

Day 4 Short Review and discussion of Day 3

Red Team Outbrief (optional) and Capstone Team Check-in

Case Study 3

Working Lunch (Teams Work on Capstone Project)

Case Study 4

Teams Work on Capstone Project

Day 5 Capstone Project Team Presentations and Feedback

Course Wrap-Up

Community of Interest Website

Student Course Feedback (post-course survey completion)

 * Including sponsor

 ** DOD acquisition terms, process and lessons

*** Description and tasking that begins with the working lunch that follows

Starting the second day, each morning began with a brief review of the previous day’s program facilitated 

by a faculty member. It was very beneficial because it allowed the students to ask questions and offer 

their observations. It often set the stage for that day’s materials. The published schedule for the third 

class can be found in Appendix E. Short descriptions of the main course elements are given in the next 

section.
5

C. Short Descriptions of the Main Course Elements

The full content for each course element is available online (see list in Appendix G). Here we provide a 

few sentences on the Objective, Approach, and any Observation(s) or Lesson(s) Learned for each main

course element and offer a Recommendation(s) to improve, as appropriate.

1. “Setting the Stage”

Objective – Provide a common framework and terminology for students about the DOD

technology development and acquisition processes. Let history and lessons-learned provide 

motivation for doing prototyping and experimentation properly and more often.

Approach – Defined DOD Budget Activities (BAs), Technology Readiness Levels (TRLs), and 

Technology Readiness Assessments (TRAs) in the context of DOD and Service stakeholders to 

evaluate the effects of oversight based on the maturity of requirements, concepts, and 

technologies. Summarized current acquisition practices; analyses of current performance, 

including data on Program Assessment and Root Cause Analysis (PARCA) of problems; and 

identify an alternative approach addressing the PARCA conclusions and based on extensive use 

of prototyping and experimentation.

Figure 3. Relationship of Budget Activities (BAs)

and Technology Readiness Levels (TRLs)

Key Observation(s)/Lesson(s) Learned – Students were not uniformly familiar with these terms or

with the common practices or recent history of both successful and unsuccessful acquisition 

programs across DOD.

Recommendation(s) - As the student participation shifts to less senior military and civilian 

participants, the lack of common terminology and an overall framework is likely to grow and this 

unit of instruction becomes even more important. It is unclear if more read ahead material would 

be useful until there is some level of common terminology among the students. 

Service	Acquisition	Executives

Explore Develop Transition

Product	Development

Technology	Development

Service	S&T	Executives

Universities,	Labs,	and	Centers

PEOs/PMs

Centers	and	Industry

~	$7B ~	$5B ~	$12B ~	$39B

BAs	1	&	2 BA	3 BA	4 BAs	5	&	7

TRL	1 TRL	6 TRL	7 TRL	8

Strategic	Discovery	of	Technology Strategic	Delivery	of	Capability
6

2. Best Practices and Lessons Learned in Rapid Technology Development

Objective – Provide real world experiences of mostly military personnel who entered the 

prototyping and experimentation framework with limited background in the topic and the nature 

of prototyping and experimentation programs managed out of the Rapid Reaction Technology 

Office (RRTO).

Approach – Used first person experiences to engage the students. The discussion focused on

personal experiences in confronting and dealing with bureaucratic, programmatic, and technical

issues.

Key Observation(s)/Lesson(s) Learned – It is useful to orient students toward future career 

challenges.

Recommendation(s) - Spend more time linking prototyping and experimentation efforts to formal 

requirements and to U.S. Combatant Command (COCOM) needs.

3. Capstone Project

Objective - Provide students with a hands-on experimental design experience. It was important to 

include some flexibility in the choice of the problem to be solved and the available prototypes to 

use and/or create. The assessment was primarily a Design of Experiment (DoE) exercise to put 

those principles to work so the students could recognize a well-designed prototyping and 

experimentation program in the future.

Approach - Created teams of five to seven members assigned to the traditional five roles of 

successful prototyping and experimentation efforts (see Section II.A.). A short description of the 

Team Project was provided ahead of the course and supplementary materials were provided upon 

arrival. All three pilot courses used the assignment in the following text box.

The Team Project: Focus on assessing benefits of pairing autonomous air and 

ground vehicles 

It is envisioned that the pairing will reduce the exposure of Warfighters in 

harm’s way while enhancing the ability of a joint force to penetrate enemy 

controlled territory to: 

• Conduct reconnaissance or surveillance missions (e.g., establish 

observation point) to gather key intelligence information,

• Provide targeting information for non-line-of-site weapons systems, or

• Quickly resupply remote forces by a means other than using manned 

systems.

Each team could choose any one of the above three “missions” or “problems” to 

address in their project. An example of one team out brief is included as 

Appendix F.
7

Key Observation(s)/Lesson(s) Learned - Commonality among Team Projects was critical for 

discussing and comparing results on the final day. Each team could see something in another 

brief that would have improved their own. The similarities with the Case Studies, particularly 

with the Smart Sensor Web, also seemed to make the teams more productive in the short time 

available. One of the more effective team briefs invented an overall framework ("campaign") for 

their project and recognized that some of the required Capstone outbrief content was not suitable 

for a brief to senior decision-makers. The team creatively developed a storyline ("limited deep 

dive") as part of their brief that was very effective while demonstrating they clearly understood 

their goal and their intended audience. As an example of a student Capstone Project presentation 

(outbrief), we have included this particular team’s presentation in Appendix F.

Recommendation(s) – Develop a catalog of several Team Projects over time for re-use. Also, to 

avoid teams of more than six students, which reduces team effectiveness, we created a "Red 

Team" which was responsible for preparing the student teams on what to expect during their 

presentation to the Senior Board on the fifth day. The Red Team was required to brief its

material to the other teams on the fourth day. The Red Team was tasked to act as staff to the 

members of the Senior Board and to provide background data and prospective questions to the 

Senior Board in advance of the next day presentations.

4. Definitions and Military Examples of Prototyping and Experimentation (P&E)

Objective – Provide the students definitions and examples of conceptual, developmental, and 

operational prototypes; as well as live, virtual, and constructive experiments. In addition, develop 

and provide the students a “take away” that would assist them in future prototyping and 

experiment efforts.

Approach – Developed a one-hour, 23-slide presentation which provided definitions and 

examples, as well as a brief discussion of the 3x3 matrix. There are many definitions for 

“prototypes” available in DOD. The goal was to have the students focus in on the three identified 

in RRTO and in the course -- conceptual, developmental, and operational prototypes. The 

instructor used examples from his extensive experience in experimentation to explain the 

differences among live, virtual, and constructive experiments.

Also, the “take away” was developed as a 3x3 matrix (see Figure 4) which conveys the 

relationships among the three prototype levels and three types of experiments.

Key Observation(s)/Lesson(s) Learned – Students often brought up the discussion of games. The 

instructor explained how games were like constructive simulations, but normally they are not 

“validated” or “accredited” sufficiently for use in constructive experiments of prototypes. They 

could be used in identifying emerging issues, but probably not more than that.

Recommendation(s) – Use the current presentation, and add a slide on games. Ensure that the 

instructor has experience in prototyping and experimentation. With time, some of the examples 

of prototypes will become outdated. They should be reviewed, and if needed, refreshed.
8

Figure 4. The 3x3 Matrix of Types of Prototypes versus Three Types of Experiments

5. Design of Experiments

Objective - Using a systems engineering approach, discuss the experimental design process from 

identifying the problem down to developing metrics, a data collection plan, and a scenario. 

Combine these with available prototypes and resources to develop an experiment plan. Discuss 

the execution of the experiment plan, analysis of results, and development of recommendations.

Approach - Used an entertaining, fictional, non-Servicecentric example (e.g., Hagar the Horrible needs to breach 

castle walls with minimal loss of men) with props (e.g., 

wooden models of siege engines) to highlight points of 

the above objective.

Useful refrain: A good or bad experiment may be equally 

costly to run. The main difference is that the good 

experiment may have a beneficial return on investment.

Key Observation(s)/Lesson(s) Learned - The use of a simple, entertaining, non-Service-centric 

example with props was very useful in conveying a difficult, but essential, amount of material. 
9

For the first pilot 

course, a more 

complex, highly 

technical and 

descriptive, example 

was used and the 

students became very 

confused. In addition, 

the first example was 

very much Armycentric, which was of 

interest to most of the 

Army personnel, but 

lacked the connection 

with personnel 

representing the other Services.

Recommendation(s) – Keep this topic in full, but always make certain that it “lives” for the 

students. Ensure the instructor has a background in both Design of Experiments and in executing 

live, virtual, and constructive experiments.

6. Case Studies

Objective – Provide the students with relevant examples of prototyping and experimentation

efforts that were well executed and had impact on DOD even if it did not lead directly to an 

acquisition Program of Record.

Approach – Selected topics and speakers who could tell the storyline while pausing to engage 

students at certain junctures of the story to ask “What would you have done, if faced with these 

circumstances?” Also ensured that at least one Case Study was related to the Capstone Project, 

giving students at least one example that is relevant and similar to their assignment.

Key Observation(s)/Lesson(s) Learned – The synergies between selected Case Studies and the 

Capstone Project allowed the teams to build their projects further and deeper than would have 

been possible in a one week course. 

Recommendation(s) – Keep some form of this synergy. The topics should include at least one 

Case Study similar to the Capstone Project. 

Available medieval siege engines

Catapult

Trebuchet

Battering	Ram

Siege	Tower

Ballista
10

Case Studies Used

a) Smart Sensor Web (SSW)

Short Description – The SSW was a 2002 experiment that was conducted to assess the 

concept that an intelligent, web-centric, distribution, and fusion of sensor information, 

provided on demand, would greatly enhance the situational awareness of dismounted infantry 

at lower echelons. A brief depiction of one of the scenarios is depicted below.

Primary Attributes for the Course – Presented by the lecturer for Design of Experiments

(DoE) who was also the Experiment Director for the SSW effort. Several synergies exist 

with the Capstone Project. This effort did not lead directly to a follow-on acquisition 

Program of Record, but shaped many follow-on developmental programs.

Key Observation(s)/Lesson(s) Learned – It was valuable to have this Case Study presented by 

the lecturer who was a key participant in the PEIA development effort and also the lecturer 

on principles of experimental design. The lecturer structured his narrative of the case around 

the material presented the first two days, especially: 1) Definitions and Examples, and 2) 

DoE.

Recommendation(s) – If a single person cannot be found to do these three tasks (definitions, 

DoE, and SSW), then strong collaboration and thorough coordination among the tasks should

be achieved by the lecturers.
11

b) Unmanned Underwater Vehicles – Mine Countermeasures 

Short Description – The Unmanned Undersea Vehicle for Mine Countermeasures (UUV

MCM) project has evolved from efforts starting in the 1990s. It provides unmanned mine 

countermeasure capabilities as a primary tool to keep critical sea lanes open.

Primary Attributes for the Course – This was a very good example of a team that adopted a 

long term "build-test-build" approach using prototyping and experimentation to incrementally 

field a revolutionary capability for a small warfare community within a small acquisition 

program context. 

Autonomy was an essential 

feature with low 

communication bandwidths 

in a cluttered environment.

Key Observation(s)/

Lesson(s) Learned – A very 

good maritime case study 

for autonomy.

Recommendation(s) – Keep 

it, or something similar, in 

the inventory as a good 

lead-in to the Capstone 

Project.

c) Have Blue

Short Description – Have Blue was the prototyping and experimentation effort directly 

leading to the U.S. Air Force (USAF) F-117 stealth attack aircraft.

Primary Attributes for the Course – Clearly a large, revolutionary prototyping and 

experimentation effort whose story can now be told for these purposes. After choosing this 

for the pilot course, the USAF highlighted it to USD(AT&L) as a prime example of how that 

Service would prefer to conduct prototyping and experimentation under Better Buying Power

(BBP) 3.0.

Key Observation(s)/Lesson(s) Learned – The speaker, Mr. Alan Brown, was captivating for 

the students because he was the knowledgeable central technical figure for the program at the

Lockheed "Skunk Works" (alias for Lockheed’s Advanced Development Projects). Many of 

the program decisions were counter intuitive and may not have been possible in the glare of 

full disclosure.

Recommendation(s) – We have recorded two of Mr. Brown's presentations for future uses in 

class or within the online community. Have Blue is an ideal example of how prototyping and 

experimentation can enable introduction of new and unconventional technologies.
12

d) Wolf Pack

Short Description – Sponsored by the RRTO, Wolf Pack was a joint experimental project that

sought to improve the combat effectiveness of small combat units. The project addressed

several capability gaps which were identified in Iraq and Afghanistan. The overall Wolf Pack 

capability embraced a variety of mounted and dismounted technologies. Collectively, these 

technologies were assigned to one of the following capability elements: 

• Situational Awareness Capability

• Communications Capability

• Electronic Warfare Capability

• Mobility Capability

• Protection Capability

• Direct Fire Capability

• Non-Lethal Capability

• Surveillance and Target Acquisition (STA) Capability

• Counter Improvised Explosive Device (CIED) Capability

• Biometric Capability
13

The Project’s developmental 

approach was to assess the military 

utility of emerging but relatively 

mature military technologies through 

a program of theoretical war games 

and a series of practical, live

experimentation events.

Primary Attributes for the Course –

The original "testimonial" by a 

warfighter who was thrust into an 

experimental task by Vice Admiral

(retired) Arthur K. Cebrowski at the 

Office of Force Transformation 

several years ago with little experience or preparation. His testimonial focused on how he 

dealt with bureaucratic issues to employ potentially game-changing capabilities.

Key Observation(s)/Lesson(s) Learned – The "testimonial" proved very effective and was 

continued as its own element of the course. However, as originally presented, there was little 

discussion of prototyping and experimentation.

Recommendation(s) – From a prototyping and experimentation perspective, this is not 

suitable as a prototyping and experimentation Case Study, per se. Some “inside the 

Pentagon” motivational discussion however appears warranted.

e) Ground Operation Robotics

Short Description – There were two presentations. One focused on basic research (BA 1) 

level efforts – i.e., conceptual prototypes. The second focused on applied research (BA 2) 

and advanced technology development (BA 3) level efforts – i.e., developmental prototypes. 

The presentations addressed: 1) multiple capabilities integrated onto testbed platforms and 

evaluated using structured experimentation in relevant environments; 2) natural language 

communication via speech, gesture, and text; and 3) autonomous search and grasp of an 

initially unseen object.

Primary Attributes for the Course – This is earlier in the developmental cycle than other Case 

Studies and offers the students a bit of an "over the technical horizon" view for their 

Capstone Project.

Key Observation(s)/Lesson(s) Learned – Both instructors provided excellent perspectives on 

S&T level efforts. The planning and executing of experiments for conceptual prototypes was 

very challenging. Student awareness of the value of experimentation with conceptual 

prototypes was made clearer with these paired case studies.
14

Recommendation(s) –

This Case Study served 

to illustrate longer term 

S&T efforts leading to 

potential prototyping 

and experimentation 

efforts later in the S&T 

pipeline.

f) Electromagnetic Rail Gun

Short Description – This is a current example of a high payoff, near-term capability with 

large technical challenges being addressed over many years in S&T. It is now approaching a 

level of feasibility that prototyping and experimentation efforts are underway with 

operational (near-production) prototypes.

Primary Attributes for the Course – Illustrates a timely, careful, developmental approach 

toward an uncertain military capability having high payoff potential. The guest speaker 

stopped several times during the presentation to provoke student discussion at a decision 

point where there were multiple prospective paths for moving forward with the effort. The 

students enjoyed the discussion about selecting a way forward. After the discussion, the 

speaker would let the students know which was selected for the actual program.

Key Observation(s)/Lesson(s) Learned –

This presentation included sensitive 

information (Distribution D) which could 

not be released for public dissemination. 

The paper could not be posted on the 

online COI site, and is not included in the 

publicly releasable DVD for this course.

Recommendation(s) – Include this as a 

Case Study.

7. Capstone Project Evaluation

Objective – To have the students identify in their own words a compelling argument for 

performance of prototyping and experimentation in advance of formal acquisition; and to give the 

students an experience briefing senior DOD decision-makers with their Capstone Project results.

Approach – Have a three to four person Senior Board of instructors and DOD executive-level 

decision makers be briefed by the students who are then attempting to gain buy-in from the 
15

leaders in the development of a prototyping and experimentation effort as a predecessor to 

initiation of formal acquisition program.

Key Observation(s)/Lesson(s) Learned - Difficult to obtain and hold a commitment of executive 

level decision makers for an entire half day.

Recommendation(s) - Schedule well in advance and use online meeting technology to enhance

participation by these decision makers.

8. Online Community Portal

Objective - Provide a resource during and after the course for the students to retrieve materials 

and to establish contacts among themselves over their careers. Additional discussion can be 

found in Appendix I and various online screen shots (like the one immediately below).

Approach - Since this was not a major element of the PEIA funded effort and in the long term 

would not be housed at NDU, we opted for a simple Google site implementation as a means of at 

least identifying a prospective 

approach to a community portal.

Key Observation(s)/Lesson(s) 

Learned - Access could be 

problematic from government 

computers based on their firewall 

restrictions. Unless hosted on a 

secure, encrypted, network, only

Distribution List A materials can 

be posted with the current 

approach.

Recommendation(s) – Improve 

access before, during, and after 

the course at a site that allows up 

to For Official Use Only (FOUO) and sensitive material.

D. Student Assessment

As mentioned earlier, the students completed a short pre-course survey and a more extensive post-course 

survey so that we could make better Capstone Project team assignments and also so that we could 

qualitatively measure progress against the established Learning Outcomes that we sought. In general, 

these surveys did not change for the three pilots, although we added a few items after the first pilot. The 

surveys are contained in Appendix C.
16

Students quickly learned that the instructional team was receptive to feedback to improve the course and 

many excellent suggestions were made and incorporated as we iterated on the course offerings. These 

student suggestions and observations for the last two pilot courses are summarized in Appendix D.

Appendix D (Student Survey Summary Reports [Courses #2 and #3]) also provides brief assessments of 

the survey and the student responses in terms of “Observations” and “Conclusions” made by the course 

instructors. For example, the first observation in Appendix D is:

• The responses are listed above in order of highest to lowest as determined in Class #2. The 

composition of the Student groups varied significantly from Class #1 to Classes #2 and #3 with 

more senior and more widely experienced students in Class #1 (a deliberate sponsor choice to 

iterate early on the course content and structure). Rank order of Class #2 and #3 experience is 

similar although Class #3 indicated higher levels across the board which may account for some of 

the later minor survey differences between Class #2 and #3 responses.

The corresponding conclusion immediately below this observation is:

• While useful data to monitor the class and to construct Capstone Project teams, the value of these 

survey questions is largely instructional for the students to think in terms of the five essential 

roles and the differing perspectives that are necessary for prototyping and experimentation

success. 

Additional observations and conclusions in Appendix D provide some great insights. They should all be 

considered in planning future courses. Here are a couple more examples of conclusions:

• The students gained a greater appreciation of the context and culture of DOD acquisition. The 

survey items are useful and the "before—after" comparisons are valuable, more so if they are 

available to the students and perhaps the Community-of-Interest over time.

• Generally high marks for the course and an essential question for a student survey including the 

opportunity to elaborate on both high and low inputs. These helped us tune materials between 

Class #2 and Class #3.

III. COURSE TRANSITION PLAN

At the outset, there was no expectation that a course of this character would become part of the 

curriculum anywhere at NDU. This evolved slightly with interest from the NDU iCollege and with the 

notion that some form of the course might be offered as an NDU elective should faculty and student 

interest warrant. The transition organizations that have been considered are the Defense Acquisition 

University (DAU), the Naval Postgraduate School (NPS), and perhaps the Air Force Institute of 

Technology (AFIT). We have pursued a collaboration with DAU by soliciting attendance by DAU 

faculty and staff in the last two pilot courses and by seeking feedback from the DAU attendees. This was 

helpful in the course development and iteration. Several meetings have been held subsequent to the last 

course offering in an attempt to better clarify a transition path from NDU to DAU. These discussions 

have involved DAU, NDU, and offices within the OUSD AT&L having responsibilities for prototype 

development and for the training requirements of the acquisition workforce within DOD. A 

representative of the NPS attended the first pilot course, but there has been no official interest in 

transitioning the course there.
17

As of the date of publication, the Office of the Deputy Assistant Secretary of the Navy – Research, 

Development, Test and Evaluation (ODASN RDT&E) had identified funding for a reprise of the course in

a three-day format. The plan is to conduct three such classes for the Naval Science and Engineering 

workforce starting in FY16 and extending into early FY17.

In addition, the DAU has been converting the PEIA presentations into five to six presentation modules for 

an online course. The DAU point of contact for this effort is Dr. Martin Falk, Program Learning Director 

for Science and Technology Management.

IV. SUMMARY

In summary, we collect a number of the key observations and recommendations provided in the body of 

the report: 

“Setting the Stage”

Key Observation(s)/Lesson(s) Learned – Students were not uniformly familiar with many terms 

or with the common practices or recent history of both successful and unsuccessful acquisition 

programs across DOD.

Recommendation(s) - As the student participation shifts to less senior military and civilian 

participants, the lack of common terminology and an overall framework is likely to grow and this 

unit of instruction becomes even more important. It is unclear if more read ahead material would 

be useful until there is some level of common terminology among the students.

Best Practices and Lessons Learned in Rapid Technology Development

Key Observation(s)/Lesson(s) Learned - Useful to orient students toward future career challenges.

Recommendation(s) - Spend more time linking prototyping and experimentation efforts to formal 

requirements and to COCOM needs.

Student Capstone Project

Key Observation(s)/Lesson(s) Learned - Commonality among Capstone Projects was critical for 

discussing and comparing results on the final day. Each Capstone Project team could see 

something in another brief that would have improved their own. Similarities with the Case 

Studies also seemed to make the teams more productive in the short time available. An example 

of a student Capstone Project presentation (outbrief) is given in Appendix F.

Recommendation(s) – Develop a catalog of several Capstone Projects over time for re-use. Also, 

to avoid teams of more than six students, which reduces team effectiveness, we created a "Red 

Team" which was responsible for preparing the student teams on what to expect during their 

presentation to the Senior Board on the fifth day. The Red Team was required to brief its 

material to the other teams on the fourth day. The Red Team was tasked to act as staff to the 

members of the Senior Board and to provide background data and prospective questions to the 

Senior Board in advance of the next day presentations.

Design of Experiments

Key Observation(s)/Lesson(s) Learned - The use of a simple, entertaining, non-Service-centric 

example with props was very useful in conveying a difficult, but essential, amount of material.
18

Recommendation(s) – Keep this topic in full, but always make certain that it “lives” for the 

students. Ensure the instructor has a background in both Design of Experiments and in executing 

live, virtual, and constructive experiments.

Case Studies

Key Observation(s)/Lesson(s) Learned – The synergies between selected Case Studies and the 

Capstone Project allowed the teams to build their projects further and deeper than would have 

been possible in a one week course.

Recommendation(s) – Keep some form of this synergy.

Observations, lessons-learned, and recommendations for the individual Case Studies are given in 

that section of the report.

Key “Take Away”

A key "take away" for the students was the following 3x3 matrix:

This matrix was termed the "PEIA Matrix" and is in the curriculum materials available online. A pictorial 

version of it is in Figure 1 of this report.

Positive Student Assessment
19

To qualitatively measure progress against the five Learning Outcomes (goals) of Table 1, the students 

completed a short pre-course survey and a longer post-course survey. Based on those surveys, we have 

assessed that our success in meeting the goals was very good across the board; however, we forced our 

self-evaluation into a tertile ranking (by thirds, Top, Middle, and Bottom) as shown in the second column 

in the Table below, with relevant student response items (identified in Appendix D) listed in the last 

column. We note again that student assessments were, on average, very good and our assessment of 

“Bottom” is therefore relative on the tertile scale and is provided to suggest areas that might be improved.

Table - Student Assessments of Learning Outcomes

Students will demonstrate an understanding of the:

Approximate

Evaluations in 

Tertiles

Relevant 

Student Survey 

Items

1) Strategic importance of prototyping and of 

experimentation for:

a) Exploring potential new capabilities, T Q2/7, Q6, Q13

b) Refining requirements, B Q6, Q13

c) Maturing technologies, M Q2/7, Q6, Q13

2) Approaches to conceive, design, conduct and 

evaluate experiments, T Q10, Q12

3) Value of appropriate risks in the various levels of 

prototyping and experimentation, M Q10, Q13

4) Value of negative results (e.g., prototype failure) as 

an acceptable outcome of experimentation, and B Q2/7, Q10

5) Critical importance within any acquisition Program 

of Record of stable requirements and adequately 

matured technologies.

T Q4/8, Q10, 

Q11
20

V. APPENDICES

A. Course Management and Logistics

The NDU Project Manager (PM) for the Prototyping and Experimentation for Improved Acquisition 

(PEIA) course was Dr. Paulette Robinson. The management and curriculum contributions for the course 

was team-based and included Dr. Joseph Lawrence, Dr. Steven Ramberg, Mr. Albert Sciarretta, and Mr. 

Andrew Gravatt. Dr. Robinson produced and provided monthly reports to Ms. Paula Trimble, the PM at 

AT&L/RRTO. The project was concluded with delivery of this final report to Ms. Trimble in November 

2015 along with a DVD of publicly releasable course materials and documents. The PEIA DVD was 

organized by Dr. Robinson with assistance from Mr. Sciarretta.

The project produced three one-week pilot courses in February 2015, June 2015, and September 2015. 

Curriculum was developed from June 2014 to August 2015. The overall instructional approach was

problem-based, using case studies to model approaches to prototyping and experimentation for the 

students. Thus, it was both pedagogical (i.e., lectures with presentations) and andragogical (i.e., case 

studies, student projects, and discussions) whenever we could make it so. Case studies for the course 

were selected by the team. Three were used in pilot courses one and two. Four were used in the third

pilot course. Changes in the curriculum occurred between the courses. Complete curriculum reviews 

were conducted two weeks before each of the pilot courses. All courses were held at NDU.

Outreach to potential students was shared by RRTO and the instructional team. Student registration and 

administration was managed by the PM, PEIA. Information collected on students included name, rank, 

organization, phone number, and email. Students were sent two emails. The first email was a verification 

of registration and the second was a logistics email that included logistical information and a map of 

NDU. Student classroom curriculum materials were organized by the PM, PEIA. The iCollege copied 

the materials and created the student binders.

Students completed a pre-course self-evaluation that assisted Dr. Ramberg in placing students into diverse 

teams. They also completed a post-course self-evaluation to provide feedback on the course. Data 

analysis was conducted by Dr. Ramberg.

The Community of Interest (COI) prototype was created by the NDU iCollege and Dr. Robinson. All 

materials were uploaded and organized by the Dr. Robinson. Included on the website was the curriculum 

background materials, student contact lists, maps, course schedule, case study slides, additional resources, 

and capstone materials. See additional discussion in Appendix I.

To assist the team in preparing course material; to support course execution, administration, and logistics; 

and to coordinate transition; course task lists were developed. The following is an example of the course 

task list for pilot course #3.

Task Person Backup Due Date

Students (Sept 14-18 Course)

Registration verification email

Pre-course self-evaluation and logistical email

Post-course self-evaluation forms

Pre survey data to Steve

Conduct post-course self-evaluation

Post-course self-evaluation data input
21

Task Person Backup Due Date

Post-course self-evaluation data analysis

Student list for sign in – U drive PEIA Course

Student certificates of completion (file in U drive)

Create list in order of curriculum materials 

Completed curriculum to iCollege

Completed course binders (44-students and 6 faculty)

Create Capstone Teams

Print student presentation drafts 

COI review 

Red team project (team 5)

Print IT user agreement forms 

Set Up (Sept 14-18 Course)

Name placards

Coffee, cups, creamer, sugar, tablecloths, trash can, 

etc.

Set up room (LH 1108, 1106, 1105, 1104)

Coffee pot (verify 9/11/15

Bottled Water

Coffee grounds (2 large)

Make Coffee each morning of course

PEIA Course/Sept Class Folder U Drive

Graphic for PEIA Poster

Sign graphic and request sent to graphics

Signs completed 

Signs directing students to the classroom 

Technology (Sept 14-18 Course)

Computers (5) for breakout room with Wi-Fi access

Student List for network access (on U Drive)

Student Wi-Fi___33 access list to Gravatt

Student Wi-Fi___33 access 

Presentation computer and PPT slides loaded 

Presentation screens and set-up

DVDs with student out brief template (4)

Red Team Materials 

Curriculum Materials Placed into COI 

Students placed in COI 

COI ownership transfer to Joe 

August Monthly Report 

Final Report Draft

Final Report 

THE PEIA REPORT

What was needed and why

Description of NDU/CTNSP Effort/Tasks

Project Overview
22

Task Person Backup Due Date

Course Management

Overall Pedagogical Approach and Learning 

Outcomes

Curriculum Summary

Setting the Stage

Funding Projects and Lessons Learned (Ermer)

Capstone Project

Definitions and Military Examples

Design of Experiments

Case Studies

Smart Sensor Web

UUV

Have Blue

Wolfpack

ARL Robotics

Electromagnetic Rail Gun 

Student Assessment

Project Evaluation 

Online Community Prototype

Logistics

Project Start and End Dates 

Deliverables 

Pilot Course Testing 

PROJECT TRANSITION PLAN

BUDGET SUMMARY

DIGITAL MATERIALS (DVD)

Project Transition 

DAU

Transition Follow-up

Transition Briefings 

Earl Wyatt

DAU Leadership
23

B. Capstone Project materials provided to the students

Capstone Project Description - You are a member of a team that has been assigned to assess the benefits 

of technology components of a potential development acquisition program. You will be briefing a senior 

Flag/SES Board to gain approval for your prototyping and experimentation plan. The information 

generated by the execution of your plan will be used by senior decision makers in the future to better 

assess technology maturity, understand potential warfighting benefits, and determine a path forward. The 

potential program will pair unmanned air and ground systems that are fully coordinated and mostly 

autonomous. It is envisioned that the pairing will reduce putting warfighters in harm’s way while 

enhancing the ability of a joint force to penetrate enemy controlled territory to: 

• Conduct reconnaissance or surveillance (e.g., establish observation point) missions to gather key 

intelligence information,

• Provide targeting information for non-line-of-site weapons systems, or

• Quickly resupply remote forces by other means.

Individually, both unmanned systems may be able to perform these tasks. However, they each have 

limitations in accomplishing them. Terrain and enemy actions normally introduce limitations on the 

ground vehicle’s situational awareness, movement, and line-of-sight surveillance/targeting capabilities. It 

is believed that the pairing will enable the ground component to have better situational awareness and to 

maneuver more rapidly on the battlefield, avoiding difficult terrain, obstacles, and even threat forces. For 

the unmanned air system, limited time-on-station and lack of stealth may be its primary limiting factors. 

It needs to be determined if the pairing’s synergistic warfighting capabilities (e.g., shared situational 

awareness, enhanced ground maneuver) provide significantly greater military benefits (e.g., 

reconnaissance, surveillance, targeting, resupply) to the joint force than either one can provide 

individually.

The Team must select from the above desired capabilities to establish goals and objectives for the 

Prototyping and Experimentation (P&E) Plan.

The experimentation assets available for you to use (or not) are list in Table 1.

Table 1. Description of Available Assets

Assets Available for 

Prototyping and 

Experimentation

Short Description

UAV Level 5 autonomy available now at TRL 9 (live, virtual, and 

computer-generated systems available now)

Level 6 available in five years at TRL 6 (virtual and computergenerated systems available now)

UGV Level 4 autonomy available now at TRL 9 (live, virtual, and 

computer-generated systems available now)

Level 5 available in five years at TRL 6 (virtual and computergenerated systems available now)

Level 6 available in ten years at TRL 3 (computer-generated systems 

available now)

UAV-UGV Pairing System Pairing software is at TRL 5. Interaction communications between 

the UAV-UGV are at Level 4 autonomy –human 

intervention/direction is needed for the pairing system to operate as 

envisioned.
24

Assets Available for 

Prototyping and 

Experimentation

Short Description

Warfighter-System Interface The Warfighter-System Interface (WSI) for the integrated UAV-UGV 

pair is currently at TRL 5 – mostly on a computer-based system, as 

opposed to common handheld controller. Envisioned UAV-UGV 

WSI cannot operate with a single human controller – independent 

sub-controllers are needed for the UAV and the UGV. All system 

controllers using the WSI must be in the same room to support verbal 

interactive communications among themselves while controlling the 

UAV-UGV pairing system, as envisioned.

NOTE: Current levels of autonomy and TRLs require the pairing to 

function – as envisioned – with significant “human intervention” by 

warfighters and civilian engineers. With Level 6 autonomy, the 

paired system will require little to no human intervention and will 

respond appropriately to the “command and control” of warfighters. 

Communications Links 

(General)

Technically available at various bandwidth rates, but may be 

constrained by potential interference with civilian and military 

systems

Live Force Live personnel to play roles of C2 (including use of WSI), 

intelligence personnel, and limited number of operators of indirect 

fire systems (manned aircraft systems and other non-line-of-sight 

weapon systems)

Fielded systems which may be 

available as live, virtual, and 

computer generated systems

Non-line-of-sight weapons (tube/missile artillery and air-to-ground 

missiles with precision guided munitions)

Manned aircraft systems (fixed wing and rotary) 

C4ISR systems

[NOTE: safety and environmental constraints may limit use of live 

weapons and manned aircraft systems – ranges have limited bombing 

capabilities]

Capstone Team Tasks

The	team	will	deliver	to	a	Flag-level/SES	Board	a	briefing	to	gain	approval	and	funding	for	a	prototyping	

and	experimentation	approach	to	assess	the	military	benefits	of	the	envisaged	capability.		The	team	

needs	to:

• Convince the Board that the approach will prove the value of the envisaged capability, refine 

requirements, and determine the next steps (e.g., disregard the technology and capability, shelve 

the technology for future consideration, pursue further development, or pursue a program of 

record).

• Convince the Board that they understand what needs to be done for prototyping and 

experimentation.

o Address the alternative types of prototyping and make the case to the Board that this 

approach is the smart one.

o Be ready to describe and discuss a representative portion of the experimentation planning 

to show the Board that the team understands the Design of Experiment process and what 

needs to be done.

• Show the Board that after the prototyping and experimentation is completed, the team should be 

in a position to take the next step (as described above).
25

• Show the Board that the team has developed a plan to address the risks that need to be addressed.

The briefing should address at least the following:

Capability Goals. Derived from the envisioned Capstone Project Description above.

Possible Prototyping and Experimentation (P&E) Program Development Approaches. Discuss 

options for moving forward – e.g., program using current technologies, program using more mature 

technologies, refining requirements. If the technologies are mature enough and the needs/requirements 

are settled then an acquisition program could be started with issuance of a request for proposal (RFP) to 

be funded with BA5 funds. If the technologies are not mature enough and/or the needs/requirements are 

not settled then a prototype and experimentation effort may be warranted using BA3-4 funds before 

“graduating” to a BA5-funded program. Technologies could also be merely assessed using prototyping 

and experimentation and then be put “on the shelf” until needed at a future date. Identify the 

development approach chosen and, presuming that an initial prototype and experimentation phase (BA3-

4) is recommended, justify the potential additional time and costs as compared to a straightforward 

acquisition program award (BA5) to a prime contractor.

Prototyping based on Technology Readiness and Capability Requirements. Based on the readiness 

of the technology as well as the stability of the capability needs/requirements, discuss which prototypes 

would be considered for assessment.

Experiment Objectives Experimentation objectives should support Program Goals previously identified. 

(In the interest of time, two are provided. The Project Team should expand on these and develop at least 

one more.)

• Objective #1 – Determine if current levels of autonomy warrant moving forward with the 

development acquisition program.

• Objective #2 – Determine if pairing these platforms provides capabilities which are significantly 

more beneficial than operating them independently.

• Objective #3 – <<TBD>>

• Objective #4 – <<TBD>> (optional)

Take no more than two of the objectives and determine the following:

• Identify Sub-Objectives (optional) (if applicable, no more than four)

For Instructor Use Only

Examples of additional objectives:

– Determine shortcomings in warfighting capabilities, including performance and effectiveness, of this 

pairing approach and identify potential mitigation strategies.

– Determine warfighter-system integration needs

– Determine gaps in needed technical capabilities and possible technical solutions 
26

• Using the most critical objectives/sub-objectives, identify four Essential Elements of Analysis 

(EEA)s 

• Using the most critical EEAs, identify four Hypotheses

• For two of the hypotheses, identify metrics for proving/disproving each one. Identify at least four 

measures of performance (MOPs) and two measures of effectiveness (MOEs) for each 

hypothesis.

• Determine two overall measures of value (MOVs) for assessing Operational Utility.

• Develop a Data Collection Plan

• Develop a Scenario which supports the Data Collection Plan

• Identify which live, virtual, and computer generated systems (especially operational, 

developmental, and/or conceptual prototypes) will be used in the scenario

• Identify potential constraints (e.g., availability of air space, safety issues, and environmental

issues) for executing the experiment.

Risks. Discuss any risks which should be considered during the prototyping and experimentation 

process.

For Instructor Use Only

Examples of sub-objectives:

– Determine impact of levels of autonomy on individual and collective tasks

– Determine when the UAV provides the UGV the most benefit in pairing

– Determine when the pairing has no benefit

– Determine when it is effective for the UAV to participate in the ISR, targeting, and logistics 

missions

– Determine what aspects of the UAV information cannot be replicated by the UGV or any 

other system

– Determine which system provides the best targeting information

– Determine if pairing should be more than one air and one ground component

– Determine minimum bandwidth rate for pairing links

– Determine shortcomings, if any, in communications links between unmanned systems and 

C2 and intelligence personnel
27

Appendix B, Annex A. Levels of Autonomy

Note: This slide (#6) is taken from open source presentation by Andy Williams, NATO Allied Command 

Transformation Operational Analysis Branch, at NATO sponsored Autonomous Systems 21-22 May 

2014 Online Workshop. See: 

http://innovationhub-act.org/drupal/node/533 and

http://innovationhub-act.org/drupal/sites/default/files/u4/AxS_Innovation_Hub_AutonomyBenefits.pdf

Appendix B, Annex B: Spec Sheets and related references for the assets available in Table 1

If you are looking for more details on future capabilities for your prototyping and experimentation 

systems, you may chose exemplars of such systems as you fill out the table of options from your 

Capstone Project Description. Here are some links that you may also find useful to these ends:

http://www.fas.org/irp/program/collect/uav_roadmap2005.pdf

http://en.wikipedia.org/wiki/Unmanned_aerial_vehicle

http://en.wikipedia.org/wiki/Unmanned_ground_vehicle

http://www.militaryaerospace.com/articles/print/volume-25/issue-12/unmanned-vehicles/canadianmilitary-chooses-irobot-ugvs-to-detect-chemical-agents-explosives-and-radiation.html
28

http://www.militaryaerospace.com/articles/2011/08/qinetiq-develops-tracked.html

http://aviationweek.com/military-government/unmanned-k-max-operates-unmanned-ground-vehicle

http://www.fas.org/irp/program/collect/uav_roadmap2005.pdf

http://nextbigfuture.com/2015/01/us-army-future-vertical-lift-helicopter.html?m=1

Appendix B, Annex C: Outbrief Requirements and Formats

A list of the “minimum required” slides were developed for each team to use as a starting point. 
29

C. Pre- and Post-Course Student Survey Questions – PEIA Course

Pre-Course	Self	Evaluation:

1) On	a	scale	of	1-5,	please	indicate	your	experience	or	knowledge	as	of	today about	the	following:

								(1=None,	2=A	Little,	3=Some	Familiarity,	4=Working Knowledge,	5=Held	a	Related	Assignment)

a) ___Setting	requirements	for	new	military	systems	or	capabilities

b) ___Technology	Readiness	Levels	(TRLs)

c) ___Science	and Technology	or	R&D	activities

d) ___Program	management	for	system	acquisition

e) ___System	demonstration or	validation	testing

f) ___Experimentation	using	prototypes

2) Please	indicate	if	the	following	are	true	(“T”)	or	false	(“F”)	statements	in	your	view	today:

a) ___Unanticipated	outcomes	of	a	demonstration	are	usually	unwelcome

b) ___System	requirements	should	not	change	once	a	system	acquisition	is	underway

c) ___The	only	possible	failure	for	an	experiment	is	a	result	of	a	poor	experimental	design

d) ___Most	DOD	acquisition	programs	do	not	conduct	prototype	development	and	

experimentation

e) ___Most	major	DOD	acquisition	programs	of	the	last	2	decades	have	exceeded	significant	

(Nunn/McCurdy)	limits	for	cost/schedule	growth	set	by	Congress

As	a	member	of	a	course	Capstone	Project	team,	you	will	be	required	to	outline	approaches	using	

experimentation	with	prototypes.		This	could	lead	to	a	new	acquisition	program	with	unmanned	

air/ground	components	that	are	fully	coordinated	and	mostly	autonomous	for	the	military	user(s).		The	

desired	capability	is	coordinated	operations	that	improve	situational	awareness	for	ground	forces	and	

perhaps	also	provide	capabilities	for	targeting	information	to	remote	weapons	systems	and	ISR	for	

others.

3) How	do	you	see	your	role(s)	in	such	a	team	project?		[mark	with	an	“X”	all	that	might	apply	for	

you]:

a) ___Setting	requirements

b) ___Determining	system utility	for	operating	units	(i.e., military	user)

c) ___Developing	prototypes	(i.e., R&D	supplier)

d) ___Acquisition	Program	Management	(i.e., system	buyer)

e) ___Designing	a	demonstration/test/experimentation	program	for	a	prototype	(i.e.,

evaluator/tester)

f)	Of	the	above	5	roles,	which	one	best	summarizes	your	overall	expertise?	_____________.

4) Which	of	the	following	do	you	expect	might	be	the	top	4	primary risk	areas	to	the	successful	

delivery	of	a	proposed	new	capability	within	cost	and	schedule?		[Select	only	4	areas]

a) ___Immature	technologies

b) ___Unstable	requirements

c) ___Lack	of	user	feedback

d) ___Budget	uncertainty

e) ___Congressional	actions

f) ___Overselling	by	vendor(s)

g) ___Other	(please	explain)	________________________________________________

5) In anticipation of the actual course, what are your overall goals and expectations for the course? 

[Short description].
30

Post-Course	Evaluation:

6) Please	indicate	by	a	check,	the	areas	below	where	you	gained significant	and	useful	knowledge	

or	insight	from	the	Course:

a) ___Setting	requirements	for	new	military	systems	or	capabilities

b) ___Technology	Readiness	Levels	(TRLs)

c) ___Science	and	technology	(S&T)	or	research	and	development	(R&D)	activities

d) ___Program	management	for	system	acquisition

e) ___System	demonstration	or	validation	testing

f) ___Experimentation	using	prototypes	often	of	widely	varying	types/maturity

g) ___Importance	of	diverse	expertise	and	experience	in	teams

7) Please	indicate	if	the	following	are	true	(“T”)	or	false	(“F”)	statements	(Yes,	you	did	this	before	

the	course):

a) ___Unanticipated	outcomes	of	a	demonstration	are	usually	unwelcome

b) ___System	requirements	should	not	change	once	a	system	acquisition	is	underway

c) ___The	only	possible	failure	for	an	experiment	is	a	result	of	a	poor	experiment	design

d) ___Most	DOD	acquisition	programs	do	not	conduct	prototype	development	and	

experimentation

e) ___Most	major	DOD	acquisition	programs	of	the	last	2	decades	have	exceeded	limits	for	

cost/schedule	growth	set	by	Congress

8) Which	of	the	following	do	you	expect	might	be	the	top	4	primary risk	areas	to	the	successful	

delivery	of	the	new	capability	from	your	Capstone	Project?	[Select	only	4	areas,	mark	with	an	

“X”]

a) ___Immature	technologies

b) ___Unstable	requirements

c) ___Lack	of	user	feedback

d) ___Budget	uncertainty

e) ___Congressional	actions

f) ___Overselling	by	vendor(s)

g) ___Other	(please	explain)__________________________________________________

9)		Please	indicate	on	a	scale	of	1-5	(5=High)	how	well	the	course	achieved	the	following	items	and

how	important	each	item	was	to	you (two	scores	for	each	item):	

Item Achieved

(1-5)

Importance

(1-5)

a) Met	your	personal	goals	and	expectations	going	into	the	

course

b) The	extent	to	which	the	stated	course	objectives	were	

clear

c) The	extent	to	which	the	stated	course	objectives	were	

accomplished

d) The	utility	of	text/presentation	material	and	study	notes

e) The	quality	of	the	interactions	among	students

f) The	quality	of	the	interactions	between	the	students	and	

the	instructors
31

g) The	quality	and	topics	of	classroom	instruction	and	

activities

h) The	quality	and	utility	of	the	online	portal	for	the	course

i) Please	elaborate	immediately	below	on	very	high	or	very	low	numbers	above	(additional	

space	is	provided	at	the	end	of	this	survey):

9) Please	rate	on	a	scale	of	1-5	(5=High)	the	following	course	attributes:

a) ____		Depth	of	material	presented

b) ____		Usefulness	of	the	case	studies

c) ____		Usefulness	of	the	Capstone	Project

d) ____		Pace	of	the	course

e) ____		Relevance	of	the	course	to	real	world	problems

f) ____		Value	of	this	course	to	you

g) ____		Your	overall	rating	of	the	course

h) ____		Other	(please	write	out	your	criterion):__________________________________	

i) Please	elaborate	immediately	below	on	very	high	or	very	low	numbers	(additional	space	is	

provided	at	the	end	of	this	survey):

10) Please	indicate	on	a	scale	of	1-5	(5=High)	how	well	the	Capstone	Project	has	provided	you	with	

an	appreciation	of	the	need,	prior	to	initiation	of	an	acquisition	Program	of	Record,	for:

a) ___		Stable	requirements

b) ___		Mature	technologies

c) ___		User	buy-in

d) ___		Prior	prototyping	and	experimentation

e) Please	elaborate	immediately	below	on	very	high	or	very	low	numbers	(additional	space	is	

provided	at	the	end	of	this	survey):

11) Please	indicate	on	a	scale	of	1-5	(5=High)	how	well	the	Capstone	Project	has	provided	you	with	

approaches	to:

a) ___		Conceive	an	experiment

b) ___		Design	an	experiment

c) ___		Conduct	an	experiment

d) ___		Evaluate	results	from	experiments

e) Please	elaborate	immediately	below	on	very	high	or	very	low	numbers	(additional	space	is	

provided	at	the	end	of	this	survey):

12) Please	indicate	on	a	scale	of	1-5	(5=High)	how	well	the	course	demonstrated	the	strategic	

importance	of	prototyping	and	of	experimentation	for:	

a) ___		Exploring	potential	new	capabilities

b) ___		Refining	requirements

c) ___		Maturing	technologies

d) ___		Appreciating	the	value	in	appropriate	risk	in	prototypes	and	possible	failure	in	

experiments

e) ___		Improved	system	acquisition

13) Please	elaborate	immediately	below	on	very	high	or	very	low	numbers	(additional	space	is	

provided	at	the	end	of	this	survey):

14) Please	share	with	us	how	you	would	improve	the	course	in	the	space	below.
32

D. Student Survey Summary Reports (Courses #2 and #3)

Class #1 = Students were given an incorrect post-Course survey, used limited pre-course data for 11

Class #2 = 19 pre-Course responses and 20 post-Course responses, average of all 1-5 evaluations = 4.15

Class #3 = 27 pre-Course responses and 26 post-Course responses, average of all 1-5 evaluations = 4.11

Q1- Student reported prior experience or knowledge 

(on a scale 1-5) Class #1 Class #2 Class #3

S&T or R&D activities 3.36 3.68 4.11

Technology Readiness Levels 4.00 3.42 3.72

Program management for system acquisition 4.45 3.21 3.67

System demonstration or validation testing 3.73 3.21 3.78

Setting requirements for new systems/capabilities 4.00 3.05 3.33

Experimentation using prototypes 3.91 2.68 3.52

Q3-Student willingness to serve in 1 of 5 Team Roles (%) Class #1 Class #2 Class #3

Designing a P&E program 100% 68% 52%

Determining System Utility 36% 58% 48%

Setting Requirements 45% 47% 59%

Developing Prototypes 73% 32% 56%

Acquisition PM 64% 32% 48%

Observation: The responses are listed above in order of highest to lowest as determined in Class #2. 

The composition of the Student groups varied significantly from Class #1 to Classes #2 and #3 with more 

senior and more widely experienced students in Class #1 (a deliberate sponsor choice to iterate early on 

the course content and structure). Rank order of Class #2 and #3 experience is similar, although Class #3 

indicated higher levels across the board which may account for some of the later minor survey differences 

between Class #2 and #3 responses.

Conclusion: While useful data to monitor the class and to construct Capstone Project teams, the value of 

these survey questions is largely instructional for the students to think in terms of the five essential roles 

and the differing perspectives that are necessary for prototyping and experimentation success.
33

Student Mix:

Observation: We did not achieve the student mix that was our goal of balanced participation, although 

close.

Conclusion: This was an essential survey question for forming diverse Capstone Teams.

Student Expectations (Q5):

Class #1:

Gain additional knowledge of the PM field (techniques and processes) and certify myself as PM 

professional.

Applicability for the T&E professional.

Understanding how prototyping and experimentation work together, and fit inside the capability 

development cycle.

I hope to develop an understanding of how the use of prototyping can reduce risk in the acquisition 

process for military systems. I hope to learn how to define/build a successful prototype program.

I'd like to understand how prototyping can help transition technology into programs of record.

To provide feedback on the course content and structure

Use capstone project lessons learned for a current project.

Class #2:

To get better insight into the potential for rapid prototyping and experimentation to accelerate 

adoption of leading edge technologies in the DOD acquisition system, as well as the possible road 

blocks. I hope to provide a valuable operational perspective and leverage previous concept 

development work I have done regarding unmanned systems in A2/AD environments.

I hope to learn more about refining requirements for large acquisition programs and gain a better 

understanding of the importance of prototype evaluation.

From R&D supplier POV, DOD 5000 is too bureaucratic, requirements are locked 3-10 years before 

EMD/LRIP, and changing the requirement to accommodate threat

Gain a greater understanding of the Prototyping and Experimentation landscape in the fiscally 

constrained environment which DOD is under.

Acquire latest approaches to system acquisition in a fiscal constrained environment.

To learn more about the DOD acquisition program and processes for getting R&D projects turned into 

Service programs of record.

Understanding the role of prototyping in acquisition and how it can improve decisions

To explore experimentation in terms of the Code of Best Practice by Alberts and its role in improve 

acquisitions.
34

Through this exercise, I hope to gain insight into the scope of an experimentation campaign. As I 

am new to this area, I am interested to see how planning might be carried out, even in the classroom 

environment- the parties that are involved and their roles, the process that is followed and the issues 

that emerge during the course of the planning.

I am new to the process, so I would like to increase my knowledge base.

Understand how this course content can be applied to AF institutionalization of Experimentation 

Campaigns.

I hope to learn new methods and best practices from others' experiences.

Identification of interdependencies within the acquisition process. Impacts of experimentation, 

prototyping and modeling on programs.

Better understand how to show the value of using modern development methods (including using 

prototypes) and enterprise strategies to defense leadership.

Develop an understanding of the acquisition and requirement development process and how that is 

integrated with experimentation and prototyping.

Prototyping and acquisition will be necessary for completion of my project for the CNO's Rapid 

Innovation Cell (CRIC). I'd like to expand my knowledge and get some real world experience in 

these areas.

To gain an introduction by means of a practical application process with respect to Science and

Technology or R&D activities. Also to exercise an opportunity on Technology Readiness Levels 

(TRLs) evaluation and designation. Exposure to the concept of experimentation using 

prototypes in a scenario setting to enhance foundational learning.

Class #3:

Understand how DoD will adapt and meet the rapid technology requirement coming from 

unconventional threat within short developmental cycle

Goals: To better understand how prototyping and experimentation can be used to lower the 

government's risk in developing/fielding new technologies and capabilities. To understand the intent 

of BBP 3.0 in pushing for a more informed buyer from the government's standpoint.

Expectations: I expect to gain a better understanding of the various perspectives from my classmates, 

given they sit in different parts of the acquisition system. I expect the curriculum and instructor(s) to 

facilitate discussions that yield fruitful learning points that I can bring back to my daily activities.

Learn more about Virtual Prototyping and its role in the future of DOD Acquisition.

1. Develop a deeper understanding and appreciation for prototyping and experimentation.

2. Have a better understanding of the role of prototyping and experimentation in system acquisition.

3. Learn from and get to know some fellow acquisition professionals.

4. Learn skills and methods that can assist me in my duties as a JCTD Manager.

To gain working knowledge of acquisitions in the technology space.

To become more familiar with the S&T and R &D process.

To gain additional insight regarding the path of DOD Acquisition process and perspectives of others 

in the DOD acquisition process.

Add to big picture knowledge across DON organizational domains of how prototyping and 

experimentation will successfully fit into the delivery of proposed new capability. Gaining better 

knowledge of those areas in which I am less experienced.

Learn more about how prototyping and experimentation can/should be implemented into the roadmap 

of a program. Gain some working knowledge to help bridge the gap between the Science and 

Technology, to include the Joint Concept Technology Demonstration community, and the Acquisition 

Community.

To gain a better understanding and working knowledge of how to utilize the development of 

prototypes to mature technology, get more out of testing and evaluation, reduce the risk to the 

program and ultimately deliver a better system to the fleet.
35

My goal is to become more effective in shepherding new technologies from commercial applications, 

through FFRDC performers into operational capabilities.

To learn practical skills in designing experimentation campaigns to include virtual and physical 

prototyping.

Learn new ways to accelerate technology development and delivery by using prototyping.

My goal is to gain insight into the prototype processes and to increase my understanding in the various 

phases. I believe that this understanding would enhance my abilities to support HQMC in their efforts 

to develop digital interoperability initiatives.

Apply knowledge gained to better plan and execute prototyping and experimentation strategies for 

advancing non-lethal technologies/capabilities to the end user.

A general review of items within my current job responsibilities.

Gain a better understanding of how other organizations operate.

Learn of DOD ""best practices".

Be exposed to ideas that could be implemented into my organization.

I hope to learn how to better promote prototyping and experimentation within acquisition program 

development.

To understand key constraints around effective rapid acquisition and build a working knowledge of 

rapid prototyping strategies.

A better understanding of how I may better coordinate my S&T efforts (BA-2/BA-3), with those of a 

technology transition team's (BA-4) follow-on efforts with the same technology as it transitions and 

evolves.

As an acquisition officer, I have held positions in every phase of DOD ACQ. I hope to interact with 

colleagues to streamline the acquisition of programs important to USPACOM.

Learn and understand the detailed process of the acquisition and development programs by means of 

prototyping and experimentation.

Bringing back teachable moments / methods to my command to enrich our collective knowledge in 

this arena.

Observation: Student goals and expectations were varied, but aligned well with the Course learning 

objectives and content.

Conclusion: The curriculum materials were aligned to the student's aims. This survey question is 

essential for course evaluation and to focus students on the course prior to the course.

The Course Outcomes (only Class #2 [green] and Class #3 [blue] had numerical responses)

Q2 and Q7 - Which of the following are True or 

False?

Before

%="True"

After

%="True"

Before

%="True"

After

%="True"

Most Major DOD Acquisitions had NunnMcCurdy breeches 74 85 78 85

Most DOD Acquisition programs do not conduct 

P&E 53 80 52 35

Unanticipated demo outcomes are unwelcome 26 40 41 65

System requirements should not change in 

acquisition 26 40 59 46

Experiments only fail when designed poorly 26 40 22 35
36

Observation: Faculty believes that all of the above are “True” to some extent or another. The responses 

are listed above in order of highest to lowest as determined after Class #2. The average of the student 

responses in Class #2 all shifted toward "True" and probably reflect the likelihood of the statement being 

true across the board for any particular program, as they learned in both classes.

Conclusion: The students gained a greater appreciation of the context and culture of DOD acquisition. 

The survey items are useful and the "before-after" comparisons are valuable, more so if they are available 

to the students and perhaps the Community-of-Interest over time.

Q4 and Q8 - Choose four (4) primary risk areas to delivery of a 

proposed new capability within cost and schedule.

% 

Before

% 

After

% 

Before

% 

After

Immature technologies 84 90 78 88

Unstable requirements 79 80 85 80

Overselling by vendors 74 30 44 73

Budget uncertainty 63 55 74 54

Lack of user feedback 47 45 52 65

Congressional actions 37 30 55 12

Other 16 20 21 0

Observation: Of course, all items listed are a risk to success. The responses are listed above in order of 

highest to lowest as determined in Class #2 (Before). Student responses shifted slightly with a significant 

increase for “immature technologies” and a notable decrease in “overselling by vendors” in Class #2. 

Class #3 differed slightly but the overall ranking in both classes are generally consistent. Here are the 

“Other” risks provided by individual students:

• Risk adverse acquisition transition partners

• Changing threats

• Biting off too much at once

• Poor experimental result

• Under-informed Government program management teams

• Integrated management and alignment of technology, prototype, test and integration, and 

acquisition across organizational domains

• Poor program strategy and management

Conclusion: The students clearly rethought these as a result of the course and mostly consistent results 

emerged from the two classes. The survey items are useful, but could possibly incorporate language from 

the "other" items that students offered. The "before—after" comparisons are valuable, more so if they are 

available to the students and perhaps the Community-of-Interest over time.
37

Q6 - Areas where students reported significant 

and useful gains in knowledge or insight

Pre-Course SelfAssessment (1-5)Pre-Course SelfAssessment (1-5)

Experimentation using prototypes (of varying 

types/maturity) 85% 2.68 77% 3.52

System demonstration or validation testing 70% 3.21 38% 3.78

S&T or R&D activities 55% 3.68 42% 4.11

Importance of diverse expertise and experience 

in teams 50% N/A 50% N/A

Technology Readiness Levels (TRLs) 45% 3.42 27% 3.74

Setting requirements for new military 

systems/capabilities 40% 3.05 23% 3.33

Program management for system acquisition 15% 3.21 35% 3.67

Observation: Significant reported gains in most areas with greatest gains in the areas that were selfreported as lower at the outset of the course and were the top desired learning outcomes for the course. 

Interesting that significant gain was reported for “S&T/R&D” although it was often cited as a top strength 

coming in to the course.

Conclusion: Good indication of course effectiveness. An essential question for the survey.

Q9 - Indicate (1-5) how well the course 

achieved these items and how important the 

item was to you

Achieved

by the 

Course

Importance 

to the 

student

Achieved by 

the Course

Importanc

e to the 

student

Quality of interaction between students and

instructors 4.65 4.45 4.54 4.70

Quality of interaction between students 4.45 4.35 4.50 4.67

Extent the stated course objectives were 

accomplished 4.45 4.30 4.12 4.50

Met student's goals/expectations entering the 

course

4.30 4.55 4.00 4.50

Quality and topics of classroom instruction 

and activities 4.30 4.05 4.27 4.63

Extent the course objectives were clear 4.25 4.15 4.08 4.41

Utility of course material and study notes 4.25 4.05 4.08 4.33

Quality and utility of the online portal 2.50 2.83 3.87 3.40
38

Observation: Ranking is nearly the same for both Classes. Most evaluations are well above the average 

of all 1-5 ratings and appear significantly so. Generally achieved the most important items in the order of 

the importance to the students of both Classes with the exception of their “incoming goals/expectations” 

which could reflect a mismatch between what they were expecting and what we delivered. The online 

portal was not readily available nor explained until late in the week. Here are the elaborations provided 

for unusually high or low scores [lettering refers to the survey form item]:

• As a pilot course, the planning and organization of the instruction is near perfect for the intended 

objective.

• Great course, exceeded my expectations. In line w/ AF approach in that it is the level of detail 

needed to put together the campaign.

• Expected to learn new things, d) very well laid out and organized, e) open/informative and

diverse, f) Alan Brown was great, g) Capstone provides hand-on + use of siege engines to 

illustrate, h) no issues w/ access (government laptop)

• Application of any concept into practice is the key. S&T (6.3) transition pressure is real. Lack of 

failure culture is alive and "not well". h) Would have been useful instead of burning CDs.

• The meeting interactions with my classmates/instructors was the most beneficial part of the 

course. I had very little acquisition/ prototyping/ experimentation experience, so I learned a lot.

• There was an online portal? [four other comments similar to this]

• c) Facilitated discussions (post review) of the case materials was most valuable. f) Sold the 

prototype concept well.

• The marks associated with stated course objectives relates to the information provided about the 

course prior to attending. I thought the course was going to focus on higher level prototyping 

push within OSD. The focus on experimentation and how to set it up was a welcome difference.

• I enjoyed the course and the content is especially valuable to my work.

• Options c-g above are all important

• Student contact is the best way to improve prototyping. Do more of it and less PPT.

• Expert instructors.

• I don’t see the need for an online portal. Distro of the course attendees was very helpful. Case 

study (Rail Gun/Have Blue) very thought provoking and useful.

• The course re enforces the importance of experimentation across DOD.

• Tempo of the course kept it interesting and constantly moving. Outcomes were therefore 

understood and engaged by all members of the course. Highly recommend keeping this course 

going in light of the high visibility and need for P&E.

• Student-instructor interaction was great and Q&A session was very informative.

• There were too many case studies, focus on four cases and study it.

• "Clear objectives and good organization to meet them along with a great teach/presenter 

discussion.

• The presentation information was clear and helpful to succeed with the capstone as well as take to 

utilize in day to day tasking.

• Instructors were excellent and very engaging with relevant topics and interactions. "

• As stated in class, look to align with current DAU courses in management system testing. It is 

critical these three areas understand it. Define, Demo and experiment and prototypes in DAU 

glossary. Move detail on use of prototypes and experimentation up in Acquisition Framework 
39

and phase of acquisition with details how to do that. More on pulling from 6.1-6.3 to get 6.4 and 

also how to align 6.1s-6.3 efforts to support 6.4.

• The greatest benefit of the class was learning from the other students and the encouragement and 

morning in which the instructors guided the conversation.

• The "5" marks are listed that way because I see them as vital to getting the most out of this 

course. I will be able to leverage a significant amount of what I learned in my efforts to support 

USMC efforts toward achieving digital inter ability.

• Didn't use online portal.

• Material was adequate, some death by MS PowerPoint on day 2. Good topic though. "

• I was not expecting a portal, but having one was great and made utilizing the material in the 

future easier.

• I was very impressed with the value, experience, and diversity of the class membership and 

instructors. "

• Text/Notes-->people can take with them and refer to later 

Topics-->Need to be timely and relevant 

---->F117 very interesting/historical 

---->Rail Gun Interesting/Happening NOW

Online-->Too many different ""databases"" online I lose track. "

• I would have liked more material on the acquisitions piece - from acquisitions professionals in the 

DOD. Also maybe from logisticians, for a different perspective.

• Low-the portal is good for materials and short term interaction with classmates but is not essential 

to course success. High- interaction with instructors and classmates is essential (recommend one 

social event to allow the opportunity to interact with all classmates)

Conclusion: Generally high marks for the course and an essential question for a student survey including 

the opportunity to elaborate on both high and low inputs. These helped us tune materials between Class

#2 and Class #3.

Q10 - Rate the following course attributes

Score

1-5

Score

1-5

Usefulness of the case studies 4.47 4.23

Usefulness of the Capstone Project 4.26 3.96

Depth of material presented 4.21 3.92

Relevance of the course to real world problems 4.21 4.38

Overall rating of the course 4.20 4.08

Pace of the course 4.11 3.96

Value of the course to you 4.11 4.08
40

Observation: Ranking is mostly the same for both Classes. Validation for the use of case studies and to 

a lesser extent the Capstone project, depth of material, and relevance. All good and consistent feedback. 

There was one “other” item offered for a course attribute:

• Willingness of faculty to take suggestions for improvement

Here are the elaborations provided for high or low scores [lettering refers to the survey form item]:

• Ratings are based upon current course curriculum; understanding that that will change over 

maturation of time and scope.

• Case studies, specifically Have Blue very useful.

• sufficient/adequate to meet needs, b) relevant, c) Seeing end results of other teams and

experiencing the process within own team was very informative/insightful, f) will be able to 

leverage this experience to help inculcate experimentation/prototyping into Service 

workforce/culture, g) met all needs/expectations

• Case studies illustrated the parts very well. D) Probably too much time in Teams.

• The Alan Brown case study of real world examples helped reinforce the class themes.

• A/C/E/F/G- P&E are relatively new to me in my young acquisition career. Coming from a 

C4ISR -- particularly F22 background-- I get a different right picture for follow on acquisition 

assignments as well as understanding the current assignment at JIEDDO. How to write our new 

directives and acquisition policy.

• I have sig experience on the acquisition side of programs. Learning about the S&T side and the 

community at large will help me understand what happens before an activity becomes a program 

of record.

• The case studies, with the exception of Nextech were excellent. Evaluating a successful program, 

like Have Blue, against the course guidelines covered in the course was valuable. The 

constructed Hagar example was a great way of teaching the concepts. NexTech was a think tank 

exercise and the value was questionable.

• The real usefulness of the Capstone Project was the interaction between the members and getting 

a better understanding of different organizational cultures.

• Course in general is well done. Look at the order of the materials. Multiple (team) check-ins 

might have merit.

• I would like to see, if it exists, current efforts which encourage experimentation. JIFX comes to 

mind, but it does not go through the process outlined. Is there, current ongoing venues which 

encourage experimentation?

• I appreciate how the course was presented in the "K.I.S.S." fashion. As an end user, I was not 

familiar with the "process". The examples to expand on the material/subject taught was very 

effective.

• The case study brought significant context to the importance of prototyping/experimentation to 

avoid cost overages associated with unanticipated problems.

• A good portion of the course seemed to be focused on convincing us of the need for 

experimentation and prototyping. I think most of us already saw the need.

• Refining requirements--feedback loop to helping refine requirements was not discussed to a 

major degree, but acknowledged. For instance, if the Capstone project required us to do an 

iteration of feedback to refine a requirement, the feedback loop would be closed.

• Improved system acquisition--was not really emphasized to a great degree but maybe that is a 

follow-on course.
41

• I really believe that P&E needs to be an integral part of the future acquisition reform and the 

Defense Innovation Initiative when we attempt to leverage new tech and concepts.

--more effort on the transition tools (experimentation and prototyping) and how they can be used 

to facilitate transitions.

--case studies on ""transitioning"" with lessons learned as why it was successful or note.

--try to focus the case studies on experiments"

• Appreciating the value (above)--managing risks and mitigation strategies to those risks is 

extremely valuable to better prepare a team when designing experiments and applying it to the 

structured process.

• Must haves for successful acquisitions.

• Help me understand the difference between prototypes and experiments.

• Most of this course was "preaching to the choir" effect for me.

• The course material did not get me to the importance but the discussions and case study and guest 

speakers did--especially Alan Brown.

• Definitely a key ingredient, but underutilized in the joint community.

• I had no idea how much effect P&E had on acquisition programs. This course helped to clarify 

that relationship as well.

• The course/capstone focused on designing experiments. The case studies on prototypes that 

enabled tech/capabilities to move forward. Suggest including some case studies that show failed 

acquisition due to no prototyping.

• Have Blue and Rail Gun were good but I don’t think the lessons learned and good generalizations 

were pulled from them.

• Again, the acquisition and requirements piece could be expounded upon by DMEs, not 

researchers or PMs.

• If leveraged properly and used as defined, prototyping and experiments are low risk high payoff 

events in the acquisition process (help buy down risk to the wrong thing being 

produced/acquired).

Conclusion: An important type of question for the student survey to continually adjust the course 

content and approach.

Q13 - How well did the course demonstrate the strategic 

importance of P&E in the following areas

Score

1-5

Score

1-5

Appreciating value of appropriate risk and possible failure in P&E 4.55 4.38

Exploring potential new capabilities 4.45 4.27

Maturing technologies 4.40 4.08

Improved system acquisition 4.16 4.04

Refining requirements 4.15 3.92
42

Observation: Ranking is same for both Classes. Validation of most important elements but a bit 

surprising that the link to the last two items was less recognized. Here are the elaborations provided for 

high or low scores [lettering refers to the survey form item]:

• a) identified a gap in institutional knowledge what already exists and/or is already being worked 

on -"new" is relative so what's new to you/me is not necessarily new

• Top down is good, but culture change, as you know, will be hard

• There is a systemic and cultural issue g USCG and DOD and if not only understood before these 

items can become game-changers to DOD acquisition

• A great S&T program will increase the likelihood of having a successful program of record.

• The importance of experimentation in exploring and maturing new capabilities were covered in 

detail. How requirements are refined during experiments was not addressed as much. Does the 

user observe and go back to his COCOM and adjust his mission requirements?

• More time should be spent on what the improved acquisition system is and how that impacts 

prototyping and experimentation current structure.

• Maturing tech was well supported by the Have Blue discussion.

• I believe, as was brought up throughout the week, that the current culture of "risk aversion" needs 

to be addressed and maybe even quantified refocusing toward prototyping and experimentation.

• Thought this course was very well put together.

• Relevance--low depth, but I assume that is what you are going for.

• F117 Case Study--AWESOME.

• Capstone was a useful exercise and cemented lessons learned.

• Case studies were invaluable because I gained a deeper appreciation for program design/decision 

points/and usefulness of experimentation.

• The case studies re enforces the real-life examples as to the value of conducting well planned 

exercises and demonstrations.

• Strongly encourage making this a 2-day (no more than 3 day) class.

• While the case studies were interesting (especially the Lockheed Skunk Works and 

Electromagnetic RailGun), they did not help with the “hows” of prototyping and experimentation. 

Would have fewer cases

• The course could be shortened with amount of information presented or material could be 

presented more in-depth. Design of Experiments probably most important aspect. Would have 

been nice to have more depth in that.

• Overall topic was a real-world problem and solution.

• Case studies help solidify the course content and were very relevant to current climate.

• Group interaction could be enhanced with a little leadership to jump start forming and accelerate 

the process.

• Very valuable course topic and timing of course was great for me was data calls receiving for 

prototypes. I gave a few definitions and importance of the topic out of the course. Interactions 

were key but more detail on prototypes and experiments to different levels in the Acquisition 

Framework would be important. How to flow it through this and use system engineering process 

to maximize work is important as well as how PMs should use it.

• The case studies were outstanding. Having people who were actually government of the 

experiment/program were amazing. They were able to provide so much context and a way of 

approaching common acquisition and S&T problem that was invaluable,
43

• There are many aspects of this course that I will be able to leverage in my work. Very useful and 

informative.

• Would have liked to see more strategies for prototyping and experimentation for ACAT level 

programs. Most case studies were for S&T (early) vs post MSA type activities. No real 

discussion as how this informs MDA decisions.

• Case studies were excellent and extremely valuable. Capstone was great to illustrate approaches 

and learn from meeting the people and instructors and learning from each other in this course 

format was invaluable! I intend to work with people I met here in my future work.

• Case studies are very interesting to listen to and learn from.

• A more hands on use of/design of an experiment would help. I talked with Al and Ann (DAU 

rep) about capability of Ft Belvoir/CERDEC NVESD"

• Day 1 was a bore -- for anyone who has been in the community for any length of time 

• DOE presentation and case study was simply wrong from an operating research point of view. 

They made EVERY mistake in the book. If you provided a proper intro to this, your definition 

would be much easier. i.e., a demonstration is one test where all FACTORS are held constant. A 

DOE term. "

• Would have liked more integration across case studies-BLUFS0 what was TRL line to be 

conveyed thru each case study.

• The case studies provided good examples and the capstone project drove home the points of the 

course.

Conclusion: An important type of question for the student survey to continually adjust the course content 

and approach.

Q11 - How well did Capstone Project provide appreciation for the 

following prior to formal acquisition:

Score

1-5

Score

1-5

Prior prototyping and experimentation 4.35 4.15

User buy-in 4.10 3.81

Mature technologies 4.00 3.65

Stable requirements 3.68 3.54

Observation: Ranking is same for both Classes. Fair reflection of the Capstone focus on conceiving and 

mostly designing the experiment. Here are the elaborations provided for high or low scores [lettering 

refers to the survey form item]:

• Not sure these are the right questions to focus on. Acquisition program. Capstone questions: 

why experiment, methodology, DOE. Not sure of the value of this question or my feedback.

• c) Critical validation of experimentation/prototyping for both expected and unexpected outcomes

• Would have been better if instructor stayed with the team
44

• F22 taught me while assigned there and watching the F35, KC Tanker program, FCS, etc. We as 

the SOS struggle with A-E and are too constrained in thinking to change quickly enough to really 

leverage commercial industry packages.

• These topics all contribute directly to the success a program

• Points C and D were central to the course and received much attention. Point A was not covered 

as thoroughly.

• Since the project had just one input, there was no change. B) and c) Lack of real "play"

• Integration of the user experience into the pre-development prototyping will shape requirements 

stability. The running of prototype experiments also can wring out technology maturity issues.

• The course homed in on the importance of making inexpensive mistakes in the realm of 

prototyping/experimentation versus costly premature production oversights.

• Stable requirements--this one is tricky. There is a push towards agile acquisition to include agile 

software development which requires stable requirements but allows for flexibility as well. So 

depending on the strategy (acquisition), "stable" requirements are relative. In fact, building on 

agility would be really programming.

• Prior prototyping and experimentation is extremely important to do this early on from a proof of 

concept all the way towards the operational/live environment. The 3x3 table provided would be 

greatly useful for the S&T RDT&E community. Shout be part of the AT&L community.

• Did not impact my opinion of these. Prior to the session, I believed all of these to be important.

• Historically acquisition programs that start really should wait until technologies are more mature. 

P&E is a much needed part of the acquisition process that has been appropriately addressed for 

years. This course helps bring it from none use to a valuable part of the process.

• Help me understand how project moves forward.

• Capstone had minimal effect on this appreciation.

• The benefit is class. The business use for "non-revolutionary" capabilities needs more emphasis.

• The course helped to highlight in greater detail the benefits of P&E prior to little "a".

• The capstone project really clarified the importance of user buy-in.

• Didn't increase my appreciation any more than existing DAU courses or real-life experiences.

• The value of the Capstone was great because it made you run through an experimental design 

right after learning about it (and with examples such as case studies). Having to run through a 

design in association with the didactic instruction made it much more retainable and drastically 

increased learning/retention.

• The capstone took a lot of time that perhaps was not well thought out. Teach DOE. Problem--

mission decomposition, requirements (data collection points, analysis of data), results (did the 

results answer the problem.

• Not very clear on the requirements piece--would like to hear from requirements professionals 

themselves.

• Prototyping of experiments are key to fleshing out requirements, TTPs, and actual TRLs.

• Stable requirements and mature technologies are essential for the testing of systems but 

should/can be ""flexible"" to lessons learned from prototyping and experiments.

Conclusion: An important type of question for the student survey to continually adjust the course content 

and approach.
45

Q12 - How well did Capstone Project provide 

you with approaches to:

Score

1-5

Score

1-5

Design an experiment 4.3 3.88

Conceive an experiment 4.1 3.77

Conduct an experiment 3.9 3.62

Evaluate results from an experiment 3.9 3.5

Observation: Ranking is same for both Classes. Fair reflection of the Capstone focus on conceiving and 

mostly designing the experiment. Here are the elaborations provided for high or low scores [lettering 

refers to the survey form item]:

• Simply put, this is the how to play book.

• Great exercise.

• no previous formal experience, b) no previous experience in this context, c) exercised critical 

thinking and provided springboard points to further explore (i.e., process)

• d) Did not think deep analysis was the point of the class

• d) not too much time was spent on this

• a) Conceiving an experiment is new to me and not as intuitive as it may be to others who work in 

depth with P&E and have an S&T background. As a TM, I am more in the system acquisition

phase(s) and spend more time in T&E, LRIP, etc. This earlier phase is what I need to understand 

at the concept and pre-system acquisition dev stages.

• Points b and c) were explained in detail through the Hagar example and the example sensor web

• Experimentation Capstone Project should be more closely associated with current environment to 

improve acquisition. Maybe a building block approach rather than one overall Capstone Project.

• c) and d) Experiment (project) ended at planning. Lack of data and "what's next".

• A greater review of the data collection plan and results, how they can be evaluated and where the 

pitfalls lie would be welcome.

• The course and Capstone Project provided a repeatable framework for conceiving, designing and 

executing experiments. I thought I knew how to do that going into the course, but didn't. I feel 

much more able to do this now.

Conclusion: An important type of question for the student survey to continually adjust the course content 

and approach.

Student feedback on ways to improve the course (Q14):

Continual assessment of resources and process: input/transformation/output. Excellent start-up as a 

pilot; little course correction required. 

Refine the Capstone project "Problem Statement." We spent a bit of time upfront defining the problem 

before diving into the scenario, objectives, hypotheses, metrics, etc. -- Day #1 is a lot of MS 

PowerPoint. Great content and presentations (but long). Can you make it more interactive in the 

afternoon? Work in groups, don't give us all of the answers up front. -- "policy" experimentation --

Garth's comment during the week.
46

Some of these are duplicate of comments I made in class on Friday and some are new. 1) Consider 

breaking up into 2-hr blocks of teaching (1-way) followed by 2-hr blocks of team work to wrestle with 

the concepts just delivered. 2) Human Centered Design classes do this (Josh Marcuse@ DSD Policy 

and Adrianne Miller @OPM. 3) Col Ermer brief/case study veered between case study and an 

advertisement for RRTO. Recommend stick to case study portion. 4) Big picture--this course 

reinforces the notion of capital-P and big $. Prototyping force capital-P PEO. Would like to see 

something (maybe separate course?) that addresses continuous lower-level prototyping @ all levels. 

Less hierarchically driven more continuous creation.

Spend a little more time on the strategic planning and need for develop planning to deliver capabilities

Bring back 1 to 2 participants from the previous class to share how they implemented the learning 

points from the course. Create a cohort for data and revisit the survey to get ROI for the course. For 

example, are you collaborating with new organizations, etc.? Provide the results from cohort to new 

participants to drive cultural change. More "jointness." Identify live exercise opportunities to plug into 

for the experimentation and prototyping implementation.

1) 6.3 pressure to transition. 2) Failure not embraced. 3) How can higher HQs (e.g.,

RDECOM/CERDEC) get educated/buy-in to lower level folks can experiments/fail more?

Great Course! Even as a non-acquisition professional, I had no problem following the material. That 

said, challenge us more! There is definitely more time for additional case studies as well as more 

background and reference materials. Thanks!

Please make sure to list an alternate POC on the info/logistics emails that came out in the weeks prior 

to the calls. I understood Dr. Robinson had a medical emergency, but there was no one else listed as a 

secondary POC. The class gave me --- as a person with very little P&E experience-- a very good 

understanding of the property

Talking about experimental campaigns as they relate to developmental planning.

Perhaps develop a small CBT for additional foundation building prior to the course. I hope this 

becomes a full DAU course or part of STM Acquisition certification levels.

Replace case study 3 RRTO with a real actual experiment, put the RRTO command brief into another 

slot in the schedule, add a section on organization culture and organization change

While the overview of RRTO was important. The NexTech cast study were not good examples of 

experimentation and could be eliminated. Overall, I felt that the course could be condensed into 4 

days. Note: In the interest of conserving resources, all materials, including surveys should be printed 

on both sides of the page. I am not sure how important it is for this course, but you may want to talk 

about past attempts to implement experimentation culture (e.g., Gen Slay at USAF/AFSC in the 1970s. 

WHY DID THEY NOT TAKE HOLD?

Have the RRTO brief by Col Ermer moved up to right after setting the stage brief. This will set 

expectations and goals for the course (up front) and makes the sequence of later briefs more 

understandable. Not sure I understand how the Capstone Project fits in line with the goals of the PEIA 

intent from the BATO study. *Overall: Great course to maintain to bring multiple organizations 

together to explain the future of prototyping and experimentation

"Rearranging order of the lessons. Col Ermer's would have been better up front. More discussion n in 

cases vice just making them monologues. 

Multiple project check-ins or have 2 parts. Second part would be ""what’s next?"". Students would 

get some ""results"" based on their approach. Fewer slides (about 30/hr.). Make short breaks (2 hr.

blocks are too long, all we do is sit). Re-think set-up for the Capstone brief (need a ""judging panel', 

even if it is just faculty). IDEA: Make the Capstone a request for funds from RRTO. The Hagar case 

was good but should involve more student work. The answers to Al's questions were in the book! 

Clarify 100% the difference between a manufacturing prototype and an experimental prototype."

Experimentation and Prototyping integration within the acquisition process is critical for the future of 

DOD. The course could improve by getting representatives from the service acquisition community to 
47

participate in the presentation of the course material. That would allow them to see and hear the level 

of understanding within the acquisition community. We had reps form the labs who had conducted 

experiments so the level of acceptance was high, but I feel the Program Management and PEO level 

attendance was less than it should be in order to build the momentum of acceptance as an acquisition 

practice.

Providing slides for a read ahead, Prove (or create) "dummy cards" of available assets that can be used 

or at least consider, government venue or funded experiment. Acquisition personnel participate in the 

overall course.

The first module addressed the acquisition lifecycle. Some example programs might have helped.

I was not entirely sure of the utility of the course for military operators. Early on, references were 

made to TRLs, 6.3 vs 6.4 funding and to a novice the interchangeable terms like "demo" "proof of 

concept" and "prototype." Operators are not exposed to nuances of acquisition are placed in an 

awkward position in becoming functional in the vernacular in a one week course. The course is 

successful in communicating the importance of P&E to an operator. But what is expected they do with 

that as they return to tactical units where the focus are METL tasks and combat readiness today; not 

two years from now? "Failing early and failing often" is not an axiom commonly shared in operations. 

However, this is critical in the world of acquisition, testing/evaluation, and probably where the course 

should select its audience.

More focus on the how of prototyping and experimentation instead of the why. 

I will provide chapter by chapter comments via email. 

Better guidance for the Red Team.

1) Fewer case studies--provide a comparison that matches AC's DoE practices. 2) Better background 

on BBP 3.0 and Acquisition cycle. 3) Have the slides for Case Studies in advance (for notes purposes). 

4) Discuss the level of autonomy (1-6) scale and its future application for policy.

The Setting the Stage brief used non-standard definition and was somewhat confusing. For example, 

the "R" in RDT&E typically is the S&T vice how it was described as development.

More time on group discussion.

Course was great from a general perspective. --There are many different types of experiments. Done 

for all levels of DOD--not just services--nice to include those in future iterations.

If you tell us to bring over laptops, please have something for us to do on it. It would be better to have 

the binder information on disc instead of hard copy for those of us who flew in to the training. 

1. More detail on the transition phase or "valley of death" navigation would be useful. A few examples 

of what works and what has not worked in transition efforts would help. 2. Add an OV1 slide to the 

template. 3. Add a risk cube to the template. 4. Provide a listing of S&T websites of contact info for 

DOD efforts that would be very useful as a quick reference. 

Great course! Time well spent. Thanks for the opportunity to attend. 

Tie the topic to strategic BBP 3.0 more. Higher level view of topic and less tactical view

Better define Red team assignment.

Include negative case studies.

Hands-on conducting/observing/analyzing a mini experiment. Could be accomplished in an afternoon 

at CERDEC/VVESP.

Bring in acquisitions and logistics to the course.

Felt that the course was presented from a somewhat narrow perspective -- e.g., the RRTO way which is 

only one piece of the long and wide spectrum of the Research to Acquisitions perspective with the time 

frame (i week) this is probably all that is possible, but if given more time and to make this course more 

comprehensive. I would bring in the services, SMEs, requirements guys, folks in acquisition, and 

PEOs.

I left wanting to see how what we learned here fits into the larger process of technology acquisitions 

from the folks that are actual decision-makers (on the acquisitions, requirements, etc.). 
48

Appreciated having the COL and LtCol’s perspectives to tie everything into the warfighters 

perspective. Overall, great course. Learned a lot and hope more people take it or receive this 

knowledge in some fashion. 

1) Provide example(s) of successful prototyping venues (w/characteristics); possible guest lecture from 

program like TSOA. 2) Matrix of characteristics of prototyping and experimentation, test, and demo to 

help visualize which event is executing. Person provided a chart on his evaluation. 

Observation: Students took this question very seriously and provided invaluable, detailed comments that 

were used between Class #2 and #3 to upgrade the materials as well as the schedule.

Conclusion: An essential question for the student survey in order to continually adjust the course content 

and approach.
49

E. Pilot Course 3 Schedule 

TIME TOPIC INSTRUCTOR

DAY 1

MORNING 

8:00-9:00 Course Management and Orientation 

a. Ben Riley Welcome 

b. Logistics 

c. Course Overview

d. Student Expectations 

Robinson

9:00-10:00 Setting the Stage

Definitions and terms will be introduced and a short 

discussion of the DOD acquisition process to provide 

context. 

Lawrence 

10:00-10:15 BREAK 

10:15-11:15 Setting the Stage (cont) Lawrence

11:15-12:15 Introduce Capstone Project Ramberg

12:15-1:00 Working Lunch—Teams Organize for 

Capstone Project 

1:00-2:30 Best Practices and Lessons Learned in Rapid 

Technology Development

Ermer

2:30-3:30 Definitions and Military Examples of 

Prototyping and Experimentation 

Sciarretta

3:30-3:45 BREAK

3:45-5:00 Design of Experiments (DOE) with Case Study Sciarretta

DAY 2

8:00-8:30 Brief Review of Day 1 Ramberg

8:30- 10:00 DOE with Case Study (cont) Sciarretta

10:00-10:15 BREAK

10:15-12:00 DOE with Case Study (cont) Sciarretta

12:00-1:00 Working Lunch – Groups Discuss their 

Capstone Project

1:00-2:15 DOE with Case Study (cont) Sciarretta

2:15-2:30 BREAK

2:30-5:00 Case Study 1:Smart Sensor Web

SSW concept (2000-2002) was an intelligent, webcentric distribution and fusion of sensor information to 

enhance local situational awareness, on demand, to 

warfighters (dismounted infantry platoons and squads).

Sciarretta

DAY 3

8:00-8:30 Review of Day 2 Ramberg

8:30-10:00 Case Study 2 – Have Blue 

Have Blue was the prototyping and experimentation 

efforts for the USAF F-117 stealth attack aircraft

Guest Speaker: Alan Brown

Gravatt (facilitator)

10:00-10:15 BREAK
50

10:15-11:30 Case Study 2 (continued) Gravatt

(facilitator)

11:30-1:00 Working Lunch – Capstone Project 

1:00-5:00 Work on Capstone Project Team

DAY 4

8:00-8:30 Review Day 3 Lawrence

8:30-9:00 Red Team Presentation Sciarretta

9:00-9:15 Red Team Feedback and Discussion

9:15-9:30 BREAK

9:30-11:30 Case Study 3 – Robotics (ARL)

Guest Speakers:

Brett Piekarski and Marshal Childers

Sciarretta(facilitator)

11:30-12:30 Working Lunch – Capstone Project

12:30-2:30

Case Study 4: Rail Gun 

Guest Speaker: Roger Ellis

Lawrence 

(facilitator)

2:30-4:30 Work on Capstone Project/Presentation Team

DAY 5

8:00-8:30 Presentation Group 1 Lawrence

8:30-8:45 Feedback Group 1

8:45-8:55 Break and Set-up for Group 2

8:55-9:25 Presentation Group 2 Ramberg

9:25-9:40 Feedback Group 2

9:40-9:50 Break and Set-up for Group 3

9:50-10:20 Presentation Group 3 Robinson

10:20-10:35 Feedback Group 3

10:35-10:45 Break and Set-up for Group 4

10:45-11:15 Presentation Group 4 Gravatt

11:15-11:30 Feedback Group 4

11:30-12:00 Overall feedback/discussion on presentations

12:00-12:45 Course Wrap-Up Lawrence (Team)

12:45-1:00 Community of Interest website Robinson

1:00-1:30 Student Course Feedback Session Robinson (Team)
51

F. Example of Student Capstone Project Outbrief 
52
53
54
55
56
57
58

G. Description of Course Files available online at http://ctnsp.dodlive.mil/ 

 

• Folder: Course Support, with following content:

o PEIA poster

• Folder: Main Curriculum, with following content:

o Orientation

o RRTO (Wyatt) Briefing

o Setting the Stage

o Technology Development

o Definitions and Military Examples 

o Design of Experiments

o Schedule PEIA September Course

• Folder: Case Studies, with following content:

o Smart Sensor Web Folder

o UUV MMS Folder

o Wolfpack Folder

o Have Blue

o Robotics Folder

o Rail Gun (Student Handout)

• Folder: Student Capstone, with following content:

o Read ahead

o Case Study Description

o Presentation which introduces the effort/task to the students

o Student Project template

• Folder: COI Website
59

H. Speaker Biographies 

Course Instructor and Case Study Speaker: Albert Sciarretta

Mr. Albert A. Sciarretta (U.S. Army Lieutenant Colonel, retired) is currently serving as a Senior Research 

Fellow in CTNSP. For more than 30 years, as a U.S. Army officer and civilian contractor, he has used his 

operational, research and development, and operations research experience to assess the military benefits 

of advanced technologies, develop S&T investment strategies, and design/execute tactical through 

operational wargames, experiments, and demonstrations. His current CTNSP effort is focused on 

developing a course of instruction addressing prototyping and experimentation. He has significant 

experience in designing and executing various DOD experiments and demonstrations utilizing 

combinations of live-virtual-constructive simulations of joint through tactical urban operations. His past 

CTNSP efforts include defining human dimension/performance issues and metrics for dismounted 

infantry, identifying operational and technical needs for command and control systems for small unit 

operations, and assessing advanced technologies for unmanned and autonomous systems and determining 

their use in military operations.

Mr. Sciarretta is President of CNS Technologies, Inc. In this position, he works primarily as an 

independent consultant, supporting various DOD organizations. Mr. Sciarretta is also a member of three 

high-visibility organizations: 1) the National Research Council (NRC) Board on Army Science and 

Technology (BAST), 2) a Chief of Staff of the Army Red Team to review U.S. Army Training and 

Doctrine Command (TRADOC) Army Capabilities Integration Center (ARCIC) Futures Concepts, and 3) 

an Assistant Secretary of the Army for Acquisition, Logistics, and Technology (ASA(ALT)) Independent 

Review Team (IRT) for evaluating Army science and technology (S&T) and acquisition programs. He 

was selected for membership on the BAST for his expertise in research and development, operations 

research, experimentation, modeling and simulation, management, as well as his knowledge of human 

performance, small unit operations, and advanced information, sensor, and test technologies. He has 

served on many NRC BAST study committees, with the most recent being “Making the Soldier Decisive 

on Future Battlefields” for which, he served as deputy chair for squad operations and as a principal coauthor for the final report.

Mr. Sciarretta is a certified Professional Engineer in the state of Virginia. He has two MS degrees –

Operations Research and Mechanical Engineering – from Stanford University; a BS degree in General 

Engineering from the United States Military Academy; and a one-year, undergraduate credit in Design of 

Military Vehicles from the British Army’s Royal Military College of Science.

Guest Speaker: Alan Brown (Have Blue)

Mr. Alan Brown retired in February, 1992, as Director of Engineering at Lockheed Corporate 

Headquarters. His two principal concerns there were the promulgation of Concurrent Engineering and 

Stealth Technology throughout the Corporation. He has given invited papers on both these subjects at 

national and international levels.

From 1975 to 1989 he was a member of the Lockheed Advanced Development Projects, colloquially 

known as the "Skunk Works". He served first as the deputy program manager for the Have Blue lowobservable research aircraft. He then became program manager and chief engineer for the F-117A Stealth 
60

Fighter from initial concept until the first production aircraft was built, 1978 to 1982, and from 1982 to 

1989 was Director of Low Observable Technology.

He joined Lockheed in 1960, starting in the physics laboratory of the Lockheed Missiles and Space 

Company in Palo Alto. He moved to the aircraft company in Burbank in 1966, working on propulsion 

installation on the Supersonic Transport and the FX and VSX aircraft (which later became the F-15 and 

S-3A respectively), and was engineering manager for the Lockheed group at Rolls-Royce on the L-1011 

commercial transport program.

He began his aeronautical career with an engineering apprenticeship at Blackburn Aircraft in England 

from 1945 to 1950. After obtaining a Diploma of the College of Aeronautics, Cranfield University, 

England, in 1952, he worked at Bristol Aeroplane Company as an aerodynamicist prior to going to the 

United States in 1956. He worked as a research associate and lecturer at the University of Southern 

California, and as a research associate at Wiancko Engineering Company before joining Lockheed.

Since retirement from Lockheed, he has taught short courses at Cranfield University, England; Linkoping 

University, Sweden; Georgia Institute of Technology; and the U.S. Navy Post-graduate School,

Monterey, California. He has been active in the University of California Mathematics, Engineering and 

Science Achievement (MESA) program for middle and high schools since 1994.

Guest Speaker: Edwin Ebinger (Unmanned Underwater Vehicles)

Dr. Ed Ebinger is a retired Naval Commander and Master EOD Technician who has been supporting the 

Chief of Naval Operations staff at the conclusion of his 20-year active duty service in 2006. Ed was a 

qualified Surface Warfare Officer (SWO) and Special Operations Officer (EOD), as well as a designated 

Foreign Area Officer (FAO). Tours of duty included multiple overseas postings, combat deployments 

and Defense Agency staff billets. During his naval service, Ed received numerous campaign, service and 

unit commendation awards.

Since 2006, Ed has provided analysis in requirements and programming to the Chief of Naval Operations 

Staff within the Expeditionary Warfare Directorate. Within this directorate he has held positions that with 

responsibility for programs and funding in support of the Navy Expeditionary Combat Enterprise 

(NECE), and has recently been assigned responsibilities for the Navy’s Mine Warfare (MIW) 

programming. His portfolio has included multiple Joint and Navy Rapid Acquisition Initiatives, ACAT I 

and JROC-Interest programs, and unmanned systems in the ground, air and maritime domain. Ed has 

extensive interface with both the S&T and Acquisition communities in the project formulation, 

prioritization, execution and transition to operations processes. Ed has been an active participant in 

multiple internal and external fora and has played an active role in shaping strategic visions and 

implementation pathways

Guest Speaker: Colonel Dan Ermer (Funding Projects and Lessons Learned)

Colonel Dan Ermer reported to his current position within the National Defense University (NDU)

Dwight D. Eisenhower School for National Security and Resource Strategy on 10 July 2015. He was 

commissioned into the Marine Corps in December 1987 following graduation from Iowa State University 

under the Naval Reserve Officers Training Corps (NROTC) Program. For 27+ years he’s served in the 

Marine Corps in various combat engineering and logistics billets, while assigned many diverse command, 
61

staff, special assignment, and joint tours of duty, both home and abroad. The highlights of his operational 

and combat tours consist of: Operation(s) Desert Shield and Storm, Operation Silent Assurance, 

Operation Resolute Response, and Operation(s) Iraqi Freedom 2-2 / 4-6 / 5-7. Prior to his recent 

assignment to the Eisenhower School, Colonel Ermer served as the Deputy Director and Chief Operations 

Officer, Rapid Reaction Technology Office for the Office of Under Secretary of Defense—Acquisitions, 

Technology and Logistics. While in this capacity, he worked to progress evolving, emerging 

technologies and deliver advanced prototypes in support of time-sensitive warfighter capability needs. 

Colonel Ermer holds a Master of Science degree in National Resource Strategy from NDU Eisenhower 

School, Washington, DC; a Master of Arts degree in Human Resources Development from Webster 

University, St. Louis, MO; and a Bachelor of Science degree in Agricultural Business from Iowa State 

University, Ames, IA.

Guest Speaker: Wade Hall (Wolf Pack)

Mr. Wade Hall (U.S. Marine Corps Colonel, retired) is a 1982 graduate of East Tennessee State 

University. Upon graduation he was commissioned a Second Lieutenant in the United States Marine 

Corps. He completed Marine Corps Officer Training at The Basic School, Quantico Marine Corps Base, 

followed by Naval Aviator flight training. He was designated a Naval Aviator in October of 1984.

He commanded at the squadron and regimental levels, served as the operations officer for the Marine 

Corps largest Marine Air Group, and the Navy’s largest advanced jet training squadron. He conducted 

aircraft carrier operations aboard the USS Nimitz and USS Kennedy. His tours have taken him worldwide from Australia to Norway, the Middle East to the Far East.

He served a joint tour with the Office of the Secretary of Defense, working for the Deputy Secretary in 

the Office of Force Transformation with the late Vice Admiral Arthur Cebrowski.

He also served at Headquarters Marine Corps – Aviation Office as the director all Marine Corps Aviation 

platform weapons systems.

Colonel Hall participated in four (4) combat operations from Desert Storm, Allied Force, and Iraqi

Freedom. He has over 3900 flight hours in six different type aircraft, with 2000 in the EA-6B Prowler 

and 330 hours in combat. He is qualified as a pilot, instructor pilot and mission commander in the EA-6B 

Prowler, the A-4 Skyhawk and aircraft commander in the KC-130 Hercules. His personal decorations 

include the Defense Superior Service Medal, Legion of Merit, Meritorious Service Medal with Gold Star, 

two Strike/Flight Air Medals with combat V, six Air Medals, and two Navy / Marine Corps 

Commendation Medals.

He is a graduate of the Joint Forces Staff College and Marine Corps War College. He holds a Bachelor’s

Degree in History and Political Science and a Master’s Degree in Strategic Studies,

Colonel Hall retired from active duty in 2009 after 27 years of honorable service. Wade is currently the 

President and CEO of Corps Defense, a business development consulting company.

Guest Speaker: Rick Nagle (Unmanned Underwater Vehicles)

Mr. Rick Nagle is a retired Navy Officer and Master EOD Technician who has been supporting PMS-408 

since his transition from twenty years of active duty service in 1997. During his Navy service, Rick was a 
62

qualified Surface Warfare Officer (SWO) with considerable shipboard experience in both the Navy EOD 

and surface warfare communities. He served as an EOD Officer in Charge on several tours of duty, and 

as a department head at Navy EOD Mobile Units, and in the latter half of his career served as 

Commanding Officer (CO) of EOD Mobile Unit SIX in Charleston, SC, and CO of EOD Mobile Unit 

EIGHT in Sigonella, Sicily. While in command of EOD Mobile Unit SIX, Rick deployed as the Task 

Unit Commander for Navy EOD MCM forces in Operation Desert Storm and led Navy EOD forces in 

several other campaigns and operations. His final assignment was as the EOD Program resource sponsor 

and requirement officer at the Pentagon. During his Navy service, Rick received numerous campaign, 

service, and unit commendation awards.

Since 1997, Rick has provided analysis and project management support to the Navy EOD Program 

Office, specializing primarily in technology transition and acquisition of maritime systems to address 

Navy validated requirements and future capability needs. Rick continues to play a contributing role, most 

significantly in the Navy's Shallow Water (SW), Very Shallow Water (VSW) and Expeditionary Mine 

Countermeasures and Maritime Homeland Defense counter-underwater improvised explosive device 

(UW/IED) investment portfolios in support of the Navy Expeditionary Combat Enterprise (NECE). Over 

his active duty and follow on career, he has authored numerous papers, presented at professional 

symposia, and has played an instrumental role in the development and staffing of investment strategies, 

strategic plans, and concepts of operations to gain stakeholder guidance and feedback and to sustainment 

support for material solution investment toward the acquisition of future naval capabilities.

Guest Speaker: Robert Simmons (Unmanned Underwater Vehicles)

Mr. Robert Simmons is the Assistant Program Manager (APM) for Underwater systems within PMS-408 

Expeditionary Missions Program Office, a component of the Naval Sea Systems Command Acquisition 

and Commonality Directorate (NAVSEA 06). He also serves as the Assistant Technical Project Officer 

for NAVSEA involving Data Exchange Agreements between the United States and 17 allied and coalition 

partner nations involving Unmanned Underwater Vehicles acquisition and technology development. He 

has a Bachelor of Science in Aerospace and Ocean Engineering from Virginia Tech and a Master of 

Science in Public Administration from American University. He is certified by the United States 

Department of Defense in Level III Program Management and Systems Planning, Research, Development 

and Engineering. He has received the Joint Meritorious Unit Award for Support of Operation Iraqi 

Freedom, the Meritorious Civilian Service Award and the James D. Collie award for Sustained 

Outstanding performance in his duties as APM for the underwater system programs. Besides leading the 

fielding of next generation equipment to the Explosive Ordnance Disposal divers and Expeditionary Mine 

Countermeasures operators, he has worked with US, International and NATO Science and Technology 

activities, mainly through collaboration with the Office of Naval Research, for over 30 years with a focus 

on technology transition to the fleet.

Guest Speaker: Tom Swean (Unmanned Underwater Vehicles)

Dr. Tom Swean joined the Office of Naval Research in March 1993 as Science Officer for the Ocean 

Technology Program. In 1996, he became Team Leader for the Ocean Engineering and Marine Systems 

Team, and in 2005 he assumed the duties of Team Leader for the Maritime Mine Warfare Team and as 

such leads the Organic Mine Countermeasures Future Naval Capability Program. His work at ONR has 

focused on developing AUV systems and component technologies for applications to mine warfare, 

explosive ordnance disposal, naval special warfare, and ocean science research. Collateral duties at ONR 

include activities as National Leader to The Technical Cooperation Program, Technical Panel-13 (Mine 
63

Warfare) , Head of U.S. Delegation to NATO Maritime Capability Group /3 (Mine Warfare and Port 

Security) and Member at Large to the NATO STO Systems, Concepts and Integration Panel. He received 

the Meritorious Civilian Service Award in 1999 and was named ONR Future Naval Capability Manager 

of the Year in 2009. In 2010, he was awarded the Arthur E. Bisson Prize for Naval Technology 

Achievement; and in 2014, he was awarded the Superior Civilian Service Award.

He started his career in 1976 as a Research Scientist for the STD Research Corporation, Arcadia, 

California addressing applications of magneto hydrodynamics to commercial and military power 

generation devices. While at STD he was a Principal Investigator on ONR contracts to develop pulsed 

power for space borne applications.

He joined federal service in 1981 as a Research Mechanical Engineer at the Naval Research Laboratory in 

Washington, DC. During his tenure at NRL he was Principal Investigator of the Fluid Dynamics Task 

Area, Head of the Fluid Physics Section of the Remote Sensing Division, and Head of the Center for 

Hydrodynamic Developments in the Laboratory for Computational Physics and Fluid Dynamics. During 

this time Dr. Swean led a group of 15 scientists and engineers in fundamental and applied studies of 

turbulent flow physics and modeling of free surface hydrodynamic signatures.

Dr. Swean attended Old Dominion University in Norfolk, Virginia, graduating with a Bachelor of Science 

degree in Thermal Engineering in 1970. He then entered Virginia Polytechnic Institute and State 

University, earning a Master of Science degree in Aerospace Engineering in 1972 and a PhD degree in 

Aerospace and Ocean Engineering in 1976. His graduate research at VPI was in the areas of hypersonic 

flows, spacecraft re-entry dynamics and turbulent wake physics. While at VPI he was named NSF 

Exchange Fellow and spent two years at the Institute for Fluid Dynamics in Bucharest, Romania. Dr. 

Swean is the author of 67 scientific and technical publications.

From 1963 to 1965, he was enlisted in the United States Marine Corps, and continued in the USMC 

Reserves from 1966 to 1970.

Guest Speaker: Brett H. Piekarski, PhD (Ground Operation Robotics)

Dr. Brett Piekarski is the Chief of the Micro and Nano Materials and Devices Branch within the Army 

Research Laboratory (ARL) and the Cooperative Agreement Manager for ARL’s Micro Autonomous 

Systems Technology Collaborative Technology Alliance. While at ARL he has also served as the 

Manager of ARL’s Specialty Electronic Materials and Sensors Cleanroom Research Facility (2002-2008), 

as a Microelectromechanical Systems (MEMS) Researcher in the Micro-Devices Branch (1997-2002), 

and as a Researcher in electronics manufacturing technologies (1988-1997). 

He has a BS in Mechanical Engineering from North Dakota, a MS in Mechanical Engineering from Johns 

Hopkins University, and a PhD in Mechanical Engineering from the University of Maryland.

Guest Speaker: Marshal Childers (Ground Operation Robotics)

In the Autonomous Systems Division of the Army Research Laboratory (ARL), Mr. Marshal Childers is 

responsible for leading the Integration and Assessment Team which supports the integration, prototyping, 

and experimentation of autonomous robotics technologies. His current focus is on integrated performance 

and transition of technologies developed by the ARL Robotics Collaborative Technology Alliance. As a 

mechanical engineer, Marshal contributes to the research and development of ARL-funded unmanned 

systems technology and has oversight of the technicians that support the Division. He is the Technical 
64

Manager for the U.S. Army Special Operations Command (USASOC)-sponsored Small Unit Support 

Improvised Explosive Device (IED) Defeat (SUSI) Program. Mr. Childers is a contracting officer's 

representative (COR) on a number of advanced development programs that integrate autonomy onto 

testbeds and evaluate the platform performance in relevant environments. 

He has an MS in Mechanical Engineering from University of Maryland, Baltimore County (UMBC).

Guest Speaker: Roger Ellis (Railgun)

Mr. Ellis is the Program Manager for the Electromagnetic (EM) Railgun Program Office at the Office of 

Naval Research. EM Railgun, an Innovative Naval Prototype (INP), is a transformational multi-mission 

solution for ship self-defense, anti-surface warfare, naval surface fire support, and time-critical strike. 

The Railgun is designed to deliver lethal, hyper-velocity projectiles at ranges in excess of 100 nautical 

miles within six minutes. Mr. Ellis directs the development of the INP to mature the technology for 

transition to a formal ship acquisition program. Mr. Ellis has been involved in naval railgun technology 

development since 2000, and previously served as the Technical Director for the effort from 2005 to 

2009.

Previous projects Mr. Ellis has been involved in include; an advanced propulsion system for a fire from 

enclosure shoulder launched weapon (co-authored patent), Tech Team 21 member for the navy’s new 

155mm Advanced Gun System for the DDX ship with oversight over the gun barrel and oscillating 

assembly, conventional gun barrel technology efforts with focus on erosion mitigation, and systems 

engineering IPT coordinator for the Naval Surface Fire Support program.

He earned a B.S. in Mechanical Engineering from Utah State University in 1995 and an M.S. degree in 

Mechanical Engineering from Virginia Tech in 1996.
65

I. Online PEIA Community of Information Prototype

The prototype of the PEIA Community of Interest (COI) was made available to all of the students who

attended the course. The purpose was to share content with the students and to provide collaborative tools 

for the students to share information and problems they are facing. All of the most current content is 

included in the COI. The COI was prototyped in a Google Site within a domain sponsored by the 

National Defense University. Should the course be adopted by another organization, they could continue 

using this site or they could design a community in another tool.

The online pages available to the students included:

• Home Page with Announcement Capability

• Course Schedule in Two Formats

• Downloadable Student Lists

• Student and Faculty Discussion Forums

• Case Study Slides

• Capstone Project—Student Assessment Documents

• Student Capstone Projects Arranged by Pilot Course

• All of the Materials Distributed in the Pilot Courses

• Resource Materials 

• Pre- and Post-Course Self Evaluations