PROGRAM  MANAGERS TOOL  KIT

Fourteenth Edition (Ver 2.0)

March 2008
Report Documentation Page Form Approved

OMB No. 0704-0188

Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and

maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,

including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington

VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it

does not display a currently valid OMB control number.

1. REPORT DATE

MAR 2008 2. REPORT TYPE

3. DATES COVERED

 00-00-2008 to 00-00-2008 

4. TITLE AND SUBTITLE

Program Managers Tool Kit

5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S) 5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

Defense Acquisition University,Fort Belvoir,VA,22060-5565

8. PERFORMING ORGANIZATION

REPORT NUMBER

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)

11. SPONSOR/MONITOR’S REPORT 

NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution unlimited

13. SUPPLEMENTARY NOTES

14. ABSTRACT

15. SUBJECT TERMS

16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF 

ABSTRACT

Same as

Report (SAR)

18. NUMBER

OF PAGES

152

19a. NAME OF

RESPONSIBLE PERSON

a. REPORT

unclassified

b. ABSTRACT

unclassified

c. THIS PAGE

unclassified

Standard Form 298 (Rev. 8-98)

Prescribed by ANSI Std Z39-18 
DAU PROGRAM MANAGERS TOOL KIT



DAU  PROGRAM  MANAGERS

TOOL KIT

Fourteenth Edition (Ver 2.0)

April 2008
DAU PROGRAM MANAGERS TOOL KIT

ii

For sale by the

U.S. Superintendent of Documents, Government Printing Office

Internet: bookstore.gpo.gov Phone: 202-512-1800 Fax: 202-512-2250

 Mail Stop: SSOP, Washington, DC 20402-0001
DAU PROGRAM MANAGERS TOOL KIT

iii

PREFACE

This Fourteenth Edition (Ver 2.0) of the DAU Program Managers Tool 

Kit contains a graphic summary of acquisition policies and managerial 

skills frequently required by DoD program managers. It is a current version of a “Tool Box” that was first developed by Charles F. Schied of the 

Defense Acquisition University (DAU) Program Management Course 

(PMC) 92-1. For convenience, the Tool Kit is sized for insertion into a 3-

hole, 5-1/2” x 8-1/2” “Day Runner.” The information in the Tool Kit is extracted from DAU course material and is based on DoDD 5000.1, DoDI 

5000.2, the Defense Acquisition Guidebook (DAG), CJCSI 6212.01D 

(March 14, 2007), and CJCSI 3170.01F (May 1, 2007). Material from 

the DAU Acker Library and Knowledge Repository was also used. 

Since the DAU Program Managers Tool Kit is a compilation of classroom presentation and teaching materials used in a number of different 

courses at DAU, the charts and tables vary in look and feel.

Users of the Tool Kit are reminded that this summary is a guide only 

and should not be used as a substitute for official policy guidance. Periodic review of official policy guidance is recommended.
DAU PROGRAM MANAGERS TOOL KIT

iv
DAU PROGRAM MANAGERS TOOL KIT

ACKNOWLEDGMENTS

As sponsor of the Tool Kit, the Learning Capabilities Integration Center 

(LCIC) recognizes the following members of the DAU staff and faculty 

for their input to this Fourteenth Edition (Ver 2): 

• Bill Bahnmaier, formerly of DAU’s Program Management and 

Leadership Department on the Capital/Northeast Regional Campus, 

for coordinating the input and editing the material received from 

various DAU faculty and staff sources.

• Charles Cochrane, formerly the DAU Program Management Center 

Director, and currently a DoD contractor supporting DAU, for his 

significant contributions on current DoD acquisition policy.

• Mike Dorohovich, a contractor from CSC Corporation supporting 

DAU, for his contribution on the DAU knowledge sharing systems.

• Duane Tripp for his input on the latest international acquisition policy 

and guidance.

• The Learning Capabilities Integration Center and its Center Directors 

and Performance Learning Directors, including Sharon Jackson, 

John Morrell, David Bachman, Leslie Deneault, John Krieger, Bill 

Kobren, John Claxton, John Snoderly, George Prosnik, Bruce Moler, 

and Bill Motley. 

• The Visual Arts and Press Department, including Kathryn 

Sondheimer for layout/graphics, Judith Greig and Carol Scheina for 

editing, and Frances Battle for liaison with the Government Printing 

Office and the commercial printer. 

Other significant contributors from the DAU faculty were John Kelley 

and Bob Daugherty of the Program Management Department; Robert 

Carlson, Bob Gustavus, Gerry Land, and Miriam Cohe of the Business, 

Cost Estimating, and Financial Management Department; and Larry 

Heller of the Logistics Management Department.

Randy Fowler

Director, Learning Capabilities

Integration Center 

Bradford Brown

Director, Center for 

Acquisition Management
DAU PROGRAM MANAGERS TOOL KIT

vi
DAU PROGRAM MANAGERS TOOL KIT

vii

Table of Contents

Preface...........................................................................................................iii

Acknowledgments ......................................................................................... v

Table of Contents..........................................................................................vii

Chapter 1—Acquisition Management ..................................... 1

The Program Manager’s Bill of Rights and Responsibilities...................2

Defense Acquisition Decision Points and Phases ..................................3

Acquisition Categories (ACAT)................................................................4

Acquisition Strategy Considerations........................................................5

Acquisition, Technology and Logistics (AT&L) Knowledge

Management System ..................................................................6–8

Acquisition, Technology, and Logistics Sharing System (AKSS) ........6

Acquisition Community Connection (ACC) .....................................6

Defense Acquisition Guidebook (DAG) ..........................................7

Integrated Framework Chart (IFC) ..................................................7

AT&L ACQuire ..................................................................................8

Best Practices Clearinghouse (BPCh).............................................8

Process Performance and Learning Tools.......................................8

Defense Acquisition Board Timeline Milestones B, C, and FRPDR.......9

Milestone Decision Information—A Possible Construct..........................9

Information For Milestone/Decision Reviews ........................................10

Interoperability........................................................................................11

S&T Linkage to Defense Acquisition Process.......................................12

Joint Capabilities Technology Demonstration (JCTD)

10-Step Life-Cycle Process Flow......................................................13

Acquisition Program vs. ATD and JCTD................................................14

Program Structure/Schedule (Example) ...............................................15

International ..................................................................... 16–17

DoD International Armaments Cooperation Policy ...............................16

The Scope of Defense Cooperation......................................................16

Defense Sales vs. Cooperative Acquisition...........................................17

International Activities Associated with Defense Acquisition Phases.........17

Funds Management .......................................................... 18–31

Resource Allocation Process—Overlap................................................18

Planning, Programming, Budgeting, and Execution (PPBE)—

Planning Phase.................................................................................19

PPBE—On-Year Program/Budget Review (Even Year)........................19

PPBE—Off-Year Program/Budget Review (Odd Year).......................20

Resource Allocation Process ................................................................20

Congressional Enactment Timetable ....................................................21

Procurement Appropriations............................................................22–23

RD&TE Appropriations (Account Numbers) .........................................24

RD&TE Appropriations (Relationship Between MFP 6 R&D

Categories and RDT&E Appropriations Budget Activities)...........24
DAU PROGRAM MANAGERS TOOL KIT

viii

Chapter 1—Acquisition Management (Continued)

Sample Navy Appropriations and Budget Activities..............................25

Appropriation Life...................................................................................26

Below Threshold Reprogramming Actions............................................26

Life Cycle Cost Composition .................................................................27

Product Improvements...........................................................................27

Cost Estimating......................................................................................28

Cost Estimating Relationships (CER)—Parametric..............................28

Cost Estimating Requirements..............................................................29

Program Cost and Schedule Breach Parameters.................................30

Performance Measurement...................................................................31

Contracting ....................................................................... 32–36

Contracting—Components of Contract Price........................................32

Types of Contracts .................................................................................32

Contract Type Features....................................................................33–34

Pre-Solicitation Process.........................................................................35

Post-Solicitation Process .......................................................................35

Other Ways to Buy.................................................................................36

Defense Industry Business Management ............... 37–43

Contractor Profitability Ratios................................................................37

Aerospace/Defense Industry Contractor Profitability Ratios.................38

Lockheed Martin Corp.—Dupont Formula Analysis (An Example)........39

Cash Cycle.............................................................................................40

Contractor Financing and Payments.....................................................40

Direct and Indirect Costs .......................................................................41

Assigning Indirect Costs ........................................................................41

Life Cycle of Indirect Cost Rates ...........................................................42

Contractor’s Cost Proposal....................................................................42

Contractor Business Planning Process Outputs...................................43

Logistics .............................................................................. 44–50

Life Cycle Logistics ................................................................................44

Principal Life Cycle Logistics (Goals/Objectives)..................................44

System Operational Effectiveness (SOE) .............................................44

Best Practice: Support Elements...........................................................45

Support Element Definitions..................................................................45

Life Cycle Logistics Linkage To Defense Acquisition Process..............46

Performance-Based Logistics (PBL).....................................................47

Performance-Based Logistics (PBL) Tool Kit ........................................47

Post-Production Support Decision Process..........................................48

Operational Availability...........................................................................48

Life Cycle Sustainment Metrics .............................................................49

Life Cycle Sustainment Outcome Enablers...........................................49

Logistics Management Community of Practice (LOG COP) ................50

Defense Acquisition Guidebook—Life Cycle Logistics .........................50
DAU PROGRAM MANAGERS TOOL KIT

ix

Chapter 1—Acquisition Management (Continued)

Organization ...................................................................... 51–52

Program Office Organization Structures (Examples) ....................51–52

“Traditional” or Functional Structure .................................................51

“Pure” Product Structure ...............................................................51

Matrix Structure..............................................................................52

Integrated Product Teams..............................................................52

Manufacturing .................................................................. 53–58

The Role of Manufacturing in the Acquisition Process .........................53

Common Production Risks That Greatly Impact Cost, 

Schedule and Performance ..............................................................53

Producibility......................................................................................54–55

Producibility—Defense Acquisition Guidebook, 4.4.6.1 Producibility ...54

Quality Management Systems...........................................................55

Key Characteristics and Variation Reduction ...............................55–56

Production Readiness Review (PRR)................................................57

Additional Manufacturing Information Sources ..................................57

Test and Evaluation ........................................................ 58–63

Test and Evaluation (T&E)—Types and Tasks ......................................58

Test and Evaluation Nomograph ......................................................59-60

Modeling and Simulation (M&S) Planning ............................................61

Hierarchy of Models and Simulations....................................................62

The Evolution of Modeling and Simulation............................................63

Planning and Control tools ....................................... 64–71

Gantt Chart (Example)...........................................................................64

Milestone Chart (Example)....................................................................64

Network Schedules................................................................................65

Network Schedules (Example)..............................................................66

Examples of Network Scheduling Software..........................................67

Schedule Development..........................................................................68

Lead Time Chart....................................................................................69

Line of Balance Technique.....................................................................70

Acquisition Program Baseline................................................................71

Risk Management .............................................................. 72–75

Risk Management Process Model ........................................................72

Root Cause Analysis .............................................................................72

Risk Management Process—Details ....................................................73

Risk and Tradeoffs Analysis...................................................................74

Program Manager’s Checklist for Review of Tradeoff 

Planning and Studies...............................................................................75

Technical Performance Measurement.........................................................75
DAU PROGRAM MANAGERS TOOL KIT

Chapter 1—Acquisition Management (Continued)

Systems Engineering ....................................................... 76–94

What Is Systems Engineering?.............................................................76

Systems Engineering Tasks ..................................................................76

Systems Engineering Policy in DoD......................................................76

Systems Engineering Process—Technical Management Processes ..77

Systems Engineering Process Linkages...............................................78

Defense Acquisition Management Framework—

Technical “V” Activities ......................................................................79

Concept Refinement Phase Systems Engineering Activities...............80

Technology Development Phase Systems 

Engineering (SE) Activities ...............................................................80

System Development and Demonstration Phase SE Activities............81

Production and Deployment Phase Systems Engineering Activities........81

Operations and Support Phase Systems Engineering Activities .........82

Requirements (User Needs) Analysis Questions .................................83

Attributes of a Well-Defined Requirement.............................................83

Systems Engineering Process—Design Operations ............................84

Systems Engineering Design Considerations—“The Fishbone”..........85

Technical Management Processes .......................................................86

Specification Development and Technical Reviews..............................87

Technical Review Definitions .................................................................88

Technical Review Best Practices...........................................................89

Specifications and Standards................................................................90

Performance vs. Detail Specifications ...................................................90

Program-Unique Specifications.............................................................91

Configuration Management...................................................................92

Interface Management...........................................................................93

Interface Control Concept......................................................................93

How To Create a Work Breakdown Structure........................................94

Basic Purposes of WBS ........................................................................94

SOFTWARE ............................................................................... 95-98

Software Development—Technical Review Relationships ...................95

Software Development—Key Life Cycle Review Factors......................96

Candidate Software Measures (Metrics)...............................................97

Quality Events For Software..................................................................97

Software Management Best Practices..................................................98

Software Acquisition Worst Practices....................................................98

Chapter 2—Leadership and Managerial Skills ............... 99

Management and Leadership..............................................................100

Empowerment, Delegation, and Coaching .................................101–102

Coaching Skills ...............................................................................102

Generic IPT Process ..........................................................................103

Formula for Effective Team Performance ............................................104
DAU PROGRAM MANAGERS TOOL KIT

xi

Team Charter.......................................................................................105

Working Groups...........................................................................106–107

Team Development Wheel..............................................................106

Typical Working Groups ..................................................................106

Management Tradeoffs for Working Groups...................................107

Team Performance Model....................................................................107

Team Decision Making ........................................................................108

Effective Meetings................................................................................109

Decision Briefing..................................................................................110

Communications..........................................................................111–113

Counseling ......................................................................................113

Time Management...............................................................................114

Management Tools and Techniques............................................115–116

ChapteR 3—Problem-Solving Tools..................................... 117

Brainstorming............................................................................... 117-118

Cause-and-Effect Diagram .................................................................119

Force Field Analysis ............................................................................119

Histogram ............................................................................................120

Scatter Diagram .................................................................................120

Surveys ................................................................................................121

Affinity Diagram....................................................................................122

Affinity Diagram (Example)..................................................................123

Pairwise Ranking ......................................................................... 124-125

Pairwise Ranking (Example) ..............................................................125

Pareto Chart.........................................................................................126

Benchmarking......................................................................................127

Flowcharting .......................................................................................128

Standard Flowchart Symbols ..............................................................129

Deployment Flowcharts...............................................................129–130

Nominal Group Technique (NGT)................................................131–132

Creative Problem Solving ....................................................................133

Knot Chart............................................................................................134

Qualitative Problem Solving.................................................................135

Gantt Chart ..........................................................................................136

Chapter 2—Leadership and Managerial Skills (Continued)
DAU PROGRAM MANAGERS TOOL KIT

xii
DAU PROGRAM MANAGERS TOOL KIT

CHAPTER 1

ACQUISITION  MANAGEMENT

• Things that make you go “Hmmm?...”

 

“The only thing most auditors fix is the blame.”

“Experience is something you got just after you needed it.”

“People are smarter than they look; listen to them.”

 

“The last 10 percent of the performance sought generates one-third 

of the cost and two-thirds of the problems.”

“Never open a can of worms unless you want to go fishing.”

“Those who believe it cannot be done, will you please get out of the 

way of those who are busy doing it?”

• Things we should always remember.

“Be honest in everything you say, write, and do.”

 

“Be good to your people, and they will be good to you.”

“Forgiveness is easier to obtain than permission.”

“Keep everyone informed; when in doubt, coordinate.”

“Be the first to deliver bad news.”

“Bad news does not get any better with time.”

“If you are sitting at your desk, you are not managing your program.”

 
DAU PROGRAM MANAGERS TOOL KIT

THE PROGRAM MANAGER’S BILL OF RIGHTS

AND RESPONSIBILITIES

RIGHTS:

Program Managers have the RIGHT to:

• a single, clear line of authority from the Defense Acquisition 

Executive;

• authority commensurate with their responsibilities;

• timely senior leadership decisions;

• be candid and forthcoming without fear of personal consequences;

• speak for their program and have their judgments respected;

• receive the best available training and experience for the job; and 

• be given adequate financial and personnel resources.

RESPONSIBILITIES:

Program Managers have the RESPONSIBILITY to:

• accept program direction from acquisition executives and implement 

it expeditiously and conscientiously;

• manage their programs to the best of their abilities within approved 

resources;

• be customer-focused and provide the user with the best, most costeffective systems or capabilities;

• innovate, strive for optimal solutions, seek better ways to manage, 

and provide lessons-learned to those who follow;

• be candid about program status, including risks and problems as 

well as potential solutions and likely outcomes;

• prepare thorough estimates of financial and personnel resources 

that will be required to manage the program; and

• identify weaknesses in the acquisition process and propose 

solutions.
DAU PROGRAM MANAGERS TOOL KIT

DEFENSE ACQUISITION DECISION points and PHASES

(DoDI 5000.2)

A B C

Pre-Systems Acquisition

USEr NEEDS &

TECHNOLOGY OPPOrTUNITIES

PrODUCTION &

DEPLOYMENT

OPErATIONS

& SUPPOrT

SYSTEM DEVELOPMENT

 & DEMONSTrATION

frP

DECISION

rEVIEw

DESIGN

rEADINESS

rEVIEw

Systems Acquisition Sustainment

• Process entry at Milestones A, B, or C

• Entrance criteria met before entering phase

• Evolutionary acquisition or single step to full capability

CONCEPT

rEfINEMENT

CONCEPT

DECISION

Program

Initiation

TECHNOLOGY

 DEVELOPMENT
DAU PROGRAM MANAGERS TOOL KIT

 DAB Review 

 Designated by DAE 

 Decision by DAE

 Component Review 

 Designated by DAE 

 Decision by Service Sec/CAE

 ITAB Review*

 Designated by ASD(NII)**

 Decision by ASD(NII)

 

 Component Review 

 Designated by ASD(NII) 

 Decision made by Svc Sec/CAE

 Does Not Meet ACAT I Criteria 

 Designated by Svc Sec/CAE 

 Decision by Svc Sec/CAE 

 

 Does Not Meet ACAT I, IA, or II 

Criteria 

 Designated IAW Component 

Policy

 Decision at lowest appropriate

level

 

 Not otherwise designated ACAT I, 

IA, II, or III 

 Designated IAW Component Policy 

 Navy/USMC ACAT IVT/IVM 

 Decision at lowest appropriate level 

ACAT ID:

ACAT IC:

ACAT IAM:

ACAT IAC:

ACAT II:

ACAT III:

ACAT IV:

Major

Defense

Acquisition 

Programs

Major AIS

Acquisition 

Programs

Major

Systems

All Others

(except 

 Navy and 

USMC)

Navy

USMC

ACQUISITION CATEGORIES (ACAT)

SECNAVINST 5000.2_ 

No Fiscal Criteria

$140M RDT&E or 

$660M Procurement 

(FY 00 Constant $)

$378M Life Cycle Cost or 

$126M Total Program Cost 

or $32M Program Cost in 

any single year 

(FY 00 Constant $) 

$365M RDT&E or $2.190B 

Procurement 

(FY 00 Constant $)

* Information Technology Acquisition Board

** ASD for Networks and Information Integration (NII); formerly ASD(C3I)
DAU PROGRAM MANAGERS TOOL KIT

ACQUISITION STRATEGY CONSIDERATIONS

(Defense Acquisition Guidebook, Chapter 2)

• Program Structure

• Acquisition Approach

• Capability Needs

• Test and Evaluation

• Risk Management

• Resource Management

— Funding Under an Evolutionary 

Acquisition Strategy

— Advance Procurement

• Systems Engineering Plan

• Interoperability

— Information Interoperability

— Other than Information Interoperability

• Information Technology

• Research and Technology Protection

— Protection of Critical Information

— Anti-Tamper Measures

• Information Assurance

• Product Support Strategy

• Human Systems Integration

• Environmental Safety and Occupational 

Health

• Modular Open Systems Approach

• Business Considerations

— Competition

◆ Fostering a Competitive Environment

–Competition Advocates

–Ensuring Future Competition for 

Defense Products ◆ Building Competition into Individual 

Acquisition Strategies

–Applying Competition to Acquisition 

Phases

–Applying Competition to Evolutionary 

Acquisition

–Competition and Source of Support

– Industry Involvement

◆ Potential Obstacles to Competition

–Exclusive Teaming Arrangement

–Sub-Tier Competition ◆ Potential Sources

–Market Research

–Commercial and Nondevelopmental 

Items

–Dual-Use Technologies

–Use of Commercial Plants

– Industrial Capability ◆ Small Business Innovative Research 

(SBIR) Technologies

— International Cooperation

◆ International Cooperative Strategy

◆ International Interoperability

◆ International Cooperation Compliance

◆ Testing Required for Foreign Military 

Sales

— Contract Approach

◆ Performance-Based Business 

Strategy

◆ Modular Contracting

◆ Contract Bundling

◆ Major Contract(s) Planned

◆ Multi-Year Contracting

◆ Contract Type

◆ Contract Incentives

◆ Integrated Contract Performance 

Management

◆ Special Contract Terms and Conditions

◆ Warranties

◆ Component Breakout

— Leasing

— Equipment Valuation

◆ Program Description

◆ Accounting Review

◆ Contract Implications

• Best Practices

• Relief, Exemption, or Waiver

• Additional Acquisition Strategy Topics

NOTE: In addition to the Acquisition Strategy, there are five plans required: Acquisition Plan (FAR/DFARS), 

Program Protection Plan and Test and Evaluation Master Plan (DoDI 5000.2), Information Support Plan (ISP) 

(CJCSI 6212.01D), and Systems Engineering Plan (USD AT&L Memo February 20, 2004).
DAU PROGRAM MANAGERS TOOL KIT

Acquisition,Technology, and Logistics

Knowledge Sharing System (AKSS)

The AKSS portal is the new knowledge repository component 

of the AT&L Knowledge Management System (AKMS). It is the 

primary reference tool for the Defense AT&L community, and 

it provides a central point to access and organize AT&L resources and 

information. AKSS has a personalized search, sort, and display capabil-

ity, and it provides a means to link information and reference assets from 

various disciplines into an integrated information source. The AKSS portal 

provides direct links to DoD acquisition policies, including USD(AT&L) 

memoranda, Federal Acquisition Regulation, and Defense Federal 

Acquisition Regulation Supplement, as well as department and service 

guidance and instructions. In addition, the AKSS portal is a trusted source of 

information on acqui­sition news, Web sites, training opportunities, and other 

relevant information. To learn more, go to <https://akss.dau.mil> and take 

the online virtual tour.

Ask a Professor (AAP) <https://akss.dau.mil/aap> is a service offered as part of AKSS. Users submit acquisition-related questions 

and receive formal responses. In addition, the AAP contains a database of questions and answers that are categorized by subject area 

and can be browsed or searched. 

Acquisition Community Connection (ACC)

The ACC is the collaborative component of the AKMS that 

focuses on acquisition-related topics and disciplines such 

as contracting, logistics, program management, and risk 

management. It consists of Communities of Practice, Special Interest 

Areas, and collaborative workspaces that 

• connect people with know-how across DoD organizations and industry;

• enable members to interact and share resources, ideas, and experiences to support job performance and avoid duplication of effort; and 

• identify partnership development opportunities.

Members may request workspaces in ACC, which provide a way for 

physically dispersed individuals to centrally locate and share documents and references as well as manage team projects. To learn more, 

go to https://acc.dau.mil and take the online virtual tour.

Acquisition,Technology and Logistics 

(AT&L) Knowledge Management System

(Composed of the following systems)
DAU PROGRAM MANAGERS TOOL KIT

Defense Acquisition Guidebook (DAG)

The DAG <https://akss.dau.mil/dag> provides links to policy, law, and 

useful content housed in communities of practice. It allows users to 

navigate through the Guidebook via a document index, graphical 

interface (Life Cycle Framework), or a search by topic. 

Integrated Framework Chart (IFC)

The AT&L IFC <https://acc.dau.mil/ifc> is a pictorial road map of key 

activities in the systems acquisition process. Users navigate through 

a graphical model of the three major acquisition process areas: Joint 

Capabilities Integration and Development System (JCIDS); Defense 

Acquisition; and Planning, Programming, Budgeting, and Execution 

(PPB&E). 

A B C

User Needs & Technology Opportunities

Design

readiness

review

Concept

Decision

full-rate

Production

Decision review

Concept Production & Deployment

refinement

Technology

Development

System Development

& Demonstration

Operations

& Support

IOC

• Process entry at Milestones A, B, or C

• Entrance criteria met before entering phase

• Evolutionary Acquisition or Single Step to 

 Full Capacity

Pre-Systems Acquisition Systems Acquisition Sustainment

(Program

Initiation) fOC
DAU PROGRAM MANAGERS TOOL KIT

AT&L ACQuire

ACQuire <http://acquire.dau.mil> is a search tool focused on 

the specific needs of the acquisition workforce. It uses the DAU 

acquisition taxonomy, trusted acquisition sites, and selected AT&L 

resources to enhance searches and derive better results. Searches 

can be conducted by individual or multiple sites; document titles; 

topic; content, via an index of major categories; and subcategories.

Courseware is also searchable via ACQuire. Users can suggest 

additional AT&L sites that should be included in ACQuire crawls. 

Best Practices Clearinghouse (BPCh)

The BPCh is an innovative “clearinghouse” approach that will 

improve all DoD’s acquisition processes by helping programs 

select and implement proven practices appropriate to the individual 

program needs. Initially, the BPCh will focus on software acquisition 

and systems engineering.

The Clearinghouse provides:

• an authoritative source for practices, lessons learned, and risks to 

avoid;

• validated practices with consistent, verifiable information;

• an active knowledge base to help with practice questions;

• an intelligent front-end to quickly get to answers;

• useful information and tools to help find, select, and implement 

practices appropriate to specific programs; and

• living knowledge through a constantly updated, expanded, and 

refined database.

Process Performance and Learning Tools 

Process Performance and Learning Tools (PPLTs) link learning 

and job support assets to complicated process flow to help users 

create plans and other AT&L products accurately and efficiently. The 

following PPLTs have been developed:

• Pricing Support Tool <http://pricingtool.dau.mil>

• Performance Based Logistics Toolkit <https://acc.dau.mil/pbltoolkit>
DAU PROGRAM MANAGERS TOOL KIT

Defense Acquisition Board Timeline 

Milestones B, C, and FRPDR

Milestone Decision Information—

A Possible Construct

 

OIPT

Review

OIPT

Review

Overarching Integrated Product Team (OIPT) Meetings

JROC

Review

JROC

Review

3-4 wEEKS

2 wEEKS 2 wEEKS

Milestone

45 DAYS

180 DAYS

30 DAYS

DAB DAB

TEMP to 

USD(AT&L)/

DOT&E

JCB

Review

JCB

Review

21 DAYS

CAIG review

Draft POE

Draft 

CArD 

to CAIG

final 

CArD 

to CAIG

CAIG Briefs 

Preliminary 

LCCE to PM

AoA 

results to 

PA&E

60 DAYS

CAIG final 

LCCE to PM

10 DAYS

final POE 

& Component 

Cost Position to 

CAIG

3 DAYS

CAIG 

report to 

OIPT

• ADM - Acquisition Decision Memorandum • CAIG - Cost Analysis Improvements Group 

• CARD - Cost Analysis Requirements Description • DAB - Defense Acquisition Board

• FRPDR - Full Rate Production Decision Review • JCB - Joint Capabilities Board

• JROC - Joint Requirements Oversight Council • LCCE - Life Cycle Cost Estimate(s)
DAU PROGRAM MANAGERS TOOL KIT

10

Information for MILESTONE/DECISION Reviews

(See DoDI 5000.2, CJCSI 3170.01F, and CJCSI 6212.01D)

Milestone/Review

Information CD A B DRR C FRP

Acquisition Decision Memorandum5 X X X X X X

Acquisition Program Baseline5 X X X

Acquisition Strategy5 (see Page 5) X X X

Affordability Assessment X X

Analysis of Alternatives3&5 (AOA) X X X X

AOA Plan X

Benefit Analysis and Determination1&8 (bundled acquisitions) X

Beyond LRIP Report2 X

Capabilities Development Document (CDD)5 X

Capabilities Production Document (CPD) X

Certification of Compliance with Clinger-Cohen7 X X X X

Certification of Compliance with BEA7 (FM MAIS only) X X X X

Clinger-Cohen Act Compliance5&7 (MS-A, MAIS only) X X X X

Competition Analysis1&8 (depot-level maintenance rule) X

Compliance with Strategic Plan X

Component Cost Analysis5&9 (MAIS; optional MDAP) X X

Consideration of Technology Issues X X X

Cooperative Opportunities1 X X

Core Logistics/Source of Repair Analysis1&8 X

Cost Analysis Requirements Description5&9 (MDAP and MAIS) X X X

Economic Analysis (MAIS)7 (may be combined w/AoA at MS-A) X X X

Exit Criteria5 X X X X X

Industrial Capabilities1 (n/a MAIS) X X

Independent Cost and Manpower Estimate5 (MDAPs; n/a MAIS) X X X

Independent Tech. Assessment (ACAT ID only) (DDR&E Option) X X

Information Support Plan1&5 X X

Initial Capabilities Document (ICD)4&5 X X X X

J-6 Interoperability and Supportability Certification X X

J-6 Interoperability and Supportability Validation X

Live Fire T&E Waiver2 (covered systems) (n/a MAIS) X

Live Fire T&E Report2 (covered systems) (n/a MAIS) X

LRIP Quantities (n/a AIS) X

Market Research X X

Milestone A Certification X

Milestone B Certification X

Operational Test Agency Report of OT&E Results X X X

Post Deployment Performance Review X

Program Protection Plan1 X X

Pgm Environ, Safety and Occup Health5 (w/NEPA schedule) X X X

Registration of Msn Critical and Msn Essential Info Sys5&7 X X X

Spectrum Certification Compliance8 X X

System Threat Assessment5&6 X X

Systems Engineering Plan X X X X

Technology Development Strategy (MDAP and MAIS) X X X

Technology Readiness Assessment5 X X

Test and Evaluation Master Plan (T&E Strategy only due at MS A) X X X X

1. Summarized in Acquisition Strategy

2. OSD T&E oversight programs only

3. MDAP: A, B, C; MAIS: A, B, FRPDR

4. Milestone C if program initiation

5. Program initiation for ships

6. Validated by DIA for ACAT ID; 

AIS use capstone InfoOps 

sys threat assessment decision

7. Milestone C if equivalent to FRP

8. Milestone C if no milestone B

9. MAIS whenever an economic

analysis is required
DAU PROGRAM MANAGERS TOOL KIT

11

INTEROPERA

BILIT

Y

DoD Policy: DoD Policy: DoD Directive 4630.05

Information Technology and National Security Systems employed by U.S. Forces shall, where required (based on capability context), interoperate with existing and planned systems and equipment, of joint, combined, and coalition forces and with other U.S. Government departments and agencies, as appropriate. 

Joint Staff (J-6) Certification and Validation

Service/

Agency

United States Joint 

Forces Command reviews Net-Ready Key Performance Parameters (NR-KPP) and integrated architectures 

J-6 Interoperability and Supportability Certification

Joint

Staff

Joint

Staff Combatant Commands

J-6 Systems Validation

• JITC witnesses/conducts interoperability tests, and• Provides Joint Interoperability Test Certification to J-6 

for validation

J-6 

Validation TEMP/ Test Results JITC

CDD

CPD

ISP*

LEGEND:CDD: Capability Development DocumentCPD: Capability Production DocumentISP: Information Support Plan

J-6

Certification

Service/

Agency

*NR-KPP certification

JITC: Joint Interoperability Test CommandTEMP: Test and Evaluation Management 

Plan
DAU PROGRAM MANAGERS TOOL KIT

12

S&T LINKAGE TO DEFENSE AC

QUISITION PROCESS

Options

(1) Concepts for new systems/upgrade

systems out of production.

(2) Insert into ongoing systems 

development or complete JCTDdevelopment.

(3) Upgrade system in production/ 

fielded systems or JCTD.

(4) Use of new technology for 

demilitarization/disposal.

Warfighting Needs

&

R&D Objectives

Systems

S&

T

• Basic Research• Applied Research

• Joint Capability Technology Demonstration (JCTD)• Advanced Technology Demonstration (ATD)• Lab/Field Demo• Warfighting Experiments

STOP

STOP

(1) (2) (2) (3) (4) (3)

IO

C

B

A

System Development

& Demonstration Production & Deployment Operations & Support

C

Concept 

Refinement

Technology

Development

System 

Design

System 

Demonstration

LRIP Full-Rate Prod & 

Deployment

ReadinessReview

FRPDecisionReview

Life-Cycle

Sustainment 

Disposal 

FO

C

ConceptDecision

oversight

panel

Oversight

Panel

MD

A

DECISION

Oversight

Panel

Adv Tech

Dev

Tech Base

Design
DAU PROGRAM MANAGERS TOOL KIT

13

Joint 

Capa

bilities 

Tec

hnolog

y

Demonstration (JCTD)

10-step life-c

ycle 

Process flow

CRB Brief

JCT

D

COCOM J(x), CoS, or CDR

Submit Proposal COCOM Sponsor with Service/Agency Partners Proposal Package Workshop JCTD Candidate and Team Formulation

and Proposal Package Development

� Joint� Coalition� Interagency

� AS&C Funds to Seed� COCOM Lead� OEs, Process SME

� Demonstration,

Assessment 

&

 Transition� Final Reporting

� Rate and Rank� JROC 

Validation

� DUSD(AS&C) Selection 

&

USD(AT&L) Approval

� Congressional Notification�

Step 

Implementation Direction

1 Step 2

Step 3

Step 9 & 10 Step 6, 7, & 8 Step 4 & 5

COCOMPrioritizedNeeds
DAU PROGRAM MANAGERS TOOL KIT

14

A

c

quisition 

Program vs. ATD and JCTD

LEGEND:ACAT—Acquisition Category CDD—Capability Development DocumentCPD—Capability Production Document

Implementation Directive/•FAR/OTA

Gain understanding of

and evaluate utility priorto acquisition decision

• Develop concepts of

operation and doctrine

JROC Approval and

Prioritization

Role of User

MotivationDocumentedNeedOversightFundingACATConfigurationand TestingRules

Fully

F

YDP Funded

Acquisition

Program

• Develop, produce,

and field system

• Cost, schedule,

performance

ICD/CDD/CPDMilestone Decision

AuthorityDoD 5000 Series/

AllFAR

ACATs

System/SubsystemPrototypes DT/OT

Max Involvement

Advanced Technology

Demonstration (ATD)

Not ACAT Effort• Demonstrate feasibility

and maturity

• Reduce technical risks

and uncertainties atrelatively low cost

Not RequiredLabs/R&D Centers

RDT&ETechnologyDemonstrationsInformal/FAR/OTASome Involvement

Joint Capability Tech

Demonstration (JCTD)

DUSD(ASC) Oversight PanelRDT&E (2 years in field)Not ACAT EffortMax InvolvementTech Demonstrations In FieldEnvironment/MUA

DT/OT—Developmental/Operational TestingDUSD(AS&C)—Deputy Under Sec Def 

(Advanced Systems and Concepts)

FAR—Federal Acquisition Regulation 

FYDP—Future Years Defense Program ICD—Initial Capabilities DocumentMUA—Military Utility Assessment
DAU PROGRAM MANAGERS TOOL KIT

15

PROGRAM STRUCTURE/SC

HEDULE (EXAMPLE)

Contract

Award

(increment 1)

RDT&EProcurementO&MMILCON

FY 01

FY 02

FY 03

FY 04

FY 05

FY 06

FY 07

FY 08

FY 09

FY 10

FY 11

FY 12

FY 13

FY 14

CR Tech Dev

Sys Integ Sys Demo

System Devel and Demo

CR

A

B

Prototype

Competition

DRR

FRP C IOC

B FRP IOC

FRP C

SI SD PROD

C

Milestones

Reviews

and Phases

Technical

Reviews

(increment 1)

Testing

(increment 1)

Deliveries

(increment 1)

OA

FOT&E

EDM

EOA

LRIP Production

LRIP Full-Rate Prod and Depl Sustainment/Disposal

Incr 2

Incr 3

(eng dev models)

DT&E

LFT&E

IOT&E

SRR

DRR

PCA PDR

TD

ASR

Prod

& Deployment Operations

& Support

CD

LRIP

SFR

Full-Rate Prod

& Depl

B IOC

Full-Rate Prod

DRR

CDR SRR
DAU PROGRAM MANAGERS TOOL KIT

16

DoD INTERNATIONAL ARMAMENTS

COOPERATION POLICY

THE SCOPE OF DEFENSE COOPERATION

Production and

 RDT&E Procurement Follow-on Support

Information Foreign Military Sales Cooperative Logistics

Exchanges Supply Support

Engineer and Direct Commercial Sales Mutual Support

Scientist Exchanges Exchanges

Cooperative R&D Cooperative Production Logistics Support

(Joint Funds)

Comparative or Coproduction/Licensing Host Nation Support

Joint Testing (Foreign Funds) Defense Industrial

Base

Standardization Reciprocal Procurement

 The Program Manager’s Focus

“PMs shall pursue international armaments cooperation to the 

maximum extent feasible, consistent with sound business practice 

and with the overall political, economic, technological, and national 

security goals of the United States. International agreements for 

international armaments cooperation programs shall complete 

the interagency consultation and Congressional notification 

requirements contained in 10 U.S.C. 2350a, Section 2751 of the 

Arms Export Control Act, and 10 U.S.C. 2531.”

— DoDD 5000.1 (Para E1.1.1)
DAU PROGRAM MANAGERS TOOL KIT

17

INTERNATIONAL ACTIVITIES ASSOCIATED WITH 

DEFENSE ACQUISITION PHASES

Cooperative Production

Coproduction

 Licensed Production

 Production Sharing

Foreign Military Sales 

Technology Opportunities

and User Capability Needs

NATO Forums

DEAs/IEPs

Staff Talks

S&E Exchanges 

Cooperative Development

International Testing

Concept Refinement

and

Technology Development

System Demonstration

of SDD Phase

Production

and

Deployment,

Sustainment

DEFENSE SALES vs. COOPERATIVE ACQUISITION

Legend:

DEA—Data Exchange Agreement

IEP—Information Exchange Project

S&E—Science and Engineering

They are Different

• Defense Sales

– Any Nation

– U.S. Contracts (FMS)

– U.S. Manages (FMS)

– Production and Support

– Dept. of State or Dept. 

of Commerce + DoD 

– USD(Policy)

– Foreign Initiated

– Foreign Funds (or U.S.

Credit/Grants)

• Cooperative Acquisition

– Allied or Friendly

– U.S., Ally or NATO

– Jointly Managed

– All Acquisition

– DoD – USD(AT&L) + 

Dept. of State and 

Dept. of Commerce

– U.S. and/or Foreign

– Foreign + U.S. Funds
DAU PROGRAM MANAGERS TOOL KIT

18

Resource 

Allocation 

Process—Overlap

FY08FY09FY10FY11FY12

CY08 CY09 CY10

Execution

Programming &

Budgeting Planning

FY09 

FY08 & prior 

FY09 & prior 

FY10 FY10 & prior 

GDF/JPG* FY11-15

POM FY12-17

Budget FY12-13

FYDP Changes FY11-15

Budget Changes FY11

FY11 FY11 &

prior

GDF/JPG FY12-17

GDF/JPG*

FY10-15

POM FY10-15

Budget FY10-11

*GDF SECDEF option in off year

J F M A M J A S O N D J J F M A M J A S O N D J J F M A M J A S O N D J

Programming &

Budgeting Enactment Execution Planning

Enactment Execution

Programming &

Budgeting Enactment Planning Execution
DAU PROGRAM MANAGERS TOOL KIT

19

Planning, Programming, Budgeting, and 

Execution (PPBE)—PLANNING PHASE

Legend:

CJCS—Chairman of the Joint Chiefs of Staff

CPR—Chairman’s Program Recommendation

GDF—Guidance for Development of the Force

JPG— Joint Programming Guidance

NMS—National Military Strategy

NDS—National Defense Strategy

NSS—National Security Strategy

QDR—Quadrennial Defense Review

PPBE—On-year program/budget review

SEP FEB MAR APR/MAY

SECDEF

QDR

Joint Staff, OSD, Unified 

Commands, MilDeps,

Defence Agencies, etc. ... 

NMS CPR

GDF JPG

SECDEF

As needed

President

National Security

Council

CJCS Unified Commands

Military Depts/Def Agencies

NSS

SECDEF Joint Staff/OSD

NDS

Fiscal

Guidance

1st Year 2nd Year

NOTE: GDF replaces SPG

BES MBI PB

Questions/Hearings

AUG OCT FEB

Issue Papers/Resolution

POM

OSD/

OMB

CPA

NOV DEC

CJCS

OSD/

OMB

Updates

FYDP

Updates

FYDP

PBDs

SECDEF/

DEPSECDEF

Components (PEO/PM) 

Answer/Reclama

PDM

Components

Military Depts

Defense Agencies

Unified Commands

(DAWG, 3 Star Group)

Legend:

BES—Budget Estimate Submission 

COCOM—Combatant Commander

CPA—Chairman’s Program Assessment 

DAWG—Deputies Advisory Working Group

FYDP—Future Years Defense Program

MBI—Major Budget Issues

PB—President’s Budget

PBD—Program Budget Decision

PEO/PM—Program Executive Officer/

Program Manager

PDM—Program Decision Memorandum

POM—Program Objectives Memorandum
DAU PROGRAM MANAGERS TOOL KIT

20

Resource Allocation Process

PPBE—Off-Year program / BUDGET

REVIEW (Odd year)

(e.g., FY 09–13 (Aug 07—Feb 08))

• No Program Objectives Memorandum submissions

• Focus priority on FY 09

• Program of Record (POR) remains FY 08 President’s Budget

• Minimize programmatic changes

• Change Proposal (CP) is vehicle to request changes to POR

• No FYDP update until FY 09 PB

• Complete Budget Estimate Submission provided to the Office of 

the Undersecretary of Defense (Comptroller) must incorporate all 

baseline changes

• Program/Budget Review Process

– 3-Star Group oversees CP review

– Deputy’s Advisory Working Group considers major issues and 

advises SECDEF

– Budget issues reviewed and coordinated through Program 

Budget Decision (PBD) process

– SECDEF makes final resource decisions

– Approved changes and decisions documented in Program 

Decision Memorandums and PBDs

DoD

President

& OMB

Congress

Authorization

Committees

Appropriation

Committees

Testimony

Authorization/

Appropriation

Acts Passed

Phase I:

PPBE

President’s

Budget 

Allocation/

Execution

Phase IV:

Phase III:

Phase II: Enactment

Budget

Committees

Appeals

Apportionment
DAU PROGRAM MANAGERS TOOL KIT

21

CONGRESSIONAL ENACTMENT TIMETA

BLE

15 15

^

^

^

BUDGETrESOLUTIONAUTHOrIZATION

APPrOPrIATION

JAN fEB APr MAr MAY JUN JUL AUG SEP OCT

“TArGET”

fISCAL

YEAr

PrESBUDGET

COMMITTEEINPUT

S

HOUSE

SENATE

HOUSE

SENATE

ACT

CONf/

rEPT

CONf/

rEPT

fLOOr

fLOOr

fLOOr

fLOOr

fLOOr

HrNGS MArKS fLOOr

BILL/rPT

BILL/rPT

BILL/rPT

BILL/rPT

HOUSE

SENATE

 rESOLUTION CONf

HrNGS MArKS fLOOr

HASC HrNGS MArK-UP fLOOr

SASC HrNGS MArK-UP fLOOr

HAC HrNGS MArK-UP fLOOr

SAC HrNGS MArK-UP fLOOr

ACT

^

fLOOr
DAU PROGRAM MANAGERS TOOL KIT

22

PROCUREMENT APPROPRIATIONS

(Account Numbers and Budget Activities)

Appropriation	 Budget Activity

Army (21 -)

Aircraft - 2031 1 Aircraft

2 Modification of Aircraft 

3 Spares and Repair Parts 

4 Support Equipment and Facilities

Missile - 2032 1 Not Used

2 Other Missiles 

3 Modification of Missiles

4 Spares and Repair Parts

5 Support Equipment and Facilities

Weapons and Tracked - 2033 1 Tracked Combat Vehicles 

Combat Vehicles 2 Weapons and Other Combat Vehicles 

3 Spares and Repair Parts

Ammo - 2034 1 Ammo 

2 Ammo Production Base Support

Other - 2035 1 Tactical and Support Vehicle 

2 Communications and Electronics 

3 Other Support Equipment 

4 Initial Spares

Navy (17 -)

Aircraft - 1506 1 Combat Aircraft 

2 Airlift Aircraft 

3 Trainer Aircraft 

4 Other Aircraft 

5 Modification of Aircraft 

6 Aircraft Spares and Repair Parts 

7 Aircraft Support Equipment and Facilities

Weapons - 1507 1 Ballistic Missiles 

2 Other Missiles 

3 Torpedoes and Related Equipment 

4 Other Weapons 

5 Not Used 

6 Spares and Repair Parts

Ammo, Navy and - 1508 1 Ammo, Navy

Marine Corps 2 Ammo, Marine Corps

Shipbuilding - 1611 1 Not Used

and Conversion 2 Other Warships

3 Amphibious Ships 

4 Not Used 

5 Auxiliaries, Craft, and Prior-Year Program Costs

Other - 1810 1 Ships Support Costs

2 Communications and Electronics Equipment 

3 Aviation Support Equipment 

4 Ordnance Support Equipment 

5 Civil Engineering Support Equipment

6 Supply Support Equipment
DAU PROGRAM MANAGERS TOOL KIT

23

PROCUREMENT APPROPRIATIONS

(Account Numbers and Budget Activities) (Continued)

Other (continued) -1810 7 Personnel and Command Support Equipment 

8 Spares and Repair Parts

Marine Corps (17 -)

Procurement - 1109 1 Not Used

2 Weapons and Combat Vehicles

3 Guided Missiles and Equipment

4 Communications and Electronics Equipment

5 Support Vehicles

6 Engineering and Other Equipment

7 Spares and Repair Parts

Air Force (57 -)

Aircraft - 3010 1 Combat Aircraft 

2 Airlift Aircraft 

3 Trainer Aircraft 

4 Other Aircraft

5 Modification of In-Service Aircraft

6 Aircraft Spares and Repair Parts 

7 Aircraft Support Equipment and Facilities

Missile - 3020 1 Ballistic Missiles

2 Other Missiles

3 Modification of In-Service Missiles

4 Spares and Repair Parts

5 Other Support

Ammo - 3011 1 Ammo

2 Weapons

Other - 3080 1 Not Used

2 Vehicular Equipment

3 Electronics and Telecommunications Equipment

4 Other Base Maintenance and Support Equipment

5 Spares and Repair Parts

Defense (97 -)

Defense-wide - 0300 1 Major Equipment

2 Special Operations Command

3 Chemical/Biological Defense

National Guard - 0350 1 Reserve Equipment

and Reserve 2 National Guard Equipment

Equipment 

Defense Production - 0360 1 Defense Production Activity Purchases

Activity

Purchase

Chemical Agents - 0390 1 Chemical Agents and Munitions Destruction—O&M

and Munitions 2 Chemical Agents and Munitions Destruction—RDT&E

Destruction 3 Chemical Agents and Munitions Destruction—

 Procurement

Rapid Acquisition Fund -2095 1 Rapid Acquisition Fund

Appropriation	 Budget Activity
DAU PROGRAM MANAGERS TOOL KIT

24

RDT&E APPROPRIATIONS

Relationship Between MFP 6 R&D Categories and

RDT&E Appropriations Budget Activities

RDT&E APPROPRIATIONS

(Account Numbers)

MFP 6 RDT&E

R&D Budget RDT&E

Category Activity Budget Activity Title

6.1 BA 1 Basic Research 

6.2 BA 2 Applied Research

6.3 BA 3 Advanced Technology Development

6.4 BA 4 Advanced Component Development and 

Prototypes

6.5 BA 5 System Development and Demonstration

6.6 BA 6 RDT&E Management Support 

 --- BA 7 Operational System Development

*NOTE: Although similar, titles of the Major Force Program (MFP) six categories (which 

are not shown above) are not exactly the same as titles of the RDT&E Appropriation 

Budget Activities. In addition, the “Operational System Development” Budget Activity for 

RDT&E BA 7 is not considered MFP 6. While correctly funded with RDT&E dollars, these 

efforts do not fall under a MFP 6 Category; rather, for MFP purposes, the efforts are considered part of the Major Force Program that the fielded operational system falls within. 

Account 

Appropriation Number

RDT&E, Army 21 - 2040

RDT&E, Navy 17 - 1319

RDT&E, Air Force 57 - 3600

RDT&E, Defense-wide 97 - 0400

Development T&E, Defense 97 - 0450

Operational T&E, Defense 97 - 0460

Legend:

BA—Budget Activity

MPF—Major Force Program

R&D—Research and Development

RDT&E—Research, Development, Test and 

Evaluation

T&E—Test and Evaluation
DAU PROGRAM MANAGERS TOOL KIT

25

SAMPLE NAVY APPROPRIATIONS AND

BUDGET ACTIVITIES

*Below Threshold Reprogramming (BTR) amount limits are cumulative over entire period of time the

 specific fiscal year appropriation is available for obligation purposes (i.e., 1, 2, 3, or 5 years).

**Reference Source: USD(C) Memo: Subject: FY 2006 Below Threshold Reprogramming Authority Policy, 10 Feb 2006 

PROCUREMENT (Proc.)

SCN-1 Not Used $20M** Lesser of 5 Full

SCN-2 Ship Conversion—Other Warships $20M or

SCN-3 Ship Conversion—Amphibious Ships 20% of

SCN-4 Not Used amount

SCN-5 Ship Conversion—Auxiliaries, Craft, and appropriated

Prior-Year Program Costs

WPN-1 Weapons Proc—Ballistic Missiles 3

WPN-2 Weapons Proc.—Other Missiles

WPN-3 Weapons Proc.—Torpedos and Equipment

WPN-4 Weapons Proc.—Other Weapons

WPN-5 Not Used

WPN-6 Weapons Proc.—Spares and Repair Parts 

OPN-1 Other Proc.—Ship Support Equipment (SE)

OPN-2 Other Proc.—Comm./Electronics Equip.

OPN-3 Other Proc.—Aviation SE

OPN-4 Other Proc.—Ordnance SE

OPN-5 Other Proc.—Civil Engineering SE

OPN-6 Other Proc.—Supply SE

OPN-7 Other Proc.—Pers. and Command SE

OPN-8 Other Proc.—Spares and Repair Parts

APN-1 Aircraft Proc.—Combat Aircraft

APN-2 Aircraft Proc.—Airlift Aircraft

APN-3 Aircraft Proc.—Trainer Aircraft

APN-4 Aircraft Proc.—Other Aircraft

APN-5 Aircraft Proc.—Modifications of Aircraft

APN-6 Aircraft Proc.— Aircraft Spares and Repair Parts

APN-7 Aircraft Proc.— Aircraft SE and Facilities

 

6.1 BA 1 Basic Research $10M** Lesser of 2 Incremental

6.2 BA 2 Applied Research $10M or 

6.3 BA 3 Advanced Tech. Development 20% of

6.4 BA 4 Adv. Comp. Dev. and Prototypes amount

6.5 BA 5 System Devel. and Demo. appropriated

6.6 BA 6 RDT&E Management Support 

 (T&E Ranges) (Civilian Salaries)

BA 7 Operational Systems Devel.

 (Post-Production)

Procurement Below Threshold Years Available

Budget Budget Activity Reprogramming Rules for Obligation Funding

Activity Description Max In Max Out Purposes Policy

 (At Line Item Level)

MFP 6 Below Threshold Years Available 

 R&D RDT&E Budget Activity (BA) Reprogramming Rules For Obligation Funding

Category Number and Title Max In* Max Out* Purposes Policy 

 (At Prog. Element Level)

 Below Threshold Years Available

eprogramming Rules for Obligation Funding

Other Appropriations / Titles Max In Max Out Purposes Policy

O&M, N Operations and Maintenance $15M No Congressional 1 Annual

 Restriction 

MILPER, N Military Personnel $10M No Congressional 1 Annual

 Restriction 

MILCON, N Military Construction Lesser of No Congressional 5 Full

 +$2.0M or 25% Restriction 

 Appropriated
DAU PROGRAM MANAGERS TOOL KIT

26

BELOW THRESHOLD REPROGRAMMING ACTIONS

APPN

RDT&E

PROC

O&M

MILPERS

MILCON 

MAX OUT

Lesser of

–$10M

20%

Lesser of 

–$20M 

–20% 

None, Unless

Otherwise

Specified 

No Specific

Congressional

Restriction

No Specific

Congressional

Restriction

OBLIGATION

AVAILABLE

Line Item

3 Years

(Shipbuilding and

Conversion: 

5 Years) 

1 Year

1 Year

5 Years

Amounts are Cumulative Over Entire Period of

Obligation Availability

APPROPRIATION LIFE

Current Period: Available for new obligations, obligation adjustments, expenditures, and outlays

O&M

RDT&E

Procurement

ShipS

MILCON

MILPERS

YEARS

Expired Period: Available for obligation adjustments, expenditures, and outlays

Cancelled: Unavailable for obligations, obligation adjustments, expenditures, and outlays

* RDT&E changed to $10M and Procurement changed to $20M for FY 03, FY 04, FY 05 per OSD Comptroller.

MAX OUT

Lesser of

Line Item

Budget Activity (BA)

Some Ba 1 Sub-activity

Limitations On Decreases

(Operating Forces)

Budget Activity 

Project

MAX INTO

+$10M*

+$10M*

+$15M

+$10M

Lesser of

+$2M 

+25%

1 2 3 4 5 6 7 8 9 10 11 
DAU PROGRAM MANAGERS TOOL KIT

27

IF . . . 

THEN . . . 

AND . . . 

Fund Purchase of the Mod Kits and 

Installation of those Mod Kits on the 

Fielded System with . . . 

LIFE CYCLE COST COMPOSITION

Life Cycle Cost

PRODUCT IMPROVEMENTS

Funding Decision Tree

Does Proposed

Modification (Mod)

Increase System 

Performance?

Is System

Currently In

Production? 

NO 

Procurement $ O&M $

NO

YES YES

Fund Development 

and Testing with . . . 

(To Include the Mod 

Kits used for Testing)

Is DT or IOT&E 

Required?

Procurement $

YES

RDT&E $ 

NO

RDT&E

Development Costs of

PME and Support Items

Systems Engineering

Program Management

Test and Evaluation

PROCUREMENT

Prime

Equipment

Flyaway Cost

Development Cost

Procurement Cost

PROCUREMENT

Initial Spares

MILCON

Facilities

Operations and

Support

• O&M , MILPERS

Disposal

• O&M (or others

 as appropriate

PROCUREMENT

Support Items

Weapon System Cost

Program Acquisition Cost
DAU PROGRAM MANAGERS TOOL KIT

28

Estimate Methods Comments

Analogy Comparison to one similar existing system; 

 based on judgments. Little or no data available; 

 relatively quick, easy, flexible. Used in early 

 phases (e.g., Concept Refinement and Tech. Dev.).

Parametric Comparison to many similar existing systems;

 based on statistical analysis. Determine 

 primary cost drivers and establish Cost

 Estimating Relationships (CERs). Used in early 

 to mid-phases (e.g., Concept Refinement and 

 Tech. Dev., and System Dev. and Dem.).

Engineering or Summation of “all” individual items in the system.

“Bottoms-Up” Uses Work Breakdown Structure (WBS) 

 for estimating purposes. Used in mid-phases 

 (e.g., System Dev. and Dem.). 

Extrapolation Comparison to historical cost of same system. 

 Based on extrapolation from actuals. Uses 

 Learning Curve Theory. Used in late phases 

 (e.g., production and replenishment spares).

COST ESTIMATING

Guidelines

1. Make sure cost data are relevant and homogeneous. Caution: Watch out for 

historical data in times of change. Prior actuals may include uncompensated 

overtime or were priced as a “buy-in.”

2. Focus on cost drivers.

3. Test sensitivities and data relationships.

Cost Estimating Relationships

(CER)—Parametric

Regression Line

Cost ($)

Predicted

Cost with

Parameter

(size)

Parameter

(e.g., size, wt., etc.)

= Similar Systems
DAU PROGRAM MANAGERS TOOL KIT

29

COST ESTIMATING REQUIREMENTS

POE

CARD

CCA

ICE

ACAT IC and ID

Program initiation and all subsequent 

milestones, including FRP DR 

Required same as ICE or CCA 

• Draft 180 days prior to OIPT/milestone 

• Draft 180 days prior to OIPT/milestone

Usually serves as ICE for ACAT IC 

programs 

• Prepared by component cost agency 

(AFCAA, DASA-CE, NCCA)

• Program initiation and all subsequent 

milestones including FRP DR

• Prepared at MDA discretion for other 

programs

Required by law for all MDAP programs *

• Prepared by OSD CAIG for

ACAT ID and ACAT IC at discretion

of USD(AT&L)

• Program initiation and all subsequent 

milestones including FRP DR

ACAT IAM and IAC

Program initiation and all subsequent 

milestones

• Includes Economic Analysis

Required same as CCA

• Draft 180 days prior to OIPT/milestone 

• Final 45 days prior to OIPT/milestone

Program initiation and whenever 

economic analysis is required

*ICE statutory requirement (Title 10, US Code, Sec 2434), Source: DoDI 5000.2

AFCAA—Air Force Cost Analysis Agency

CAIG—Cost Analysis Improvement Group

CARD—Cost Analysis Requirements 

Description

CCA—Component Cost Analysis

DASA-CE—Dep Asst Sec of Army (Cost 

and Economics)

FRP DR—Full Rate Production Decision 

Review

ICE—Independent Cost Estimate

MDA—Milestone Decision Authority

MDAP—Major Defense Acquisition Program

NCCA—Naval Center for Cost Analysis

OIPT—Overarching Integrated Product Team

POE—Program Office Estimate

USD(AT&L)—Under Secretary of 

Defense (Acquisition, Technology, and 

Logistics)

ACAT II and ACAT III: POE (and CCA and MDA discretion)

at program initiation and all subsequent milestones.

Legend:
DAU PROGRAM MANAGERS TOOL KIT

30

Program 

Cost and 

S

c

hedule Breac

h

Parameters

ScheduleMilestonesCostReports Required

if Breach Occurs

SAR

(Title 10, US Code, Sec 2432)

6 month slip from previous SAR15/30% PAUC Growth(Current/Original)15/30% APUC Growth(Current/Original)Quarterly SAR

Unit Cost Report (UCR)

(Title 10, US Code, Sec 2433) (Nunn-McCurdy)

N/A15/30% PAUC Growth(Current/Original)15/30% APUC Growth(Current/Original)Quarterly/Exception DAES 

Service notifies Congress (>15/30%) SecDef Certification (>25/50%)

AP

B

(Title 10, US Code, Sec 2435)

Slip beyond threshold date

* Default threshold = objective plus 6 

months; MDA discretion

15%/30% PAUC Growth 15%/30% APUC Growth(Current/Original)* RDT&E Growth* Procurement Growth* MilCon or O&M Growth* 10% default; MDA discretionProgram Deviation Report

Applicable to Major Defense Acquisition Programs (MDAPs)

ACAT II and III: 15%/30% PAUC/APUC APB breach parameters are N/A; actual breach parameters at MDA discretion

Legend :APUC—Average Procurement Unit Cost APB—Acquisition Program BaselinePAUC—Program Acquisition Unit Cost SAR—Selected Acquisition Report DAES—Defense Acquisition Executive Summary

Critical Cost Growth (APB/SAR/UCR)PAUC or APUC increases of 25% of current baseline or 50%of original baseline requires Sec Def certification to Congress: -- Essential to national defense -- No alternatives  -- Cost are under control -- Management is in place to keep costs under control

Significant Cost Growth (APB/SAR/UCR)PAUC or APUC increases of 15% of current baseline or 30% of original baseline is “reportable” to Congress
DAU PROGRAM MANAGERS TOOL KIT

31

PERFORMANCE MEASUREMENT

COST and SCHEDULE PERFORMANCE MEASUREMENT

1. Define the work 

2. Schedule the work

3. Allocate budgets }

5. Prepare and monitor performance profiles:

4. Defining, Planning and Budgeting:

Element/Cost Account—300

Work Packages Planning Packages

(6-month coverage) (remainder of effort)

$45

Task A 

 

Task B

Task C

Tasks D-X

$10

$ 3

$ 4

BCWP

ACWP

BCWP

BCWS

 BCWP (cum)

BAC

 ACWP (cum) 

BAC

PERFORMANCE INDICES

Cost Performance Index CPI =

Schedule Performance Index SPI =

Percent Complete =

Percent Spent =

=

ESTIMATE AT COMPLETION

 EAC BAC

(Lowest Estimate) CPI(cum)

 EAC = ACWP(cum) +

(Highest Estimate)

BAC - BCWP(cum) 

EAC - ACWP(cum)v

BAC – BCWP(cum) 

{CPI(cum) X SPI(cum) }

TCPI(eac) =

TO COMPLETE PERFORMANCE

INDICES

WBS

CBB

C

O

S

T

ACwP NOw

BCwS

BCwP

COST VArIANCE

PMB

BAC

EAC

TAB

Mr }

SCHEDULE

VArIANCE

}}

TIME

BCWP - ACWP

 BCWP 

BCWP - BCWS

 BCWS

VARIANCES

Cost Variance CV = BCWP - ACWP

Schedule Variance SV = BCWP - BCWS

Cost Variance % CV% = 

Schedule Variance % SV% =

Variance at

Completion VAC = BAC - EAC

Legend:

ACWP—Actual Cost of Work Performed

AUW—Authorized Unpriced Work 

BAC—Budget at Completion 

BCWP—Budgeted Cost of Work Performed 

BCWS—Budgeted Cost of Work Scheduled

CBB—Contract Budget Base (NCC+AUW) 

CPI—Cost Performance Index

CV—Cost Variance

EAC—Estimate at Completion (Government) 

Cost Account

MR—Management Reserve

NCC—Negotiated Contract Cost

PMB—Performance Measurement Baseline

SPI— Schedule Performance Index

SV— Schedule Variance

TAB—Total Allocated Budget

TCPI—To Complete Performance Indices

VAC—Variance at Completion

WBS—Work Breakdown Structure
DAU PROGRAM MANAGERS TOOL KIT

32

TYPES OF CONTRACTS

Cost Family—Appropriate when product not well defined; high risk; contractor provides 

best efforts; Government pays all allowable costs. Fee varies by type.

CPFF—Fee same regardless of actual cost outcome.

CPIF—Actual fee earned computed by applying share ratio to over/under run, subject to 

min/max fee limits.

Fixed Price Family—Product well defined, low risk; contractor must deliver product.

FFP—Price fixed regardless of actual cost incurred.

FPI(F)—Final price computed by applying share ratio to over/underrun, subject to ceiling 

price limitation.

AF—Either stand-alone Cost Plus Award Fee (CPAF) or combined with cost or fixed 

price types. AF unilaterally determined by government based on subjective evaluation of 

contractor’s performance.

Fee Limits: CPFF—Fee limited to 15% for R&D; 10% for production and services. No 

statutory or FAR/DFARS regulatory limitation on other contract types.

CONTRACTING—COMPONENTS OF

CONTRACT PRICE

Contract Price = Cost + Profit/Fee

Other

Direct Cost

Subcontracts

TYPICAL CONTRACT TYPE BY PHASE

 CR TD SDD/SI SDD/SD PROD

 CPFF, FFP CPFF,FFP CPFF, CPIF CPIF, CPAF FPI(F), FFP

FCCM

Engineering

Labor

Manufacturing

Labor

Raw

Material

Purchased

Parts Engineering

Support

Manufacturing

Support

Material

Handling Standard

Comm

Items Interdivisional

Transfers

G&A

Direct Cost Indirect Cost

Direct Labor Direct Material Overhead

Legend:

AF—Award Fee 

CPAF— Cost Plus Award Fee

CPFF— Cost Plus Fixed Fee

CPIF— Cost Plus Incentive Fee

CR—Cost Reimbursement

FAR/DFARS— Federal Acquisition Regulation/ 

 Defense FAR Supplement

FCCM— Facilities Capital Cost of Monies

FFP— Firm Fixed Price

FPI(F)—Fixed Price Incentive (Firm Target)

ODC—Other Direct Cost

G&A—General and Administrative (Expense)

PROD—Production

R&D—Research and Development 

SD—System Development

SDD— System Development and Demonstration

SI— System Integration

TD— Technology Development
DAU PROGRAM MANAGERS TOOL KIT

33

CONTRACT TYPE FEATURES

Promise

Contractor Risk

Cash Flow

Progress Payments %

Administration

Fee Limit %

FIXED

PRICE

Delivery

High

Delivery

75/90/95

Low

None 

COST

REIMBURSEMENT

Best Efforts

Low

As Incurred

N/A

High

15/10/6 on CPFF

PRICE = COST + FIXED FEE

Cost Plus Fixed Fee (CPFF)

FEE

COST TARGET COST

MAX

TARGET

MIN

Cost Plus Incentive Fee (CPIF) 

Share

 Ratio

FEE

FIXED

FEE

(SHARE 100/0)

COST     ESTIMATED COST

• Risk Highest to the Government

• Obtains Fee Regardless of Cost 

(Target) PRICE = (Target) COST + (Target) FEE

• All Reasonable Cost Paid

• Shared Risk Between Min/Max Fee 
DAU PROGRAM MANAGERS TOOL KIT

34

CONTRACT TYPE FEATURES

(Continued)

Fixed Price Incentive (Firm Target) (FPI(F))

Firm Fixed Price (FFP)

PRICE = COST + PROFIT 

COST

0/100 SHARE

 PROFIT

COST TARGET COST

PTA

Share 

Ratio 

CEILING 

PRICE

PROFIT

Target 

Profit

Point of Total 

Assumption = CEILING PRICE – TARGET PRICE + Target Cost 

(PTA) GOVERNMENT SHARE 

(Target) PRICE = (Target) COST + (Target) Profit
DAU PROGRAM MANAGERS TOOL KIT

35

PRE-SOLICITATION PROCESS

POST-SOLICITATION PROCESS

LEGEND:

FBO/CBD – FedBizOps/Commerce 

Business Daily 

Market

Research

Acquisition

Strategy Requirement

Acquisition

Plan

Post Draft RFP

on Electronic

Bulletin Board

FBO/CBD Notice

Advisory Multi-Step

Finalize

RFP 

CBD Notice of

RFP

Release

RFP Release

to Industry

Source

Selection Plan/

Strategy

RFP Release

Briefing to

SSA

SSA

Decision

Receipt of Oral

and Written

Proposals 

Initial Eval Clarifications

Limited Communications

Competitive

Range

Determination

Face-to-Face

Discussions/

Negotiations

Receive and Analyze

Field Surveys 

(if requested)

Prepare for Discussions

with

Remaining Offerors

Debrief

Unsuccessful

Offerors

Contract Award

(Distribution) 

 Brief

SSAC

Request Final

Proposal

Revision

Receive and

Analyze 

Final Revision 

Brief

SSA

RFP – Request for Proposal

SSA – Source Selection Authority

SSAC – Source Selection Advisory Council
DAU PROGRAM MANAGERS TOOL KIT

36

• GSA Multiple Award Schedules (MAS)

– General Services Administration contracts for both products and 

services–available to all agencies.

• Government–Wide Agency Contracts (GWACs)

– Similar to MAS but more restricted in products and services 

available.

• Indefinite Delivery/Indefinite Quantity Contracts

– Task orders (services) and delivery orders (products) issued under 

omnibus umbrella contract. 

• Other Transactions (OT)

– Defined: Vehicles used for basic, applied and advanced research 

projects and prototype development. OTs are not contracts, grants, 

or cooperative agreements.

– Objective: Attract commercial companies and consortia that 

historically have not done business with the Department of 

Defense because of statutory and/or regulatory requirements. OTs 

are not subject to the Federal Acquisition Regulation. Designed to 

increase DoD access to dual-use technologies.

– Research Projects:

◆ Where practical, government cost share should not exceed cost 

share of other parties.

◆ Use OT when standard contract, grant, or cooperative 

 agreement is not appropriate.

– Prototype Projects:

◆ Must be directly relevant to weapons or weapon systems 

proposed to be acquired or developed by DoD.

– Constraints:

◆ At least one nontraditional contractor participating. 

◆ If no nontraditional contractor participates, 1/3 of cost paid by 

parties other than federal government or senior procurement 

executive justifies transaction.

– OT Guide for Prototype Projects, January 2001.

OTHER WAYS TO BUY
DAU PROGRAM MANAGERS TOOL KIT

37

CONTRACTOR PROFITABILITY RATIOS

The basic concept of profitability ratios is to measure net income against revenue or against 

the investment required to produce it. There are three principal profitability ratios with which you 

should be familiar. They are:

1. Return on Sales, which shows what percentage of dollars are left after the company 

has paid for all costs, interest, and taxes. It is expressed as:

Return on Sales = Net Income

 Sales 

2. Return on Total Assets, which looks at the efficiency with which management has used 

its resources, the company’s assets, to generate income. It is computed as:

ROA = Net Income

 Total Assets 

As noted, Return on Assets addresses how well management utilizes the assets of the firm in 

generating income. The ROA formula reflects the combined result of Return on Sales and the 

total asset turnover ratio (total sales/total assets), broken down as follows:

ROA = Net Income X Total Sales

 Total Sales Total Assets

3. Return on Stockholders’ Equity measures the rate of return on the owners’ 

investment—their equity in the company. This is also known as Return on Equity:

ROE = Net Income

 Stockholders’ Equity

ROE can also be broken into two components: return on assets and financial leverage (a ratio 

reflecting the relationship of creditor to owner financing—expressed as total assets/ stockholders 

equity). This is shown by:

ROE = Net Income X Total Assets

 Total Assets Stockholders’ Equity

These profitability ratios give three different viewpoints concerning the “bottom line” on the income 

statement—how much net profit is being made on each sale, how much is being made for the 

assets that are employed, and how much is being made for the company owners. Contractor 

profitability ratios for the aerospace/defense industry for the period of 1980 to date are shown 

on page 38.

From an owner’s perspective, another profitability ratio you may be aware of is Earnings Per 

Share:

EPS = 

Legend: EPS—Earnings Per Share ROA—Return on Assets ROE—Return on Equity

Net Income

Number of Shares of Common Stock Outstanding
DAU PROGRAM MANAGERS TOOL KIT

38

Aerospace/Defense Industry

Contractor Profitability Ratios

Aerospace/Defense Industry

Contractor Profitability Ratios

Return Asset Return Financial Return

On Sales Turnover On Assets Leverage On Equity

(NI/S) (S/TA) (NI/TA) (TA/SE) (NI/SE)

1980 4.3 1.21 5.2 3.08 16.0

1981 4.4 1.18 5.2 3.06 16.0

1982 3.3 1.12 3.7 3.24 12.0

1983 3.5 1.17 4.1 2.98 12.1

1984 4.1 1.15 4.7 3.00 14.1

1985 3.1 1.13 3.6 3.17 11.1

1986 2.8 1.07 3.1 3.13 9.4

1987 4.1 1.07 4.4 3.32 14.6

1988 4.3 1.02 4.4 3.39 14.9

1989 3.3 1.00 3.3 3.24 10.7

1990 3.4 1.00 3.4 3.38 11.5

1991 1.8 1.06 1.9 3.21 6.1

1992 -1.4 0.86 -1.2 4.33 -5.2

1993 3.6 0.97 3.5 3.80 13.2

1994 4.7 0.92 4.3 3.44 14.8

1995 3.8 0.92 3.5 3.17 11.1

1996 5.6 0.91 5.1 3.35 17.1

1997 5.2 0.92 4.8 3.60 17.3

1998 5.0 0.96 4.8 3.73 18.0

1999 6.5 0.95 6.2 3.52 21.8

2000 4.7 0.91 4.3 3.30 14.2

2001 3.9 0.92 3.6 3.22 11.6

2002 4.1 0.90 3.7 3.16 11.7

2003 3.1 0.84 2.6 3.81 9.9

 Average 3.8 1.0 3.8 3.4 12.7

Source: Aerospace Industries Association 

Legend:

NI/A—Net Income/Sales

NI/SE—Net Income/Stockholders’ Equity

NI/TA—Net Income/Total Assets

S/TA—Sales/Total Assets

TA/SE—Total Assets/Stockholders’ Equity
DAU PROGRAM MANAGERS TOOL KIT

39

XX

LOCK

HEED MARTIN CORP—Dupont 

Formula 

Analysis (An 

Example)

==

4.00%3.00%2.00%1.00%0.00%

-1.00%-2.00%-3.00%-4.00%-5.00%

Return on Sales (%)

(Net Income/Sales) 

1999 2002 2003 2000 2001

5.00%4.00%3.00%2.00%1.00%0.00%

-1.00%-2.00%-3.00%-4.00%-5.00%

Return on Assets (%)

(Net Income/Total Assets)

1999 2002 2003 2000 2001

5.0004.5004.0003.5003.0002.5002.0001.5001.0000.5000.000

Financial Leverage

(Total Assets/Shareholder Equity) 

1999 2000 2001 2002 2003

5.00%4.00%3.00%2.00%1.00%0.00%

-1.00%-2.00%-3.00%-4.00%-5.00%

Return on Assets (%)

(Net Income/Total Assets)

1999 2002 2003 2000 2001

Fiscal

Year

Total Asset

Turnover

Return

on Assets % 

Financial

Leverage

Return

on Equity 

%

Return on

Sales 

% =

X X 4.757 =X = X 4.249 =X = X 4.301 =X X 4.600 =

19992000200120022003

1.53%

-2.11%

-4.36%

1.88%

3.31% X

0.8260.8070.8660.9851.216

1.26%

-1.71%-3.77%

1.85%4.02%

1.26%

-1.71%-3.77%

1.85%4.02% X 3.874 =

6.01%7.25%

-16.23%

8.53%

15.59%

Return

on Assets % X =

===

X

20.00%15.00%11.00%10.00%

5.00%0.00%

-5.00%

-10.00%-15.00%-20.00%

Return on Equity (%)

(Net Income/Shareholder Equity) 

1999 2002 2003 2001 2000

1.41.21.00.80.60.40.2

0

Total Asset Turnover

(Sales/Total Assets) 

1999 2000 2001 2002 2003
DAU PROGRAM MANAGERS TOOL KIT

40

CASH CYCLE

— Delivery 

CONTRACTOR FINANCING AND PAYMENTS

*Internal Contractor Financing— Retained Earnings

— Govt. specified 

— Offer or proposed 

— Interim 

— Advance 

— Private

• Trade Credit

• Bank Credit 

— Revolving Credit 

— Term Loan 

— Government

— Periodic

— Partial

FINANCING (External*) PAYMENTS

Commercial Noncommercial Commercial Noncommercial

• For Noncommercial 

— Progress Payments

• Performance-based

• Cost Incurred-based

• % Complete 

–Unusual Progress Payments 

–Assignment of Claims 

–Guaranteed Loans 

–Advance Payments 

Cash

received

Accounts

receivable

Sale

(DD 250) Finished goods

inventory

Raw material

inventory

Contract

award

Cash

disbursed

Cash

disbursed

Wages

payable

Work in process

inventory

Accounts

payable
DAU PROGRAM MANAGERS TOOL KIT

41

Indirect costs

• Costs that can’t be traced

to a single contract

because they are

associated with

multiple contracts

• Example: electricity for

the company’s facilities

Direct costs

• Costs that can be

traced to a single

contract

• Examples: material

and labor to assemble

an aircraft

DIRECT AND INDIRECT COSTS

Indirect costs are

assigned to

contracts using

indirect rate(s).

(Traced directly)

(Traced directly)

Direct

material

Direct 

labor 

Contract

ASSIGNING INDIRECT COSTS

Calculation of Indirect Rates

Note: See DAU Indirect-Cost Management Guide at

<www.dau.mil/pubs/gdbks/icmguide.asp>.

INDIRECT RATE = Indirect Cost Pool

Allocation Base
DAU PROGRAM MANAGERS TOOL KIT

42

LIFE CYCLE of Indirect COST Rates

BIDDING ON

CONTRACTS

ADJUSTING

PAYMENT and

CLOSING

CONTRACTS

PAYING

CONTRACTS

FORWARD

PRICING

RATES

BILLING

RATES

ACTUAL

RATES

Direct material $ 40,000

Material handling 10% 4,000

Direct engineering labor 6,000

Engineering overhead 100% 6,000

Direct manufacturing labor 12,000

Manufacturing overhead 150% 18,000

Other direct costs 6,000

Subtotal 92,000

General and administrative 25% 23,000

Total cost 115,000

Profit 15% 17,250

Cost of money for facilities capital 1,500

Price $133,750

CONTRACTOR’S COST PROPOSAL

EXAMPLE
DAU PROGRAM MANAGERS TOOL KIT

43

CONTRACTOR BUSINESS PLANNING PROCESS

OUTPUTS

MANPOWER PLAN

Rates

for

Estimating

SALES

ANNUAL

OPERATING PLAN

CAPITAL

INVESTMENT PLAN

PRODUCTION and

ENGINEERING PLAN

MASTER DELIVERY 

SCHEDULE

IR&D/B&P PLAN

TOP MANAGEMENT

GOALS, OBJECTIVES,

STRATEGIES

OVERHEAD

FORECAST

BUSINESS BASE

SALES FORECAST

$ $

TREND
DAU PROGRAM MANAGERS TOOL KIT

44

System

Performance

Technical

Effectiveness

System

Effectiveness

System

Availability

Process

Efficiency

Capabilities

functions

Priorities

reliability

Maintainability

Supportability

Producibility

Operations

Maintenance

Logistics

Life Cycle Cost / Total Ownership Cost (TOC)/

Cost as an Independent Variable (CAIV)

Affordable

Operational

Effectiveness

◆ System Training

◆ System Documentation

◆ Supply Support (including Spares)

◆ Sustainment Planning

◆ Test and Support Equipment

◆ facilities

◆ Packaging, Handling, and Transportation

◆ Manpower

SYSTEM/PrODUCT SUPPOrT PACKAGE:

System Operational Effectiveness (SOE)

PRINCIPAL LIFE CYCLE LOGISTICS

GOALS/OBJECTIVES

LIFE CYCLE LOGISTICS

• The planning, development, implementation, and management 

of a comprehensive, affordable, and effective systems-support 

strategy, within Total Life Cycle Systems Management. Life cycle 

logistics encompasses the entire system’s life cycle including 

acquisition (design, develop, test, produce, and deploy), sustainment 

(operations and support), and disposal.

• Influence product design for affordable System Operational 

Effectiveness.

• Design and develop the support system utilizing PerformanceBased Logistics.

• Acquire and concurrently deploy the supportable system, including 

support infrastructure.

• Maintain/improve readiness, improve affordability, and minimize 

logistics footprint.
DAU PROGRAM MANAGERS TOOL KIT

45

Maintenance Planning – establishes maintenance concepts and requirements.

Manpower and Personnel – identification of personnel skills, grades, and 

quantity required to support operation and maintenance of system.

Supply Support – determine requirements to acquire and manage spares 

and repair parts.

Support Equipment – identify all equipment required to support operation 

and maintenance of the system.

Technical Data – scientific and technical information used to support systems acquisition.

Training and Training Support – determine requirements to acquire training 

devices and conduct training of operators and maintenance personnel.

Computer Resources Support – identification of facilities, hardware, 

software, and support tools to operate and support embedded computer 

systems.

Facilities – identify real property required to support system.

Packaging, Handling, Storage, and Transportation – identify designs and 

methods to ensure the system is preserved, packed, stored, handled, 

and transported properly.

Design Interface – relationships of logistics-related design parameters to 

readiness and support resources requirements; influence design for supportability.

SUPPORT ELEMENT DEFINITIONS

BEST PRACTICE:

Support Elements

Maintenance 

Planning

Manpower and

Personnel

Supply 

Support

Support

Equipment Training and 

Training Support

Technical

Data 

Design

Interface

Computer

Resources

Support

SYSTEM

Facilities

Packaging, Handling,

Storage, and

Transportation

Note: Under Performance-Based Logistics (PBL), support elements are still valid even though the

government is buying results and solutions, not specific resources or processes.
DAU PROGRAM MANAGERS TOOL KIT

46

LIFE

C

YCLE logistics LINKAGE TO DEFENSE AC

QUISITION PROCESS

Modifications &

Technology Insertion

Acquisition

Strategy

Support

Strategy

FRP DECISION REVIEW DESIGN READINESS REVIEW

User Needs & Technology Opportunities 

C IOC FOC

Sustainment Pre-Systems Acquisition

Sustainment Disposal

Produce Product

Produce

Support System

Readiness & Total

Ownership Cost

Objectives

Perfor

manceBased Logistics

(PBL)

Evaluate Alternative

Logistics

Concepts

Support

Concepts

IOC

Systems Acquisition

B A

INCREMENT 3 CONCEPT DECISION

SE Process SE Process

Evaluate

Product Support

Capabilities

Develop Initial

Product

Support Strategy

Product

Support Plan

Demo Product

Support

Capability

PBL & Product

Support

Implementation 

PBL & Product

Support

Management

Operations/

Sustainment

Single Step or

Evolution to Full

Capability 

Demil &

Disposal

Demonstrate

Supportability

& Life Cycle

Affordability

De

monstrate

Supportability

& Life Cycle

Affordability

Inputs to:

• CDD

• TEMP

• APB

• Contract

FACTORY

CLOSED

INCREMENT 2

Design Product

for Supportability

Design Develop

Support Syste

m

Post-Deployment

Support

Maintain/Improve

Readiness & Improve 

Affordability

GOAL 4

GOAL 1

OPERATIONS

& SUPPORT

PRODUCTION & DEPLOYMENT SYSTEM DEV & DEMONSTRATION

CONCEPT

REFINEMENT

TECHNOLOGY

DEVELOPMENT

SYSTEM

INTEGRATION

SYSTEM

DEMONSTRATION

LOW-RATE

INITIAL

PRODUCTION

FULL-RATE

PRODUCTION

& DEPLOYMENT

GOAL 2

Post-Deployment

Evaluations

Acquire & Deploy

Supportable System

GOAL 3
DAU PROGRAM MANAGERS TOOL KIT

47

Performance-Based Logistics (PBL)

• The purchase of support as an integrated, affordable, performance 

package designed to optimize system readiness and meet performance goals for a weapon system through long-term support 

arrangements with clear lines of authority and responsibility. 

• PBL is DoD’s preferred approach for product support 

implementation.

Performance-Based Logistics (PBL)

tool kit

PBL 12-Step Process Model

<https://acc.dau.mil/pbl>
DAU PROGRAM MANAGERS TOOL KIT

48

Sufficient Spares

and Repair Parts for

System Life?

Yes

Yes

No

No

Monitor

Usage Data

Multiple

Sources?

Evaluate

Alternatives

Monitor

Vendors

Sole Source

Reprocurement

Competitive

Reprocurement

Substitute

Parts

Component

Redesign

Life of Type

Buy

POST-PRODUCTION SUPPORT DECISION PROCESS

Logistics

 Down Time

(LDT)

Corrective 

Maintenance Time 

(CMT)

Administrative

Delay Time

(ADT)

Preventive 

Maintenance Time 

(PMT)

• Locating tools

• Setting up test 

equipment

• Finding personnel

 (trained)

• Reviewing manuals

• Complying with

supply procedures

• Parts availability

“in the bin”

• Needed items

awaiting 

transportation

• Preparation time

• Fault location time

• Getting parts

• Correcting fault

• Test and checkout

• Servicing

• Inspection

OPERATIONAL AVAILABILITY

Standby

Time

Operating

Time

Uptime

Downtime

AO = Uptime +
DAU PROGRAM MANAGERS TOOL KIT

49

Materiel Availability (Key Performance Parameter (KPP))

• A Key Data Element

• Used In Maintenance and Logistics Planning

• Average percentage of time entire population of systems is materially capable for 

operational use during a specified period

• Formula: Number of End Items Operational 

 Total Population of End Items

Materiel Reliability (Key System Attributes (KSA))

• Mean Time Between Failure (MTBF)

• Measure of How Often System Fails/Requires Repair

• Key Data Element In Forecasting Maintenance/Logistics Needs

• Formula: Total Operating Hours

 Total Number of Failures

Ownership Cost (KSA)

• O&S costs associated with materiel readiness

• Focused on Sustainment of the System

• Essential Metric For Sustainment Planning and Execution 

• Useful For Trend Analyses

• Supports Design Improvements/Modifications

• Uses CAIG O&S Cost Estimating Structure Selected Cost Elements

Mean Downtime

• Measure of How Long a System Will Be Unavailable After a Failure

• Used In Maintenance/Logistics Planning Process

• Formula: Total Down Time for All Failures 

 Total Number of Failures

life cycle sustainment metrics

(CJCS 3170, 1 May 2007)

life cycle sustainment OUTCOME ENABLERS <https://acc.dau.mil/log>

• Performance Based Logistics (PBL)—<https://acc.dau.mil/pbl> 

• Corrosion Prevention—<https://acc.dau.mil/corrosion> 

• Item Unique Identification (Iuid)/serialized Item Management (SIM)—<https://acc.dau.mil/uid> 

• Technical Data/IETM

— Interactive Electronic Technical Manuals (IETM)—<https://acc.dau.mil/ietm> 

— Data Management—<https://acc.dau.mil/dm> 

• Condition Based Maintenance (CBM+)—<https://acc.dau.mil/cbm>

— Prognostics and Diagnostics—<https://acc.dau.mil/phm>

— Reliability Centered Maintenance (RCM)—<https://acc.dau.mil/rcm>

• Continuous Process Improvement (CPI)—<https://acc.dau.mil/cpi-lean> 

• Title 10 Requirements/ 50/50, Partnering

— 50/50—<https://acc.dau.mil/depot>

— Partnering—<https://acc.dau.mil/ppp>

• Depot Maintenance Plan—<https://acc.dau.mil/depot>

• Diminishing Manufacturing Sources and Material Shortages (DMSMS)/Obsolescence 

Planning

— Diminishing Manufacturing Sources and Material Shortages (DMSMS)—

<https://acc.dau.mil/dmsms>

— Aging Systems—<https://acc.dau.mil/agingsystems>

— Obsolescence Management—<https://acc.dau.mil/obsolescence>

— Continuous Modernization—<https://acc.dau.mil/modernization>

— Technology Insertion—<https://acc.dau.mil/techinsertion>

— Lead Free Electronics/Solder <https://acc.dau.mil/leadfree>

• Training—<https://acc.dau.mil/training>

• Integrated Supply Chain Management (SCM)—<https://acc.dau.mil/scm>

• Radio Frequency Identification (RFID)—<https://acc.dau.mil/rfid> 

• Predictive Modeling—<https://acc.dau.mil/m&s>

• Long-term Performance Based Agreements (PBA)—<https://acc.dau.mil/pbar>

Mean

Down

TimeMateriel Reliability

Materiel Availability

Ownership Cost
DAU PROGRAM MANAGERS TOOL KIT

50

Logistics Management 

Community of Practice (LOG CoP)

Transforming the Way We Work

Where YOU can ....

Find Helpful Tools and Templates

• Latest PBL Resources

• Supportability Best Practices

• Contracting Lessons Learned

Get Ahead In YOUR Career

• Logistics Training and Education

• Latest OSD Policy and Direction

• Logistics Conferences/Events

• Link to Top DoD Web sites

Connect With Professionals

• Share Experiences and Ideas

• Start and Join Discussions

• Locate DoD and Industry Experts

<http://acc.dau.mil/log>

DEFENSE ACQUISITION GUIDEBOOK

LIFE CYCLE LOGISTICS

<https://akss.dau.mil/dag>
DAU PROGRAM MANAGERS TOOL KIT

51

“Pure” Product Structure

PROGRAM OFFICE ORGANIZATION STRUCTURES

(Examples)

“Traditional” or Functional Structure

Note: Functional divisions shown are notional.

Note: Functional divisions shown are notional.

PM

Staff

Engineering Logistics Production Business/

Finance

Staff

Staff

Engr

Functional

Divisions

Engr Log Bus

Functional

Divisions

Engr Log Bus

Functional

Divisions

Log Bus

Staff Staff

Product/

Project

Manager

System A

Product/

Project

Manager

System B

Product/

Project

Manager

System C

PM

LEGEND: Engr—Engineering Log—Logistics Bus—Business
DAU PROGRAM MANAGERS TOOL KIT

52

PROGRAM OFFICE ORGANIZATION STRUCTURES

(Continued)

PEO

Staff

PM

Program A

SYSCOM 

Head

Functional

Directors

Engineering Logistics Production Bus/Fin

PM

Program B

PM

Program C

Note: Functional divisions shown are notional.

Staff

Staff

DPM

Program Manager

APMs Log Eng Test Prod Bus/Fin Contracts

Primary

Vehicle Logistics SystemsEngineering

Test & 

Evaluation

Frame Engine/Drive

Train

Suspension 

& Steering

Fire Control

Level 1

IPT

Level 2

IPTs

Level 3

IPTs

Note 1

Note 1: Functional titles shown are notional.

Note 2

Note 2: IPTs often align with WBS elements.

Matrix Structure

Integrated Product Teams
DAU PROGRAM MANAGERS TOOL KIT

53

THE ROLE OF MANUFACTURING IN THE

ACQUISITION PROCESS

SUSTAINMENT

FULL

RATE CR TD LRIP DEMONSTRATION

 INTEGRATION

A B C

• Execute the Manufacturing Plan

– Reflect Design Intent

– Repeatable Processes

• Continuous Process Improvement

PRODUCTION

• Influence the Design

Process

• Prepare for Production

RDT&E

• Producible Design

• Factory Floor 

Characterized

MILESTONE C EXIT 

REQUIREMENTS

CURRENT DoD 5000 PROCESS

• Unstable requirements/engineering changes

• Unstable production rates and quantities

• Insufficient process proofing

• Insufficient materials characterization

• Changes in proven materials, processes, subcontractors, 

vendors, components

• Producibility

• Configuration management

• Subcontractor management

• Special tooling

• Special test equipment

COMMON PRODUCTION RISKS THAT GREATLY 

IMPACT COST, SCHEDULE, and PERFORMANCE

• Uniform, Defect-Free 

Product

• Consistent Performance

• Lower Cost

NET RESULT:
DAU PROGRAM MANAGERS TOOL KIT

54

PRODUCIBILITY

DEFINITION:

The measure of relative ease of manufacturing a product. The product 

should be easily and economically fabricated, assembled, inspected, 

and tested with high quality on the first attempt that meets performance thresholds.

Producibility Issues:

• Design engineering, NOT manufacturing, is the technical group 

responsibility for producibility. Program offices and design engineers 

often dislike producibility because it usually requires performance 

functionality sacrifices (especially if cost is a set value, i.e., CAIV).

• Many design engineers do not have proper training or experience in 

designing for producibility. Manufacturing facilities must be explicitly 

recognized as a major design constraint. This includes process 

capabilities and rate capabilities at each facility.

The PM is responsible for Producibility

PRODUCIBILITY

Defense Acquisition Guidebook,

4.4.6.1 Producibility

• Producibility: degree to which system design facilitates timely, affordable, optimum-quality manufacture, assembly, and delivery of system

• Producible system design should be a development priority

• Design engineering efforts concurrently develop:

— Producible and testable design

— Capable manufacturing processes

— Necessary process controls to:

 Meet requirements

 Minimize manufacturing costs

• PM should use existing manufacturing processes whenever possible

• When design requires new manufacturing capabilities, PM needs to 

consider process flexibility (e.g., rate and configuration insensitivity)

• Full-rate production necessitates:

— Stable systems design

— Proven manufacturing processes

— Available production facilities and equipment
DAU PROGRAM MANAGERS TOOL KIT

55

QUALITY MANAGEMENT SYSTEMS

Defense Acquisition Guidebook, 4.4.7 Quality

• The PM should allow contractors to define and use their preferred 

quality management system that meets required program support 

capabilities. 

• The PM will not require International Standards Organization 

(ISO) registration of a supplier’s quality system since there have 

been instances where ISO 9001-registered supplier products were 

deficient or life-threatening.

• Contractor’s quality management system should be capable of the 

following key activities: 

– Monitor, measure, analyze, control, and improve processes; 

– Reduce product variation; 

– Measure/verify product conformity; 

– Establish mechanisms for field product performance feedback; and 

– Implement an effective root-cause analysis and corrective action 

system.

Notes: ISO 9000 Series International Quality Standard is considered a Basic Quality system, but the 

focus is still on “Document what you do. Do what you document.” 

Advanced Quality Systems (AQS), such as the new SAE AS9100B Aerospace industries’ quality standard, 

focus on achieving customer satisfaction via use of key characteristics identification and control, variation 

reduction of key characteristics, flow-down of similar process control requirements to suppliers, and many 

other advanced process-oriented control and improvement techniques.

KEY CHARACTERISTICS AND

VARIATION REDUCTION

GOAL—Minimize and control variation on both key product character-

istics and corresponding key manufacturing process characteristics: 

 

• Key Characteristics: The features of a material, process, or 

part whose variation has a significant influence on product fit, 

performance, service life, or manufacturability—per SAE AS9100B. 

• Major Sources of Variation: Insufficient design margins, process 

(manpower, machinery, methods, etc.), measurement systems, 

supplier’s products. 

WHY: Direct correlation between deviation from nominal value (i.e., 

variation) on key characteristics and product quality and functionality. 

TOOLS: Quality Function Deployment (QFD), Design of Experiments 

(DOE), Statistical Process Control. (See control chart on next page.) 
DAU PROGRAM MANAGERS TOOL KIT

56

KEY CHARACTERISTICS AND VARIATION

REDUCTION

(Continued)

The (X bar) and R Control Charts are used to monitor manufacturing processes. Upper 

or Lower Control Limits (UCL or LCL) are NOT design specification parameters. Instead, 

they are predicted boundaries for stable processes, calculated using (X double bar) 

(average of sampled process Means), (R Bar) (the average of the sample Ranges, which 

are the spreads between extreme values per sample), plus the selected data sample size 

and process-keyed statistical formulas. Values outside the UCL and/or LCL indicate possible 

process instability, likely due to uncommon “special” causes of variation. Caution: A process 

in control is desirable because it is predictable, yet it could fail to meet design requirements 

due to inherent “common” variation and/or because the process average isn’t centered on 

the design nominal value.

Reference: The Memory JoggerTM II; ©1994 by GOAL/QPC

UCL

R (Control Chart)*

UCL

LCL

.40

.30

.20

.10

.90

.85

.80

.75

.70

.65

.60

.55

1     2     3     4     5     6     7     8     9    10   11   12   13   14   15   16   17   18   19   20   21   22   23   24   25

X (Control Chart)

R =

.178

–

1     2     3     4     5     6     7     8     9    10   11   12   13   14   15   16   17   18   19   20   21   22   23   24   25

x

–

*Note: No lower control limit for R chart for sample size below 7.

–
DAU PROGRAM MANAGERS TOOL KIT

57

PRODUCTION READINESS REVIEW (PRR)

WHY WE DO THEM”

• Risk Management Tool: Identify program risks and issues and opportunities early and often 

(small, incremental, proactive—vice big, single, reactive) 

• Assess capability of contractor (and subcontractor) to deliver on time, within cost, a product 

that meets performance and quality requirements

• Assess actual contractor performance (metrics)

• Assess effectiveness of contractor’s corrective/preventative actions

• Measure improvement of contractor’s performance

HOW TO DO THEM

• Write a charter that the program office and contractor both understand

• Coordinate with the Defense Contract Management Agency (DCMA)—use their capability

• Establish areas of assessment with metrics

– Producibility

– Engineering Change Orders (ECO)/design stability

– Manufacturing process control (key characteristics)

– Cost, time of scrap, rework, and repair

– Tooling status

– Subcontractor management (same metrics as listed above)

• Ask questions, touch things, talk to shop floor workers

– See what is actually happening on the factory floor rather than the conference 

room, i.e., go see and talk to the people doing the work

When to Do Them

• Early and often (see Defense Acquisition Guidebook, 4.3.3.9.3, Production Readiness Reviews)

• Concurrently with other technical reviews, such as the System Functional Review (SFR), 

Preliminary Design Review (PDR), and the Critical Design Review (CDR)

• In Systems Integration and Systems Demonstration 

• “Final” PRR occurs at end of Systems Demonstration (before Milestone C)

• PRRs should be held in LRIP and beyond IF major changes (to design, manufacturing 

processes, rates/quantities, etc.) occur during LRIP

ADDITIONAL MANUFACTURING

INFORMATION SOURCES

• DAU’s Production, Quality and Manufacturing Information Web site: 

– Go to <www.dau.mil>, select Knowledge Sharing, select Acquisition Community 

Connection, then see Participate in a Community, and select Production, Quality and 

Manufacturing 

– Contains references to subjects including DoD Manufacturing Requirements, and Best 

Business Practices, such as Lean Enterprise, e-Commerce, Six Sigma, Basic and 

Advanced Quality Systems, Supply Chain Management, etc.

• Best Manufacturing Practices Center of Excellence—<www.bmpcoe.org>

• Lean Aerospace Initiative (LAI)—<http://web.mit.edu/lean>
DAU PROGRAM MANAGERS TOOL KIT

58

Use Integrated DT/OT—Single integrated contractor/government DT and OT team;

shared test events and test data; independent data analysis and reporting.

ACAT I and II Programs—Require an independent, dedicated IOT&E to proceed 

beyond Low Rate Initial Production (LRIP).

T&E Required Before Going Beyond Low Rate Initial Production

• Production Qualification T&E—Verify Design Article meets Spec/PM responsible Per­form­­ed by Contractor and/or Government/DPRO assistance valuable. Readiness for IOT&E.

• Live Fire T&E (LFT&E)—Vulnerability and Lethality/Developmental Agency fund and 

execute DOTE oversight, approval, and Congressional reporting for ACAT I, II, and 

selected programs.

• Initial Operational T&E (IOT&E)—Operational Effectiveness and Suitability/Independent Service OTA plan and manage. DOTE oversight, approval, and Congressional reporting for ACAT I and selected systems.

Models and Simulations UsedThroughout the Acquisition Process

TEST and EVALUATION (T&E)—Types and Tasks

Developmental T&E (DT&E)/Operational T&E (OT&E) Comparisons

IOT&E

• Operational effective/suitable

• Operational Test Agency (OTA) responsible

• “Typical” user personnel

• Many test articles/each test

• “Combat” environment/threats

• “Production Rep” test articles

• Contractor may not be allowed 

DT&E

• Technical performance measurement

• Developmental agency responsible (PM)

• Technical Personnel

• Ltd. test articles/each test

• Controlled environment

• All types of test articles

• Contractor involved

Test Requirements

• Test Interfaces

• Eval. Strategy

• Systems Engineering

• Design for Test

• S/W Human T&E

• TES/TEMP

• Subsystem T&E

• Software Only T&E

System DT&E

• Computer Software

Configuration Item T&E

• Reliability, Availability, 

and Maintainability

• Supportability

• Interoperability

• Production Quality

• Live Fire T&E

• Certificate of 

Readiness for IOT&E 

AGONIZE OVER THRESHOLDS!

 To support T&E during:

 Requirements Definition

 T&E Planning

 Engineering Design

 Fabrication

 Integration

 Systems DT&E

 OT&E

 Training

 Operations

System OT&E

• Effectiveness

• Suitability

• Acceptance Test

 • Manufacturing Test

• Data Collection

• Reporting

PRODUCTION (PAT&E, FOT&E)

T&E Tasks and Events

INTEGRATION and TEST

(H/W IN THE LOOP)

OT&E: EOA, OA, IOT&E/OPEVAL

DEPLOY and OPERATIONAL SUPPORT

FABRICATION and TEST (BENCH, LAB)

REQUIREMENTS DEVELOPMENT

ENGINEERING DESIGN

DEVELOPMENT T&E: 

QUALIFICATION T&E, 

ACCEPTANCE T&E, LFT&E
DAU PROGRAM MANAGERS TOOL KIT

59

What is this nomograph? A two-dimensional graphical 

representation of the cumulative binomial distribution. 

Why use nomograph? It enables a relatively simple solution to a 

complex mathematical calculation. 

What does it do? It allows you to calculate the performance of an 

item with associated statistical confidence. 

When do you use it?

• When your requirement includes a “Confidence Level” with a specific 

level of performance. For example: THIS missile must hit THAT 

target 90 percent of the time with 80 percent statistical confidence? 

• When the performance of an item under test can be characterized by 

a binomial distribution.

What are the characteristics of a binomial distribution?

• Result of each event (firing) is an independent from other events.

• Probability of success of each event is constant.

• Each event results in a “success” or a “failure.” (In other words, there 

are no points for being close; each event must be scored as a hit or a 

miss.)

What are some examples of binomially distributed events?

• Coin flip

• Missile launch

• Rocket firing

• Starting a car 

BOTTOM LINE: Each of these test events must be graded as “pass” 

or “fail,” and you must determine the success criteria before the test 

begins.

The nomograph can be used (pre-test) as a test planning device to 

determine how many tests will be necessary to verify that specified 

performance has been met. The nomograph can also be used (posttest) to evaluate test data.

Note: There are two axes on the nomograph. One axis is the total number of trials. The other 

axis in the total number of failures. Additionally, the nomograph is non-linear.

Test and Evaluation Nomograph
DAU PROGRAM MANAGERS TOOL KIT

60

0.99

0.98

0.97

0.96

0.95

0.94

0.93

0.92

0.91

0.90

0.85

0.80

0.75

0.70

0.65

0.60

0.55

0.50

0.999

0.995

0.99

0.98

0.95

0.90

0.80

0.70

0.60

0.50

0.40

0.30

0.20

0.10

0.05

0.02

0.01

0.005

0.001

1000

700

500

400

300

200

140

100

70

50

40

30

20

10

5

2

0

NUMbER OF TRIALS OR SAMPLE SIzE (n)

FAILURES

(r)

1

2

3

4

5

7

9

RELIAbILITy, R

CONFIDENCE LEvEL

How do you get a solution?

• From the data, determine the number of trials (total number of coin flips or missile shots, etc.) and locate the appropriate 

line on the nomograph.

• Determine the number of failures and locate the appropriate line on the nomograph. • Draw a point at the intersection of these two lines on the nomograph. • Any straight line drawn through this point is a valid solution for the data set used.

For example:

• Requirement: Your missile must hit the target at least 90% of the time, with at lease 80% confidence. • Given: You fired 20 missiles with19 hits and 1 failure. • What is the statistical confidence that you will have 90% success in the field with these missiles fired against THAT target? 

Answer: 60% confidence.

• Did you meet the requirement? NO, you achieved only 60% confidence of hitting THAT target 90% of the time, and the 

requirement was 80% confidence or better. One other way to look at the same data is to say that you did achieve 90% probability of success, but you only had 60% confidence in this result; either way you look at it, you did not meet the requirement.

NOTE: If you had fired 30 missiles and missed only 1 time, you would have achieved the 80% confidence along with the required

90% performance level.

T&E Nomograph (Continued)
DAU PROGRAM MANAGERS TOOL KIT

61

MODELING and SIMULATION (m&s) Planning

Integrated Digital

Environment (IDE) Planning

• Establish a business process

 improvement team

• Identify high payback process

areas

• Identify potential legacy systems

and data repositories

• Identify user base including

remote sites

• Capacity of PC workstations

• Bandwidth of communication lines

• Where servers are/will be located

• Identify legacy system host

platforms

How do we plan?—A notional approach

Simulation Support Plan (SSP)

(Required by Army, Marine

Corps, and Air Force)

• ACCESS YOUR SERVICE CENTERS OF M&S EXPERTISE

• Establish a simulation coordinating group; the EARLIER the better

• Design long-term M&S applications and the Integrated Digital Environment 

through the acquisition strategy, Test and Evaluation Management Plan (TEMP), 

Source Selection Plan (SSP)

• Constructive, virtual, live 

• CONTINUOUS PLANNING PROGRAM PLANNING 

 

 

Test and Evaluation Strategy/TEMP

Simulation Based Acquisition: A New 

Modeling and Simulation (M&S)

Planning

• Identify high payback process areas

• Identify potential legacy systems,

Service/Joint-standard simulations,

architectures and data repositories

• Identify where user and simulators

are/will be located

• Determine capabilities and architectures 

of existing simulations

• Network bandwidth requirements

• IDE utilization opportunities

• Interoperability/interface/immersion 

requirements

• Required capability cap

• Design M&S architectures

• Establish a simulation and Verification,

Validation, and Authentication (SVV&A)

planning process

• Establish long-term plan, budget,

document and implement

• Manage, update, and implement the 

SSP
DAU PROGRAM MANAGERS TOOL KIT

62

Hierarchy of 

Models and 

Simulations

FUNCTIONS

RESOL

UTION S

UPPORTED MODELS and SIM

ULATIONS FOR

CE OR S

YSTEM LEVEL

Increasing

Resolution 

Actual 

Performance

Increasing 

Aggregation

Comparative 

Results

Test and

Evaluation

Operational Req’ts. DevelopmentEffectiveness Analysis

Tactics Development Mission Planning and Rehearsal

Design

Manufacturing

Cost

Support

Development

Tech Reqts Dev.

Joint/Combined 

Forces

Air Wings Battle Groups 

Air Wing

EW A/C

AircraftSystem

Fire Control

Radar

Corps

Air Vehicle

(System/Subsystem/Component)

(One-on-One)

(Many-on-Many)

Theater/

Campaign

Mission/

Battle

Engagement

Engineering

(Groups of Systems— 

Force Package)

The Hierarchical Aggregation of Models reduces fidelity 

significantly and must be evaluated. - (PM, DIS, and DMSO)

DIS = Distributed Interactive Simulation, DMSO = Defense Modeling and Simulation Office
DAU PROGRAM MANAGERS TOOL KIT

63

The Evolution of Modeling and Simulation

Distributed

Interactive

Simulation

(DIS)

Full

System

Prototypes

CAD

CAM

CAE

CADAM

SimulationBased Design

(Virtual

Prototyping

with mfg. design,

risk/cost analyses,

mat’l. designation)

CONCEPT DEVELOPMENT

DEPLOYMENT and SUPPORT

MANUFACTURING 

ENGINEERING DESIGN 

and ANALYSIS INTEGRATED SYSTEM DATA

Test and 

Evaluation

VIRTUAL PROTOTYPE

PHYSICAL MOCK-UP

FINAL PRODUCT 

Simulation-Based 

Acquisition 

The Advancing Computer and Communications Technology Revolutions
DAU PROGRAM MANAGERS TOOL KIT

64

PLANNING AND CONTROL TOOLS

1Q 2Q 3Q 4Q 1Q 2Q 3Q 4Q 1Q 2Q 3Q 4Q

Contract Award

IBR

SRR

PDR

CDR

Symbol Meaning

Activity

Planned event completion

Actual event completion

Actual completion behind schedule 

Forecast completion behind schedule

• Shows when key events are scheduled and when they are actually 

accomplished.

• Primary strengths are simplicity and depicting information at the “big picture” level.

• Does not show progress related to events or dependencies between events.

Milestone Chart (Example)

Planned activity schedule

Status of activity

Forecast completion behind schedule

Forecast completion ahead of schedule

Activity

Preliminary Design

Design Analysis

Define Interfaces

Interface Specs

Preliminary Drawings

NOTE: There is no 

standard set of Gantt 

chart symbols.

J F M A M J J A S O

Symbol Meaning Current

Date

• Shows planned start and finish dates; may also show progress

• Depicts activities as horizontal bars imposed over a time line

• Primary strengths are simplicity and depicting overall project plan and status

• Can show dependencies between activities (can be difficult to read as the 

number of activities and dependencies between activities increases)

Current

Date

Gantt Chart (Example)
DAU PROGRAM MANAGERS TOOL KIT

65

Network Schedules

PLANNING AND CONTROL (Continued)

• Graphically portray dependencies and constraints among project 

activities and the sequence in which the activities occur.

• Allows managers to conduct a systematic, disciplines and thorough 

review of the activities required to complete the project.

• Provides information about early and late start and finish times.

• Used to determine the project’s critical path, and slack or float in 

schedule activities.

• Generally, there are two types of networks: Arrow Diagramming 

Method (ADM), and Precedence Diagramming Method (PDM).

Arrow Diagramming Method (ADM)

• Also known as Activity-on-Arrow (AOA); information about activities is 

shown above/below the arrows connecting events in the schedules. 

Events are usually shown as circles, squares, or rectangles (see 

following page).

• ADM generally treats all relationships (see below) as finish-to-start 

(i.e., first activity must finish before the next activity can start). 

• ADM can show other relationships (e.g., start-to-start, finish-to-finish) 

through the use of “dummy” activities.

Precedence Diagramming Method (PDM)

• Also known as Activity-on-Node (AON); information about activities is 

shown in/on the network nodes. Nodes are usually shown as squares 

or rectangles (see following page).

• Lines connecting the nodes show the relationships between the 

activities.

• PDM can show all forms of schedule relationships, including lead and 

lag situations (see below).

Finish-to-Start.

Activity “A” must finish

before Activity “b” can

start.

Finish-to-Finish.

Activity “A” must finish

before Activity “b” can

finish.

A Start-to-Start.

Activity “A” must start

before Activity “b” can

start.

A Start-to-Finish.

Activity “A” must start

before Activity “b” can

finish. Rarely used.

b

b

A b

A

b
DAU PROGRAM MANAGERS TOOL KIT

66

pLANNING AND CONTROL TOOLS

Network Schedules (Example)

Slack = LS – ES or LF – EF

Critical Path

Dummy ActivityTask ID (ES, EF)Duration (LS, LF)

Task ID

Duration

F (18, 26)9 (18, 26)

h (27, 38)12 (27, 38)

E (17, 24)

8

(19, 26)

G

(22, 31)

10 (29, 38)

b

(6, 16)

11

(8, 18)

D

(6, 21)

16

(13, 28)

A (1, 5)5 (1, 5)

C (6, 17)12 (6, 17)

I  (39, 48)10 (39, 48)

J (49, 53)5 (49, 53)

Task ID Duration Predecessor

A 5 --b 11 AC 12 AD 16 AE 8 bF 9 C, DG 10 Dh 12 E, FI 10 G, hJ 5 I

A

5

Precedence Diagramming Method (PDM)

or

Activity-on-Node (AON)

Legend

1 51 5

C

12

6 176 17

F

9

18 2618 26

h

12

I

10

J5

27 3827 38

b

11

6 16

8 18

D

16

6 21

13 28

E

8

17 2419 26

G

10

22 3129 38

39 4839 48

49 5349 53

Arrow Diagramming Method (ADM) *

or

Activity-on-Arrow (AOA)

ES EF

LS LF

ES = Early StartEF = Early FinishLS = Late StartLF = Late Finish

Legend

ES = Early StartEF = Early FinishLS = Late StartLF = Late Finish

Event Critical Path * NOTE: Also sometmes referred to as Crtcal Path Method (CPM) or Program Evaluaton Revew Technque (PERT )

NOTE  1: Task ID, Duraton, and Predecessor nfo s

the same for both examples

NOTE 2: Eght-hour clock s used n both examples

NOTE 1

NOTE 2NOTE 2
DAU PROGRAM MANAGERS TOOL KIT

67

PLANNING AND CONTROL TOOLS

Examples of Network Scheduling Software

Gantt Chart View

Network View

NOTE: Task name, duration, and predecessor information is the same as in Network Schedule on the previous page.
DAU PROGRAM MANAGERS TOOL KIT

68

PLANNING AND CONTROL TOOLS

Schedule Development

IMP PM

Pri Sys

IPT A

Sec Sys SE

IPT D H/W

IPT B IPT E S/W

IPT C IPT F

Program Org Structure

Intermediate Schedule

Master Schedule

Detailed Schedule

Detailed Schedule

Design EngineeringIntegration

AssemblyTest and Evaluation

WBS

1.0

1.1

1.1.11.1.21.1.31.1.4

1.3

1.3.11.3.21.3.3

1.4

1.4.11.4.21.4.3

1.4.4

1.5

1.5.11.5.2

1.2

1.2.11.2.21.2.3

SOW,

Specs,

SEP, etc.

Work

Packages

Program

Requirements

Technical

Documents Program Plans

Resource

Planning

Schedules

Task

Lists

ICD,

CDD,

CPD,

etc.

S.E.

Process

IPT C IPT C IPT C IPT C IPT C

Track 1Track 2Track 3Track 4Track 5

Responsibility Matrix

Duration & Resource

Estimates

Network (Task Sequence)

Historical Data

SMEs
DAU PROGRAM MANAGERS TOOL KIT

69

26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

WORKING DAY

S PRIOR TO COMPLETION (LEAD TIME)

(SU

BCONTRACT

PART)

(PURC

HASE PART)

(PURC

HASE PART) (PURC

HASE PART)

(SU

BASSEM

BLY “B”)

(SU

BASSEM

BLY

“A”)

FINAL ASSEM

BLY

(FA

BRICATED PART

INHOUSE)

TEST

S

HIP

6

8

7

4

1

2

3

5

12 11 10

9

LEAD TIME

C

HART

(Use with Line of Balance on next page.)
DAU PROGRAM MANAGERS TOOL KIT

70

80

70

60

50

40

30

20

10

0

8070605040

3020

10

0

O D N F J A M J M J 2 1 4 3 5 6 7 9 8 10 11 21

SHTNOM ORTNOC PL STNI O

0807060504030201

0

denn la P

l aut c A

62 25 42 23 22 21 02 19 18 17 16 15 14 13 12 11 1 8 9 0 5 6 7 1 2 3 4 0

NOI TCUDORP DAEL TIME RAHC T 22 KRO W TNOM/ SYAD H

6

1

4 7 8

2

3 5 9

01 11 12

t ohspanS ni e imt 1: a M y

LINE OF BALANCE TECHNIQUE
DAU PROGRAM MANAGERS TOOL KIT

71

ACQUISITION PROGRAM BASELINE

Key Performance Parameter (KPP)—An Example

Objective

Threshold

X

.85

.80

.75

.70

.65

(Prob)

Exit

Criteria

(.77)

Full-Rate Production

Current

Estimate

Growth (Maturity) Curve

S

U

R

V

I

V

A

B

I

L

I

T

Y

Exit

Criteria

(.73)

* In this example, the current estimate falls below the threshold—this represents a baseline breach of 

performance.

• Every program must have an APB starting at program initiation (normally Milestone B).

• The APB reflects the threshold and objective values for a minimum number of cost, 

schedule, and performance parameters that describe the program over its life cycle.

• Cost thresholds and objectives reflect major elements of life cycle cost (RDT&E, 

procurement, PAUC, APUC, etc.) 

• Schedule thresholds and objectives reflect critical events (milestone decisions, start of 

DT/OT, first flight, IOC, etc.) 

• Performance thresholds and objectives are key performance parameters (KPPs) 

extracted verbatim from the CDD/CPD.

• The JROC requires four mandatory KPPs: net-ready, survivability, materiel availability, 

and force protection. Two additional KPPs are required for selected programs: system 

training and energy efficiency.

• The MDA may add other significant performance parameters if necessary.

• The APB is signed by PM, PEO, and CAE, as appropriate, and approved by MDA.

System Integration System Demonstration Low-Rate Initial Prod

MS B DRR MS C FRP DR

Baseline

Values

Legend:

MS—Milestone

DRR—Design Readiness Review

FRP DR—Full-rate Production

Design Review

CDD—Capabilities Design Document

CPD—Capabilities Production Document

MDA—Milestone Decision Authority

PM—Program Manager

PEO—Program Executive Officer

System Development and

Demonstration Phase

Production and

Deployment Phase

CAE—Component Acquisition Executive

IOC—Initial Operational Capability

APUC—Average Procurement Unit Cost

PAUC—Program Acquisition Unit Cost

RDT&E—Research, Development, Test 

and Evaluation

JROC—Joint Requirements Oversight 

Council

DT/OT—Development Test/Operational Test
DAU PROGRAM MANAGERS TOOL KIT

72

RISK MANAGEMENT PROCESS MODEL

Rsk Identfcaton

Rsk Analyss

Rsk Mtgaton

Plannng

Rsk 

Trackng

Rsk Mtgaton 

Implementaton

Root cause analyss

of the rsks dentfed

Root cause analyss

A Process, Not an Event

root cause analysis

• Risk identification includes analysis to identify the root causes of the 

risks identified.

• Root causes are identified by examining each WBS product and 

process element in terms of the sources or areas of risk.

• An approach for identifying and compiling a list of root causes to:

– list WBS product or process elements;

– examine each in terms of risk sources or areas;

– determine what could go wrong; and

– ask “why” multiple times until the source(s) is discovered. 

DoD Risk Management Guide

August 2006
DAU PROGRAM MANAGERS TOOL KIT

73

PrioritizedList

Avoid Transfer Assume

Quantify Some

Qualify All

Identify

Select how to Mitigate

Analyzeprioritize/list Trackin

g

status toplan

Control

Plan for eachprogram byphase/evaluation

Mitigation Implementation through Integrated Product and Process Development (IPPD)

Periodic

Update

Raw List

Update asneeded

Integrateinto Program Plans & Control Toolsas necessary

RISK management process

—details
DAU PROGRAM MANAGERS TOOL KIT

74

RISK MANAGEMENT

1. Develop program plans to the work package level.

2. Identify and analyze risk at the lowest work package/WBS level.

3. Mitigate and actively manage the highest risk work packages; risks that 

you can’t mitigate you must accept.

Tradeoff Analysis*

1. Identify alternative solutions

2. Select evaluation criteria/factors and MOEs, i.e., cost, schedule, 

performance criteria

3. Weight evaluation criteria

4. Develop utility functions for each factor 

5. Conduct evaluation (weighted utility summary table where weight is 

multiplied by utility function value)

6. Perform sensitivity check

7. Select highest scored alternative

RISK and TRADEOFF ANALYSIS

Risk Planning

Risk Management Plan

 (The Process) 

Risk ID

Technical

Cost

Schedule

Lessons learned

WBS

Risk Analysis

Networks

Simulation

Watch lists

Templates

Risk Mitigation

Avoidance

Control

Assumption

Transfer

Risk Management

*With Cost As an Independent Variable (CAIV), aggressive cost objectives are established 

as a result of trading performance and schedule for cost.

SOW DELIVERABLES WBS

OBS

WORK PKG

PLANNING PKG

GANTT/

NETWORK

CHART

COST

EST.

COST

VS

TIME

RISK MGMT

PROCESS:

ANALYZE AND

MITIGATE RISK

Specs Contract

Risk Tracking

Monitoring

Reports

Feedback

T&E Results
DAU PROGRAM MANAGERS TOOL KIT

75

TECHNICAL PERFORMANCE MEASUREMENT

The Concept

TECHNICAL 

PARAMETER 

VALUE

e.g.,

MTBF

Weight Fuel 

Consumption

Tolerance Band

Threshold

Planned 

 Value 

Current 

Estimate

TIME

Variation

Planned

 Profile

Achievement

 to date

Milestones 

}

1. ARE ALL VIABLE ALTERNATIVES BEING

EXPLORED?

– Is each alternative clearly defined?

– Have the alternatives been 

prescreened? How?

– Are affordability limits established?

– Can all of the screened-out 

alternatives be defended?

2. ARE SELECTION CRITERIA IDENTIFIED?

– Are all significant criteria identified?

– Do the criteria discriminate among 

alternatives? 

– Are the criteria measurable?

– Have the criteria been pre-approved? 

3. IS THE CRITERIA WEIGHTING SYSTEM 

ACCEPTABLE?

– Are rationales for criteria weights 

explained?

– Are criteria weights consistent with 

guidance?

– Are criteria weights consistently 

distributed in the tree?

4. ARE UTILITY (SCORING) CRITERIA

DETERMINED?

– Is defensible rationale established for 

each criterion?

– Are criteria developed from operational 

measures of effectiveness where 

possible?

– Do all plans use the same numerical 

scale?

– Is the location of the “zero point” 

explained?

5. ARE EVALUATION METHODS

DOCUMENTED?

– Are test data reliability estimates 

(confidence levels) incorporated?

– Are models validated? When? By Whom?

6. HAS SENSITIVITY BEEN ESTIMATED?

– Are error ranges carried through with 

worst-on-worst case analysis?

– Have the effects of changes in the utility 

curve shapes been examined?

– Have rationales for the limits been 

developed?

PROGRAM MANAGER’S CHECKLIST FOR REVIEW

OF TRADEOFF PLANNING AND STUDIES
DAU PROGRAM MANAGERS TOOL KIT

76

What is Systems Engineering?

• International Council on Systems Engineering (INCOSE) definition:

Systems Engineering is an interdisciplinary approach and means 

to enable the realization of successful systems. It focuses on 

defining customer needs and required functionality early in the 

development cycle, documenting requirements, then proceeding with 

design synthesis and system validation while considering the

complete problem...

Focus is on technical systems development and integrative engineering to meet requirements.

 -- www.incose.org 

Systems Engineering Tasks

• Ensure essential technical things get done

• Verify technical solutions to satisfy customer capability requirements

• Develop a total system design solution

– Design in downstream life-cycle needs (open system approach)

– Balance cost, schedule, performance, and risk

• Generate and track technical information needed for decision making 

and configuration management

Systems Engineering Policy in DoD

(Signed by the Honorable Mike Wynne, USD(AT&L) (Acting), February 20, 2004)

• All programs, regardless of ACAT shall:

– Apply an SE approach

– Develop a Systems Engineering Plan (SEP)

◆ Describe technical approach, including processes, resources, 

and metrics

◆ Detail timing and conduct of SE technical reviews

• Director, Defense Systems (DS), USD(AT&L) tasked to provide 

guidance for DoDI 5000.2

– Recommend changes in Defense SE

– Establish a senior-level SE forum

– Assess SEP and program readiness to proceed before each DAB 

and other USD(AT&L)-led acquisition reviews
DAU PROGRAM MANAGERS TOOL KIT

77

s

ystems engineering process—Tec

hnical 

Management 

Processes

Techncal

Plannng

Requrements

Management 

Confguraton

Management

Transton

Techncal

Assessment

Decson

Analyss

Desgn

Soluton

Verfcaton

Requrements

Development

Valdaton

Integraton

Implementaton

Logcal Analyss

and Allocaton

Transton

Realzaton

Processes

Desgn Processes

Rsk

Management

Interface

Management

Data

Management
DAU PROGRAM MANAGERS TOOL KIT

78

s

ystems engineering process linkages

Logical (functional)

Analysis

Stakeholder requirements

Definition

Requirements

Development

Note: Parts of “V” process in dark gray boxes match the Systems Engineering Process Model. This process is followed

in phase activities “Vs” on succeeding pages. 

Operational Expectations

User Expectations

Configuration

Item Expectations

tions

Output/Transition Input

requirements

(User Needs) Analysis

realization &

Assessment

Validation

Verification

Integration

Design

Implementation

Design Solution

Development

& Definition

functional Expectations

Transition

Hardware fabrication Operator/Maintainer

Training

Software

Creation/Coding
DAU PROGRAM MANAGERS TOOL KIT

79

defense acquisition management framework—Technical “V” Activities

Pre-Systems Acquisition

User Needs &

Technology Opportunities

PRODUCTION & DEPLOYMENT OPERATIONS

& SUPPORT

LRIP/IOT&E

SYSTEM DEVELOPMENT

 & DEMONSTRATION

FRP

DECISION

REVIEW

DESIGN

READINESS

REVIEW

C IOC B

Systems Acquisition Sustainment

• Process entry at Milestones A, B, or C (or within phases)

• Program outyear funding when it makes sense, but no later 

 than Milestone B (unless entering at C)

A

CONCEPT

REFINEMENT

CONCEPT

DECISION

FOC

TECHNOLOGY

 DEVELOPMENT

Technical Activity “V” for Each Phase

Legend:FOC—Full Operational CapabilityIOC—Initial Operational CapabilityLRIP/OT&E—Low Rate Initial Production/ Operational Test and Evaluation
DAU PROGRAM MANAGERS TOOL KIT

80

CONCEPT REFINEMENT PHASE systems Engineering (SE)

ACTIVITIES

TECHNOLOGY DEVELOPMENT PHASE SE ACTIVITIES

Demo & Validate Sys

Concepts & Te chnology Maturity

Versus Defined User Needs Trades

INPUTS OUTPUTS

• ICD & Draft CDD

• Preferred Sys Concept

• Exit Criteria

• T&E Strategy

• Support & Maintenance

Concepts & Technologies

• AoA • SEP • TDS

Develop System Perf (&

Constraints) Spec & Enabling/

Critical Tech Verification Plan

Demo/Model Integrated System

Versus Performance Spec

Develop Functional Definitions for

Enabling/Critical Technologies &

Associated Verification Plan

Interpret User Needs.

Analyze Operational Capabilities

& Environmental Constraints

Demo System Functionality Versus Plan

Demo Enabling/Critical Technology

Components Versus Plan

Develop System Concepts, i.e.,

Enabling/Critical Te chnologies, Update

Constraints & Cost/Risk Drivers

Trades

Decompose Functional Definitions into Critical

Component Definition & Tech Verification Plan

• Sys Performance Spec

• LFT&E Waiver Request

• TEMP • SEP • PESHE • PPP • TRA

• Validated System Support & Maintenance

 Objectives & Requirements

• Inputs to: –IBR –ISP –STA –CDD –Acq Strat

 –Affordability Assessment

 –Cost/Manpower Est.

SRR

ICD

AoA Plan

Exit Criteria

Alternative Maintenance & 

Logistics Concepts

Prelim Sys Spec

T&E Strategy

SEP

Support & Maintenance Concepts & Tech

Inputs to: -Draft CDD -TDS -AoA

 -Cost/Manpower Est.

Decompose Concept Performance

 into Functional Definition 

& Verification Objectives

Analyze/Assess

System Concept Versus

Functional Capabilities

Analyze/Assess Concepts 

Versus Defined User Needs & 

Environmental Constraints

Assess/Analyze Concept &

Verify System 

Concept’s Performance

Develop Component Concepts, i.e., 

Enabling/Critical Technologies, 

Constraints & Cost/Risk Drivers

Analyze/Assess

Enabling/Critical Components

Versus Capabilities

Decompose Concept Functional

 Definition into Component Concepts 

& Assessment Objectives

Develop Concept

Performance (& Constraints)

Definition & Verification Objectives

Interpret User Needs,

Analyze Operational Capabilities &

Environmental Constraints

INPUTS OUTPUTS

Trades

ITR ASR

Trades
DAU PROGRAM MANAGERS TOOL KIT

81

SYSTEM DEVELOPMENT AND DEMONSTRATION

PHASE Systems Engineering ACTIVITIES

Production and Deployment Phase

Systems engineering Activities

Trades

INPUTS OUTPUTS

Trades

• Sys Performance Spec

• Exit Criteria

• Validated Sys Support &

Maintenance Objectives &

Requirements

• APB • CDD • SEP • ISP • TEMP

Interpret User Needs, Refine

System Performance Specs &

Environmental Constraints

Develop System

Functional Specs &

System Verification Plan

Evolve Functional Performance

Specs into CI Functional (Design to)

Specs and CI Verification Plan

Evolve CI Functional Specs

into Product “Build-to” 

Documentation and Inspection Plan

• Initial Prod Baseline • Test Reports

• Elements of Product Support

• Risk Assessment • TEMP

• SEP • TRA • PESHE

• Inputs to: –CPD –STA –ISP

 –Cost/Manpower Est.

Fabricate, Assemble, Code to

Build-to Documentation

System DT&E, LFT&E & OAs,

Verify System Functionality &

Constraints Compliance to Specs

Individual CI

Verification DT&E

Integrated DT&E, LFT&E &

EOAs Verify Performance

Compliance to Specs

Combined DT&E/OT&E/LFT&E

Demonstrate System to Specified User

Needs & Environmental Constraints

FCA SVR PRR

CDR

TRR

SFR

SRR

PDR

FTA

RCM

LORA

MTA

FMECA

OUTPUTS

J-6 Interoperability

& Supportability Validation

Full-Up System Level LFT&E

Independent IOT&E

LFPT

Report to

Congress

INPUTS

JITC Interoperability

Certification Testing

OTRR

BLRIP

Report to

Congress

Analyze Deficiencies To

Determine Corrective Actions

• Test Results

• Exit Criteria

• APB • CPD • SEP •TEMP

• Product Support Package

Verify & Validate

Production Configuration

• Production Baseline

• Test Reports

• TEMP • PESHE • SEP

• Input to: –Cost/Manpower Est.

PCA

Modify Configuration (Hardware/Software/

Specs) To Correct Deficiencies
DAU PROGRAM MANAGERS TOOL KIT

82

Operations and Support Phase

Systems Engineering Activities

INPUTS OUTPUTS

Data for In-Service Review:

• Input to CDD for Next Increment

• Modifications/Upgrades to Fielded

Systems 

• SEP

• Process Change—

Hardware/Support

• Materiel Change

Trades

• Service Use Data

• User Feedback

• Failure Reports

• Discrepancy Reports

• SEP

Monitor and Collect

All Service

Use Data

Analyze Data to

Determine

Root Cause

Determine

System Risk/

Hazard Severity

Develop

Corrective

Action

Implement and 

Field

Assess Risk of

Improved System

Integrate and Test

Corrective Action 

Trades

In-Service

Review

•
DAU PROGRAM MANAGERS TOOL KIT

83

REQUIREMENTS (USER NEEDS)

ANALYSIS QUESTIONS

• What are the reasons behind the system development?

• What are the customer expectations? How will they measure the 

performance of the system?

• Who are the users and how do they intend to use the product?

• What do the users expect of the product?

• What are their levels of expertise?

• With which environmental characteristics must the system comply?

• What are existing and planned interfaces?

• What functions will the system perform, expressed in customer 

language?

• What are the constraints—hardware, software, economic, procedural 

– with which the system must comply?

• What will be the final form of the product—model, prototype, mass 

production?

• Specific, Clear, and Unambiguous: Contains no vague terms

• Understandable: Stated with sufficient detail in everyday 

language

• Concise: Contains no unnecessary words

• Consistent: Top-to-bottom consistency with identical usage of 

terms and conformance to standards 

• Stable: Baselined and under configuration control

• Traceable: Derived from the mission profile or the contractor’s 

design policies

• Verifiable: Determine whether the product is satisfying the 

requirement

• Feasible: Can achieve, produce, and maintain the requirement 

ATTRIBUTES OF A WELL-DEFINED REQUIREMENT
DAU PROGRAM MANAGERS TOOL KIT

84

Systems Engineering Process—

Design Operations

X

f1

r1

f2 f3 fN

X

X

X X

X X

r2

r3

rN

• Customer Needs

• Tech Base

• Prior Systems Engineering Output

• Program Decision requirements

• Budget

requirements

Development

• Technical Planning

• requirements Management

• Configuration Management

• Decision Analysis

• Technical Assessment

• risk Management

• Interface Management

• Data Management

Do what? functions

How well? Performance

Environment? Interfaces

ICD req’ts Spec req’ts

Implied req’ts

Questions for requirers

Technical

Management

Integration,Verification/Validation,T&E

Design Loop requirements Loop

Design

Solution

Logical

Analysis

Implementation

Decision

Database

TIMELINE

HI / LO

Baselines

functional

(System) Allocated

(Perf)

Product

(Detail) Specifications

System

Item Perf Item Detail

Process

Material

TO

CrUISE

CLIMB

LAND DESCEND

CLIMB CrUISE

LOITEr

ATTACK

(67 min., 50 km range)

fLY

TAKEOff CrUISE LAND

(2 min.) (60 min.,

50 km range)

(5 min.)

AIrCrAfT

AIr ENGINE COMMUNICATIONS

frAME

• Analyze functions

• Decompose function

• Allocate requirements

f1

f2

f3

fN

X X

X X

X

X X

Hw1 Hw2 Sw1 SwN
DAU PROGRAM MANAGERS TOOL KIT

85

S

YSTEMS ENGINEERING

Design 

Considerations

—“The 

Fishbone”

SystemPerformance

TechnicalEffectiveness

SystemEffectiveness

SystemAvailability

ProcessEfficiency

Capabilities

Functions

Priorities

Reliability

Maintainability

Supportability

Producibility

Operations

Maintenance

Logistics

Life Cycle Cost / Total Ownership Cost

SYSTEM AVAILABILITY

Reliability / Maintainability / Supportability:RAM, COTSProducibility: Value Engineering, Quality,

Manufacturing Capability

Affordable

Operational

Effectiveness

SYSTEM PERFORMANCE

Open Systems DesignInteroperability

Architectural

Impacts on System

Design Considerations

Survivability and SusceptibilityAnti-TamperHuman Systems IntegrationAccessibilityInsensitive MunitionsSystem SecurityInformation AssuranceCorrosion PreventionCOTSDisposal and DemilitarizationEnvironment, Safety, Occupational Health

Design Tools

Modeling & Simulations

Operations: Interoperability, Information Assurance, Survivability andSusceptibility, Human Systems Integration, System Security, Anti-TamperMaintenance: Corrosion Prevention and Control, Accessibility,Interoperability, Unique Identification of ItemsLogistics: SupportabilityProducibility: Value Engineering, Quality, Manufacturing Capability

PROCESS EFFICIENCY
DAU PROGRAM MANAGERS TOOL KIT

86

TECHNIcAL MANAGEMENT PROCESSES

(See top of Systems Engineering Process Chart, page 77.)

HOW DO I MAKE DECISIONS ?WILL IT DO A JOB THAT JUSTIFIES THE EXPENSE? 

ARE WE DOING THE RIGHT THING ?

HOW DO I KNOW IT WORKS?WILL IT MEET THE PERFORMANCE CRITERIA?WILL IT ALL WORK TOGETHER ?DO WE KNOW WHAT WE HAVE ?ARE WE READY TO GO ON ?HOW DO I RUN THIS PROGRAM ?

TRADE STUDIESEFFECTIVENESS ANALYSISRISK MANAGEMENTTECHNICAL PERF MEASURESMODELING and SIMULATIONTECHNICAL PERF MEASURESCONFIGURATION MANAGEMENT TECHNICAL REVIEWSINTEGRATED PLANNING
DAU PROGRAM MANAGERS TOOL KIT

87

SPECIFICATION DE

VELOPMENT and TECHnical RE

VIEWS

Pre-Systems Acquisition Systems Acquisition Sustainment

System Development & 

Demonstration

Operations & 

Support

Concept

refinement

Technology 

Development

System

Integration LrIP frP & Deployment

IOC fOC

Sustainment

Disposal

Specifications System Specification

Item Performance Specifications (Design-to)

ASr Srr Sfr

S/w reviews

PDr CDr Trr

Item Detailed Specification (Build-to)

ConfigurationBaselines

Functional Baseline

Allocated Baseline

Product Baseline

Prr SVr

• Increased emphasis on evolutionary acquisition and spiral development• Technology development strategy: Approved by Milestone Decision Authority (MDA) at Milestone A• Systems Engineering Plan: Integrated with acquisition strategy and submitted for MDA approval at each milestone

review, regardless of Acquisition Category (ACAT)

Technical reviewsand Audits

A B

C

SE Process

Configuration Control

Product Upgrades

Planning for Disposal

(repeated in Each Phase)

PC

frP

A

(ICD)

Capability Development 

Document (CDD) Document (CPD)

Production & Deployment

System 

Demonstration

ASr

S/w reviews

B

C Drr

Initial Capabilities Document

(ICD)

Capability Development 

Document (CDD)

Capability Production 

Document (CPD)
DAU PROGRAM MANAGERS TOOL KIT

88

Technical Review Definitions

ASR (Alternative Systems Review) – Assess that preferred system 

solution meets needs.

SRR (System Requirements Review) – Ensure system performance 

capabilities are consistent with technology solution and traceable 

to Initial Capabilities Document and draft Capability Development 

Document.

SFR (System Functional Review) – Ensure functional performance 

capabilities are consistent with cost, schedule, and risk constraints 

and system ready to proceed into preliminary design.

PDR (Preliminary Design Review) – Ensure system preliminary 

design and functional/allocated baseline are captured in item 

performance specifications for each configuration item in system.

CDR (Critical Design Review) – Determine that system meets stated 

performance capabilities, product baseline is captured in item detail 

specification, and system is ready for fabrication, demonstration, and 

test.

PRR (Production Readiness Review) – Verify design is ready for 

production and adequate production planning accomplished; system 

capabilities are traced to final production system. 

TRR (Test Readiness Review(s)) – Assess test readiness and 

approve test plans.

SVR (System Verification Review) (synonymous with Functional

Configuration Audit) – Verify configuration items perform to 

specification and system is ready to proceed into Low-Rate Initial 

Production and Full-Rate Production within cost, schedule, and risk 

constraints.

PCA (Physical Configuration Audit) – Verify item produced (product 

baseline) matches design documentation (e.g., drawings, etc.) and 

item detail specification in the contract.
DAU PROGRAM MANAGERS TOOL KIT

89

Technical Review Best Practices

Technical reviews:

• Are a fundamental part of the SE process and serve as a technical

assessment product for the program manager

– Should be event-based

– Objective entry criteria need to be defined up front

– Are only as good as who conducts them

– Engagement of Technical Authority

– Chair independent of program team

– Independent subject matter experts, determined by Chair

– Involve ALL STAKEHOLDERS

• Should review entire program from a technical perspective

– Cost, schedule, and performance

– By ALL STAKEHOLDERS

– Involve all technical products (specs, baselines, risks, cost 

estimates)

• Should result in program decisions and changes

– Rather than a “check in the box”

• Taken as a whole series, form a major part (backbone) of the SEP

Easy in principle,

difficult in practice.
DAU PROGRAM MANAGERS TOOL KIT

90

SPECIFICATIONS AND STANDARDS

A New Way of Doing Business (Acquisition Reform)

(SecDef Memo of June 29, 1994)

1. Use Performance-Based Specifications

2. Cancel/Convert Manufacturing and Management Standards to 

Performance or Nongovernment Standards (NGSs)

3. Encourage Contractors to Submit Alternative Solutions to Military 

Standards/Specifications

4. Prohibit Use of Military Specifications/Standards Except when 

Authorized by Service Acquisition Executive or Designee

Design/Fab. Require desired Specify exact parts and

outcomes or functions, components

Specific design to 

contractor

Processes Few, if any Specify exact processes

Physical Give specifics only for Specify more physical

Characteristics interfaces, environment, characteristics than

or human factors for interfaces,

environment, etc.

Interface Detailed interface data do Detailed interface data

Requirements NOT solely make a perf.

spec. a detail spec.

Materials Leave specifics to Require specific materials

contractor

Test and Evaluation State performance Prescribed testing 

need; contractor picks process

test procedure

PERFORMANCE DETAIL / DESIGN

PERFORMANCE vs. DETAIL SPECIFICATIONS
DAU PROGRAM MANAGERS TOOL KIT

91

• Program unique specifications advantages:

– Helps avoid duplication and inconsistencies.

– Enables good estimates of necessary work and resources.

– Provides consistent communication among players as people 

rotate.

– Can be used to prepare test plans.

– Can be used a long time after the system has been put into 

operation.

– Serves as an interface between customers, developers, and 

designers.

• Can act as negotiation and reference document for engineering

changes.

MIL-STD-961

Standard Performance Specification

Standard Design Specification Program-Unique Specifications 

No waiver required to use

STD PRACTICE

Program-Unique Specifications

• Defines mission/technical performance 

requirements. Allocates Requirements to 

functional areas. Defines interfaces.

• Defines performance characteristics of 

configuration items (form, fit, function). Details 

design requirements only to meet interfaces. 

“DESIGN-TO.”

• Includes “how to” and specific design

requirements. Usually includes specific 

processes and procedures. “BUILD-TO.”

• Defines process performed during fabrication.

• Defines production of raw materials or 

semi-fabricated material used in fabrication.

System

(Hardware or

Software) Item

Performance

(Hardware or

Software) Item

Detail

Process

Material

Functional

(“System”)

Allocated

(“Design-to”)

Product

(“Build-to”)

Product

Product

Specification Content Baseline

Specs
DAU PROGRAM MANAGERS TOOL KIT

92

CONFIGURATION MANAGEMENT PLANNING

• The decisions on 

– Which baselines the government should eventually control

– The data needed

– When that control should be established

... are strategic management decisions that involve

– Acquisition strategies—sources, competition, etc.

– Logistics support plans—repair levels, data needs, open systems,

etc.

– Technology insertion—stable vs. rapidly moving technologies, etc.

• Government should control the Functional Baseline (document system level 

requirements)

• DoD PMOs increasingly choose to leave Allocated Baselines under 

contractor control until late in development. (Documents the Configuration 

Item (CI) level design requirements.) 

– Promotes contractor design flexibility

– Relieves PMO from administrative burdens of managing design

Engineering Change Proposals

– Requires effective implementation of Integrated Product and

Process Development

• When and if to control baselines is dependent on support philosophy and 

acquisition management strategy.

“A management process for establishing and maintaining consistency

of a product’s performance, functional, and physical attributes with its

requirements, design, and operational information throughout its life.”

• Identify and document the functional and physical characteristics of 

configuration items.

• Control changes to configuration items and their related documentation.

• Record (or Status Accounting in DoD terms) and report information 

needed to manage configuration items effectively, including the status of 

proposed changes and implementation status of approved changes.

• Audit configuration items to verify conformance to specifications, drawings, 

interface control documents, and other contract requirements.

Configuration Management

Nongovernment Standard: EIA Standard-649

Also see MIL HNBK 61

– Adopted by DoD on November 22, 1996
DAU PROGRAM MANAGERS TOOL KIT

93

• The Government PMO:

– Identifies external interfaces

– Establishes interface standards 

(baselines)

– Maintains interface stability

• The contractor:

– Manages internal interfaces

– Establishes interface requirements 

to include internal and external 

interfaces

– Controls interfaces to ensure

◆	Accountability

◆	Timely dissemination of changes

 

The Government increasingly chooses to manage interfaces, 

leaving design details to contractors.

• Identifies, documents, and controls all functional and physical

characteristics

• Interfaces:

– What? 

◆	Common boundary

◆	Types: mechanical, electrical, operational, software

◆	Functional and physical characteristics

– Where?

◆	Within one contractor’s design

◆	Among contractor’s items and GFE

◆	Among multiple contractors’ items

◆	Among systems

– Controlled by Interface Control Working Group

– Documented in Interface Control Documents

INTERFACE MANAGEMENT

Will it all work together?

INTERFACE CONTROL CONCEPT

?
DAU PROGRAM MANAGERS TOOL KIT

94

BASIC PURPOSES OF WBS

HOW TO CREATE A

WORK BREAKDOWN STRUCTURE

Sys Eng

System

Training Test Mfg

computer sensor

Crew

Simulator

HW

System XXX

SW

Application

SW

displays

Grouping for

Spec.

Development

Interface

Earned Value Management

Evaluation

Risk

Assessment

$$$

Management

Product 

Tree

ECP

Impact

IPT

Setup

CATT Support

Appendix

A

B

C

MIL HDBK

881 System

Define

the Product

Tailor

(Supporting Processes)

Create and Refine

as Design Matures

SE Process

Note: Oval shapes on periphery 

 identify WBS purposes

Tech 

Review

Structure
DAU PROGRAM MANAGERS TOOL KIT

95

Note: SSR = Software Specfcaton Revew. The SSR s a unque sub-systemrevew held to assess SW requrements pror to start of desgn.

System

Requrements

Analyss

SR

R

SF

R

SS

R

PD

R

CD

R

TR

R

SR

R

SF

R

SS

R

PD

R

CD

R

TR

R

System

Desgn

Software

Requrements

Analyss

Software

Desgn

Software

Unt Codng

Software

Unt Testng

SW Unt Integraton

and Testng

Syste

m

Sub-Syste

m

SW Ite

m

(SI)

SW Ite

m

(SI)

HW Ite

m

(HI)

SW Item

(SI)

Software

Unts

Software

Unts Software Unts

Software Item

Qualfcaton

Testng

HI and SI

Integraton

and Testng

Subsystem

Integraton

and Testng

System 

Qualfcaton

Testng

DT and OT&E Testng

software de

velopment

—tec

hnical re

view relations

hips
DAU PROGRAM MANAGERS TOOL KIT

96

Software de

velopment

—ke

y life c

ycle re

view factors

Rev 3.1

Intal Stage

Desgn & Revew

Actvtes

Allocated

Baselne(s)

Functonal

Baselne 

SS

R

CDR TRR PRR

SVR

PCA Software Development

Key Factors

• Are requrements traceable to operatonal msson profles?• Is top-level requrements allocaton reasonable? Are software and hardware allocatons realstc?• Has a software archtecture been establshed? Is t reasonable?• Have realstc computer resource utlzaton lmts been establshed?• Have communcaton and nterface requrements been establshed?• Have crtcal requrements (safety, securty, prvacy) been specfed?

Software Development

Key Factors

• Have software requrements been establshed? Are they prortzed? Are derved requrements understood? • Have condtons for software acceptance been establshed?• What modfcatons are needed as a  result of prototype testng?• What are reuse & COTS levels?• Are personnel, tranng, and support mpacts understood?• What excepton condtons/capacty  lmts have been taken nto account?• Is the proposed lower-level break-out of software functonalty reasonable/?

• Do results of Detaled Desgn support subsequent codng and mplementaton actvtes?• Are nterfaces specfed n suffcent detal to support codng and mplementaton?• What systems are n place for  software confguraton management durng Software Support?• How wll changes be ncorporated nto the felded software?• Do results of software tests to date verfy performance, relablty, safety, securty, and other crtcal requrements?

Software Development

Key Factors

Intermedate Stage

Desgn & Revew

Actvtes Fnal Stage

Desgn & Revew

Actvtes Product Baselne(s) 

PDR

SRR SDR

SFR
DAU PROGRAM MANAGERS TOOL KIT

97

candidate software measures (Metrics)

• Software Size

• Requirements Volatility

• Software Effort/Staffing

• Software Progress

• Problem/Change Report Status

• Rework/Scrap

• Computer Resource Utilization

• Milestone Performance

• Build Release Content

• Software Complexity

• Effect of Reuse

• Earned Value

Check out the handbooks at the DoD’s Practical System and Software 

Measures site at <www.psmsc.com>.

Software measures 

should be risk- or issuedriven and are phasedependent.

Desk

Checkng

Walkthroughs

Formal

Inspectons

Jont

Revews

• Process-drven

• Test & ntegraton plannng key

• Includes qualfcaton testng

• Software tem/confguraton tem orented

• Whte vs. black box testng

• Ineffectve

• Better than nothng

• Indvdually done

• May have defned procedures

• Team orented revew

• Results may be recorded

• Around 40% defect removal

• Use specally traned teams

• Formal process

• Team atttude crtcal

• Rgd entry/ext crtera

• Bass for SW metrcs

• Geness for process mprovement

• Around 70% defect removal

• Preparaton crtcal

• Entrance/ext crtera key

• Frequently abrdged

• Hgh-level revew

• May not be hgh-leverage, SW-qualty event

Spectrum of

Qualty Events

for Software

ComputerBased

Testng

Actvtes

Human-Based

Qualty Actvtes

Quality Events For Software
DAU PROGRAM MANAGERS TOOL KIT

98

• Adopt continuous risk management

• Estimate cost and schedule empirically

• Use software metrics to help manage

• Track earned value

• Track software defects against software quality targets

• Treat people as the most important resource

• Use life cycle configuration management

• Manage and trace requirements

• Use system-based software design

• Ensure data and database interoperability

• Define and control interfaces

• Design twice, but code once

• Carefully assess reuse risks and costs

• Inspect requirements and design

• Manage testing as a continuous process

• Test frequently

• Use good systems engineering processes

software management Best practices

• Use schedule compression to justify new technology on a timecritical project

• Have the government mandate technological solutions

• Specify implementation technology in the RFP

• Use as many “silver bullets” as possible

• Expect to recover more than 10% schedule slip without a reduction in 

delivered functionality

• Put items out of project control on the critical path

• Plan on achieving more than 10% improvement from observed past 

performance

• Bury as much of the project complexity as possible in the software as 

opposed to the hardware

• Conduct critical system engineering tasks without software expertise

• Believe that formal reviews alone will provide an accurate picture of 

the project

• Expect that the productivity of a formal review is directly proportional 

to the number of attendees above five

Software Acquisition Worst Practices
DAU PROGRAM MANAGERS TOOL KIT

99

chapter 2

lEADERSHIP and MANAGERIAL SKILLS

• More things that make you go “Hmmm?... ”

“An authority is a person who just happens to know the source.”

“A conservative is a person who believes nothing should be done 

the first time.”

“Diplomacy is the art of hearing all parties arguing in a dispute and 

nodding to all of them without ever agreeing with any of them.”

“The meeting raised our confidence that the contractor can actually 

accomplish the task and that it will occur in our lifetime.”

“This is the earliest I’ve been late.”

“The world would be a much better place if people weren’t allowed 

to have children until they’ve proven they can successfully manage 

a DoD program.”

“Everyone is bound to bear patiently the results of his/her own 

example.”

“The superior person is firm in the right way, and not merely firm.”
DAU PROGRAM MANAGERS TOOL KIT

100

MANAGEMENT and leadership

Implement

Change Plan

Coordinate Staff

Monitor and

Control

Drive

Change

Set the

Direction

Align the

People

Energize the

People

Build

Relationships

Coach and

Mentor

Make Things Happen

Create and Nurture an Environment for Success

Demonstrate Integrity

Do Things Right

MANAGEMENT

LEADERSHIP

Do the Right Things

Program Managers Must Balance Both Roles

Organize
DAU PROGRAM MANAGERS TOOL KIT

101

Reasons for Empowerment, Delegation, and Coaching

• Allows managers more time for managerial and leadership roles (e.g., longterm planning, coordinating ongoing activities, monitoring and controlling 

activities, and providing feedback to employees)

• Increases employee capability and motivation

• Enhances employee career growth

• Improves teamwork

• Maximizes limited resources

• Pushes responsibility and accountability further down in the organization

Steps for Empowerment, Delegation, and Coaching

1. Select the task or tasks to be assigned

2. Select the person or team; evaluate their current capabilities to complete 

the task or tasks

3. Provide training and/or coaching, if necessary, to improve their 

capabilities

4. Solicit input from the person or team regarding the task or tasks

5. Agree on the tasks, objectives, responsibility, authority, and deadline

6. Provide guidance, assistance, and support, as necessary

7. Establish metrics to measure progress

8. Monitor progress

9. Provide feedback

10. Identify lessons learned

11. Evaluate performance

EMPOWERMENT, DELEGATION, and COACHING

EMPOWERMENT

Assigning an employee 

or team responsibility 

and authority to take 

actions and make 

decisions in pursuit 

of the organization’s 

goals.

DELEGATION

Assigning an employee 

(usually a subordinate) 

a specific task or tasks 

to complete.

COACHING

Providing employees 

with the tools, 

knowledge, and 

opportunities they 

need to develop their 

potential and increase 

their effectiveness.

NOTE: Some people use “empowerment” and “delegation” interchangeably, while others see a subtle 

distinction, e.g., delegation often refers to an individual, while empowerment is usually associated with 

groups or teams. Empowerment usually includes more authority and freedom related to making decisions 

and taking actions, while delegation is usually more bounded.
DAU PROGRAM MANAGERS TOOL KIT

102

EMPOWERMENT, DELEGATION, and COACHING

(Continued)

Leaders should ensure the components shown above are present.

CapabilityS

knowledge, skills, 

experience

Direction

organizational values, 

vision, purpose, goals

Authority

the right to take actions 

necessary to meet goals

FreedomSS

able to take initiative within 

prescribed boundaries 

Trust

organizational values, 

vision, purpose, goals

Resources

materials, facilities, 

people, money, time, etc. 

SSInformation

access to and sharing of 

essential information

Responsibility

or assignment to complete a 

specific activity or activities

Accountability

assignment to complete a 

specific activity or activities

Support

right tools and 

resources to do job

• Active Listening. Give your full attention. Focus 

on the message, not formulating your response 

to it. Establish and maintain eye contact, 

paraphrase key points, and avoid making 

judgments.

• Questioning. Ask questions to promote 

discovery of new knowledge and stimulate 

thinking. Use open questions that require some 

thought to complete.

• Giving Feedback. One of the most valuable, 

but least used tools in communication. People 

are often uncomfortable giving feedback to 

others, particularly when they believe it could be 

perceived as negative. Offer factual, specific, but 

non-judgmental (and unemotional) feedback.

• Sharing. Share your experiences. Make 

suggestions on overcoming difficulties or how to 

proceed.

Coaching Skills
DAU PROGRAM MANAGERS TOOL KIT

103

GENERIC IPT PROCESS

Identify a Need for a Team—Determine whether the creation of a team is the 

best method to accomplish the intended purpose.

Staff the Team—Determine what functional disciplines and organizations/activities 

need to be represented and who the team members will be.

Conduct Team Startup Activities—Conduct activities to get the team started, 

such as establishing operating agreements, assigning roles and responsibilities, and 

conducting team training sessions. Activities also include discussing and agreeing 

on the team’s intended purpose, and developing shared goals, critical success 

factors, and metrics to measure team progress toward goals. A common output of 

these activities is the Team Charter. (See page 105.)

Develop a Plan of Action—Take specific action steps or processes for how the 

team will perform. This includes assigning action items, establishing target dates, 

determining what resources are needed, etc.

Execute the Plan—Perform the work necessary to accomplish the project goals 

and produce the team deliverables.

Assessment and Realignment—Conduct periodic assessments of team 

performance, and use metrics to measure progress toward goals. Make adjustments 

as necessary.

Conduct Team Closeout Activities – Deliver the final product or service, 

update program documents, and compile lessons learned.

Identify the

Need for

an IPT

Staff the

Team

Conduct

Team Startup

Activities

Develop a

Plan of Action

ConductTeam

Closeout

Activities

Execute

the Plan

Assess and

Realign
DAU PROGRAM MANAGERS TOOL KIT

104

Formula for effecti

ve team performance

Direction + Commitment + Energy + Capability + Empowerment +

Effectiv

e

Team

Enablers

Performanc

e

Leadership

What’s In It

For The Customer? For Me? DA

U

PL

M OJ

T

Coaching

Mentoring

Individual Factors (Examples)

• Personal Standards

• Personal Interests

• Personal Values

• Career Goals

• Work Ethic

• Securit

y

• Welfare

*

Customer Factors (Examples)

• Improved Capabilit

y

• Improved Readiness• Reduced O&M Costs

• Moral

e

*Effective Team Enablers

• Communication• Collaboration• Contribution• Trust

OJT = On-the-Job TrainingDAU = Defense Acquisition UniversityPLM = Performance Learning Model

What’s In It
DAU PROGRAM MANAGERS TOOL KIT

105

Team Charter

Team Charter. A document describing key aspects of why a team is 

established, what is expected of it, and what authority and responsibil-

ity it has. The person or entity creating (i.e., “chartering” or authoriz-

ing) the team normally provides some general guidance; however, the 

team may benefit considerably by developing the “meat and potatoes” 

of the charter, resulting in increased commitment of all team members. 

Examples of topics that may be included in a charter follow:

• Purpose. Describe why the team exists and what it is intended to 

accomplish.

• Goals/objectives. List specific, measurable items the team is 

focused on achieving to help it exceed its customer’s expectations.

• Critical success factors. List the critical actions the team must 

perform to ensure it is successful in fulfilling its purpose.

• End products/deliverables. Describe the item(s) the team is 

responsible for delivering.

• Authority and accountability. Describe what team members are 

allowed/not allowed to do without authorization from a higher level. 

Describe what they are responsible for completing.

• Metrics. List measures of progress for critical success factors and 

goals/objectives.

• Program schedule. List key program/team milestones and events.

• Team membership. List team members and contact information.

• Roles and responsibilities. List specific assignments for improving 

team performance (e.g., time keeper, recorder or scribe, scheduler, 

etc.). Also, list specific tasks and/or action items the team is assigned 

to complete. 

• Resources required. Describe the funding, materials, equipment, 

support, etc., the team needs to complete its mission.

• Program organizational structure. Define where the team fits within 

the overall program office structure.

• Program organizational structure Describe or depict where the 

team fits in the overall program office structure.

• Operating agreements/ground rules. List agreed-upon guidelines 

describing how team members will interact, what processes they will 

use, and what they expect of one another. 

• Customers, suppliers, stakeholders. List key individuals, teams, 

and organizations involved with the team’s output.
DAU PROGRAM MANAGERS TOOL KIT

106

TYPICAL WORKING GROUPS

• Logistics Support Management Team (LSMT)

• Test and Evaluation Working Group (TEWG)

• Computer Resources Working Group (CRWG)

• Requirements Interface Working Group

• Interface Control Working Group (ICWG)

• Technology Assessment Working Group

• “Tiger” Team

• Process Action Team (PAT)

• Integrated Product and Process Teams (IPPTs)

RECOGNIZE WHICH PHASE OF

TEAM DEVELOPMENT YOU ARE

IN AND TAKE POSITIVE ACTION

TO WORK THROUGH.

WORKING GROUPS

TEAM DEVELOPMENT WHEEL

Performing

Creative

Trusting

Effective

Confident

Forming

Milling

Confusion

Polite

Purposeless

Norming

Cohesion

Purpose

Feedback

Relevancy

Storming

Conflict

Frustration

Resistance

Cliques

Note: There can be an additional phase—“Adjourning”—when the team disbands, 

says good bye, and reflects on lessons learned. This is a “celebration” phase.

This diagram is based on Dr. Bruce Tuckman’s 1965 study of small groups, which 

identified the traditional five phases experienced by project work teams.
DAU PROGRAM MANAGERS TOOL KIT

107

 Disadvantages

• Takes more time

• Hard to terminate

• Paralysis by analysis

MANAGEMENT TRADEOFFS FOR WORKING GROUPS

 Advantages

• More ideas and solutions

• Consensus positions

• Strong commitments

TEAM PERFORMANCE MODEL

• Decision Making

• Resolving Issues

• Communicating

• Planning

• Executing

• Controlling

• Customer Focus

• Leadership

• Values

• Vision

• Purpose

• Goals and Objectives

• Critical Success Factors

• Awareness

• Roles and Responsibilities

• Operating Agreements

• Team Accountability

• Empowerment

• Trust

• Five Cs

• Team Identity

• Self-Assessment

Communication

Commitment

Cooperation

Contribution

Caring

Team Processes

Team Principles

Team Dynamics

Team Foundation

Thinking

Learning

Charter

• Diversity

• Conflict

• Comfort Zones

• Communications

• Focus

• Organizational Climate

• Trends
DAU PROGRAM MANAGERS TOOL KIT

108

—Don’t try to force consensus. Listen to other positions and 

reactions before expressing your own point. 

—No winners or losers. Don’t assume that someone must “win” 

and someone must “lose” if the discussion reaches a stalemate.

—Don’t avoid conflict. Don’t change your mind simply to reach 

agreement and maintain harmony.

—Avoid majority votes, compromises, or horse trading to reach an 

agreement.

—It’s OK to disagree. Differences of opinion are natural and 

expected.

Note: Groupthink. A phenomenon—to be avoided—where team members become so concerned 

about preventing disagreement or conflict that they abandon critical thinking to simply go along with 

whatever consensus seems to be emerging.

Team Decision Making

Good team decision making is a critical element of team performance. It involves 

examining the decision context (e.g., current program environment, assumptions, constraints, pressures, stakeholder inputs, etc.), determining who needs to be involved in 

the decision, verifying how much time is available to make the decision, and deciding 

on the decision-making process.

Generally Accepted Team Decision-Making Methods

• Unilateral. One person makes the decision, usually the team leader. 

Variations:

— Directive or Authoritative. The person making the decision does so primarily 

using his/her knowledge, experience, and program guidelines/constraints, but 

is also influenced by his/her own reasons and motives.

— Consultative. The person making the decision may seek input from other 

team members, but ultimately, he/she still makes the decision on his/her own.

• Majority. Each team member votes, and the majority decides the course of action.

• Consensus. Team members may not completely agree with the most preferred 

approach, but they have the opportunity to express their point of view, understand 

the logic behind the decision, and support it. Consensus is generally the preferred 

decision-making method for most team issues, especially when the commitment of 

all team members is important.

Guidelines for Achieving Consensus:
DAU PROGRAM MANAGERS TOOL KIT

109

Effective Meetings

PriorTo the Meeting

• Determine and clarify the purpose for the meeting

• Determine expected meeting outcomes

• Identify meeting attendees

— Subject matter experts

— Key decision makers

— People directly affected by potential 

decisions/outcomes

• Determine meeting format

— Face-to-face, virtual teleconference, 

teleconference, Web tool

• Determine date/time/location

• Develop and distribute meeting agenda (at least 24 hours prior)

— Specific topics, presenter, estimated time, desired outcome

• Meeting logistics

— Room setup, IT support needed

During the Meeting

• Opening

— Start on time

— Review agenda

— Set or review ground rules

— Clarify roles

• Conducting

— Address one item at a time

— Facilitate discussions

— Encourage open communication and information sharing

— Maintain focus and pace

— Specify topics, presenter, amount of time devoted to item

• Closing

— Summarize agreements and decisions

— Review action items

— Ask for agenda items for the next meeting

— Set the date / time of the next meeting

After the Meeting

Review and publish minutes
DAU PROGRAM MANAGERS TOOL KIT

110

DECISION BRIEFING

Elements of a Decision Briefing

• Outline—Agenda

• Purpose of Briefing/Issue(s)

• Background

• Assumptions

• Alternatives Identified

• Evaluation Criteria/Process

• Analysis of Identified Alternatives

• Recommended Alternative

• Rationale for Recommendation

• Recommended Implementation Plan

• Key Risks for Recommended Implementation Plan

What to Expect from the Person/People Receiving the Briefing

• Challenges to assumptions, definitions, methodology

• Questions concerning compliance with or 

changes to policy

• Sensitivity of the issue and/or 

recommended alternative to change

• Questions or challenges to analysis, 

tradeoffs, rationale for recommendations, and implementation plan

• Questions concerning risks for the recommended implementation 

plan

NOTE: Questions may be open-ended or closed (e.g., yes/no answers)
DAU PROGRAM MANAGERS TOOL KIT

111

Messages pass through filters; first through the filter of the person 

sending the message, and then through the filter of the receiver. Filters 

sometimes act to enhance the message, and at other times, they can 

be barriers. Filters consist of factors such as personality, tone of voice, 

body language, facial expressions, accents, perceptions, attitudes, 

emotions, knowledge, functional background, the medium of communication used (verbal, written, e-mail, etc.) and much more. Each 

person’s filter is different, sometimes resulting in the receiver interpret-

ing the message differently than the sender intended.

One of the most important communications skills (and often a barrier 

to effective communications) is listening. Learning to “actively listen” 

can increase communications effectiveness significantly

Active listening involves:

• Establishing and maintaining eye contact.

• Focusing on what is being communicated.

• Not making judgments about the sender’s 

information.

• Not formulating your reply before the sender has 

finished sending his/her message.

• Paraphrasing key points the sender makes (when the sender 

pauses—don’t interrupt to paraphrase what’s being communicated).

Effective program management requires 

the right people to get the right information 

at the right time. Program communications 

must take place vertically (up and down), 

horizontally, and externally.

Program Office

COMMUNICATIONS

Message

Feedback

Sender Receiver

Filter

Filter
DAU PROGRAM MANAGERS TOOL KIT

112

Communications (Continued)

Communications Plan

One way to ensure the right people get the right information at the right times is to develop a 

program (and/or team) communications plan. The plan may include:

• Key entities (program management leadership, IPTs, customer, 

contractor(s), and key stakeholders).

• What information they should provide.

• What information they should receive.

• How it is provided/received.

• Format, frequency/interval, and other factors considered important for the particular 

program/situation.

• Types of meetings, such as regular status meetings and program management reviews.

• Reports (e.g., status reports, cost/sched perf reports, action item lists).

• Issues and the policy for elevating them to higher levels.

• Other forms of communication, and how and by whom they are used.

Interpersonal Negotiation Techniques

Purpose: Resolving conflicts

Objective: Seek to satisfy both parties’ interests

Methodology:

— Acknowledge the conflict and its effect on performance.

— Separate people and emotions from the issue.

— Present issues in terms of the underlying interests or 

requirements, i.e., the most important aspects of what you need 

to achieve.

— LISTEN to the other party’s interests/requirements; be able to restate their interests to 

their satisfaction (indicating you understand what interests they are trying to achieve).

— Agree on what the issue is.

— Look for common goals and common interests.

— Identify as many possible alternatives to resolve the issue and satisfy the interests of 

both parties.

— Resist the urge to compromise (“meet in the middle”). Instead, look at the issue from 

different perspectives: Challenge assumptions and constraints.

— Agree on the alternative that best meets both parties’ interests.

— Obtain the commitment of all members of both parties on what will be done to implement 

the solution.
DAU PROGRAM MANAGERS TOOL KIT

113

DIRECTIVE

• Give advice

• Evaluate

• Motivate

• Explain

• Reassure

NONDIRECTIVE

• Don’t display authority

• Listen carefully

• Don’t advise

• Facts only; no opinions

• Employee find solution

Advantages

• Effective with inexperienced

 personnel

• Quick

• Take charge attitude

Advantages

• Develops commitment

• Good training

• Employee responsible

• Supports delegation

Disadvantages

• Perceived insulting

• Does not support delegation

• Manager keeps responsibility

Disadvantages

• Takes time

• Skill/patience required

• Ineffective with

 inexperienced personnel

COUNSELING PROCESS

1. Set up interview—private, confidential, 

 unhurried

2. Encourage discussion—open questions, 

 active listening

3. Help employee think it through—deal with

 facts, no opinions or own views

4. Let employee find the solution—his/her solution

to the problem 

COUNSELING

COMMUNICATIONS (Continued)

Counseling
DAU PROGRAM MANAGERS TOOL KIT

114

Time Management

1. List all the tasks you have to complete.

2. Prioritize the tasks based on urgency and importance of completion 

using the format shown below.

3. Do Priority 1 tasks first. If possible delegate some of them.

4. The key to effective time management is to schedule time to work on 

small pieces of Priority 2 tasks. 

— If not completed early, they will eventually become Priority 1 tasks.

5. Reassign or delegate Priority 3 tasks if possible.

— A common tendency is focusing on Priority 3 tasks (because of their 

urgency) instead of Priority 2 tasks (because of their importance).

6. Priority 4 tasks are time wasters/busy work and should be avoided.

Priority 1 Important Priority 2 Important

Urgent Not Urgent

Priority 3 Urgent Priority 4 Not Urgent

Not Important Not Important

 Common Time Robbers Avoidance Techniques

• Incoming telephone call ] Screen for importance

 ] Allow voice mail to pick up the call

 ] Limit length of calls (e.g., 2 min.)

• Outgoing telephone calls ] Do as many at one time as possible

 ] Itemize topics before calling

 ] Stick to the topic; don’t socialize

• Unscheduled visitors ] Screen for importance

 ] Do not invite visitor into your office

 ] Remain standing

 ] Schedule a time for visitor to return

• Improper delegation ] Re-delegate

 ] Make a record of delegated tasks

 ] Assign deadlines

• Poorly conducted meetings ] Have a pre-published agenda

 ] Stay focused on subject

 ] Use a time keeper/gate keeper
DAU PROGRAM MANAGERS TOOL KIT

115

Management Tools and Techniques

1. Activity-Based Management (ABM). Uses detailed economic 

analyses of important business activities to improve strategic 

and operational decisions. ABM increases the accuracy of 

cost information by more precisely linking overhead and other 

indirect costs to products or customer segments. Traditional 

accounting systems distribute indirect costs using bases such 

as direct labor hours, machine hours, or material dollars. ABM 

tracks overhead and other indirect costs by activity, which can 

then be traced to products or customers.

2. Balanced Scorecard. Defines what management means 

by “performance” and measures whether management is 

achieving desired results. The Balanced Scorecard translates 

mission and vision statements into a comprehensive set of 

objectives and performance measures that can be quantified 

and appraised. These measures typically include: financial, 

customer value, internal business process, learning and growth, 

and employee performance.

3. Cycle Time Reduction. Decreases the time it takes a company 

or program to perform key activities throughout its value chain. 

Cycle Time Reduction uses analytic techniques to minimize 

waiting time, eliminate activities that do not add value, increase 

parallel processes, and speed up decision processes within an 

organization. Time-based strategies often emphasize flexible 

manufacturing, rapid response, and innovation in order to 

attract the most profitable customers.

4. Groupware. Refers to a broad range of technologies that allow 

people in organizations to work together through computer 

networks. These products range from sophisticated electronic 

mail packages to applications that link offices and employees. 

Organizations use such technology-aided communications 

to better inform strategic and financial decisions and to more 

effectively and economically bring together working groups. 

(DAU has a Groupware capability in its Management Decision 

Center, which is used for management decision making by 

offices and agencies throughout DoD.)
DAU PROGRAM MANAGERS TOOL KIT

116

Management Tools and Techniques

(Continued)

5. Outsourcing. Occurs when a company or Government agency 

uses third parties to perform non-core business activities. 

Contracting third parties enables a company or agency to focus 

its efforts on its core competencies. Many companies find that 

outsourcing reduces cost and improves performance of the 

activity. Third parties that specialize in an activity are likely to 

be lower cost and more effective, given their scale. Through 

outsourcing, a company or agency can access the state of the 

art in all of its business activities without having to master each 

one internally.

6. Business Process Reengineering. Involves the fundamental 

redesign of core business processes to achieve significant 

improvements in productivity, cycle times, and quality. In 

Business Process Reengineering, companies start with a 

blank sheet of paper and rethink existing processes to deliver 

more value to the customer. They typically adopt a new value 

system that places increased emphasis on customer needs. 

Companies and/or Government agencies reduce organizational 

layers and eliminate unproductive activities in two key areas. 

First, they redesign functional organizations into crossfunctional teams. Second, they use technology to improve data 

dissemination and decision making.

7. Strategic Planning. Is a comprehensive process for 

determining what a commercial business or Government 

agency should become and how it can best achieve that goal. 

It appraises the full potential of a business and explicitly links 

the business objectives to the actions and resources required 

to achieve them. Strategic Planning offers a systematic process 

to ask and answer the most critical questions confronting a 

management team—especially large, irrevocable resource 

commitment questions.
DAU PROGRAM MANAGERS TOOL KIT

117

cHAPTER 3

PROBLEM-SOLVING TOOLS

BRAINSTORMING

PURPOSE: To stimulate the free flow of ideas.

METHOD: Group members take turns generating ideas. One idea 

stimulates another and then another. Freewheeling of ideas is encouraged. Brainstorming stops when all group members run out of ideas. 

The next page lists questions that may suggest new ideas for you.

GROUND RULES:

Put prejudgment aside. Remember, all ideas can be thought of 

as starters.

No criticism allowed. This is not the time to judge an idea. Don’t 

criticize other ideas no matter how ridiculous they may seem. 

The ideas can be discussed in detail later; at this time, the objective is to generate more ideas.

Welcome free-wheeling or blue-skying. Let those wild ideas 

come out—otherwise you may conceal your creative process. 

The impractical ideas may trigger other ideas that are possible to 

use.

Strive for quantity, not quality. The more ideas brought out, the 

better the chance of a great solution.

Combine and rearrange ideas. Single ideas aren’t the only way 

to make a suggestion. You can make additions or combinations 

of previously suggested ideas to create still better ideas.

Record all ideas exactly as expressed. This keeps the mind 

free of remembering what was said and allows you to build on 

previous ideas.
DAU PROGRAM MANAGERS TOOL KIT

118

BRAINSTORMING

(Continued)

Why does it work? 

Some of the reasons why brainstorming enhances a group’s 

creativity are that it:

• Increases involvement and participation.

• Produces the most ideas in the shortest time.

• Reduces the need to give the “right” answer.

• Frees up the group; allows the members to have fun and is 

interesting.

• Reduces the possibility of negative thinking.

QUESTIONS TO STIMULATE YOUR BRAIN CELLS:

1. Can we use this idea elsewhere? As is? With changes?

2. If we change it, is there anything else like it? Any related issues?

3. Modify? Change? Rearrange? Meaning, color, motion, sound, 

odor, taste, form, shape, layout, etc.?

4. Magnify? Add what? More, stronger, larger, newer?

5. Minimize? Subtract what? Eliminate, smaller, lighter, slower, split?

6. Substitute? Who, what, when, where?

7. Reverse? Opposite, backwards, upside down, inside out?
DAU PROGRAM MANAGERS TOOL KIT

119

Force Field Analysis

A Force Field Analysis illustrates the relationship and significance of 

factors that may influence the problem or goal. This analysis helps us 

better understand driving and restraining forces.

Used for making decisions, force field analysis can help:

• Identify realistic improvement opportunities;

• Develop systematic action plan for problem resolution; and

• Create criteria for evaluating effectiveness of improvement actions.

Cause-and-Effect Diagram 

The cause-and-effect diagram is a graphical illustration of the relationship 

between a problem or goal (the effect) and its potential contributors (the 

causes). Sometimes called the “fishbone” or Ishikawa diagram.

Used for analyzing problems, a cause-and-effect diagram

can help:

• Determine root causes of a given effect; and

• Identify areas where there is a lack of data.

GOAL: Stop Smoking

+ Driving Forces

Poor Health

Burned Clothing

Poor Example

Cost

Impact on Others

Restraining Forces –

Habit

Addiction

Taste

Advertisements

Stubborn

ENVIRONMENT PEOPLE MACHINE

POOR

PHOTOCOPY

QUALITY

METHODS MATERIALS

Original

Settings

Pencil

Hardness

Drying

Time

Bad Paper

Handling

Poor

Maintenance

Bad Settings

Contamination

Paper

Alignment

Hands

Dirty

Bad

Originals Humidity

Operating

Hours

Sharpness

Lamp Dirty

Speed

Roll Condition

Not Cleaning-up

Not Level

Poor

Quality

Poor

Storage

Copy Paper

Writing

Pressure
DAU PROGRAM MANAGERS TOOL KIT

120

Scatter Diagram 

A scatter diagram depicts the correlation between two variables (X 

and Y).

Used for data analysis, a scatter diagram can help:

• Confirm a hypothesis that two variables are related; and

• Provide both visual and statistical means to test the strength of a 

potential relationship.

Histogram 

The histogram chart displays the distribution of a measurable characteristic (for example: weight, length, speed, etc.). A histogram shows 

what the variability of the data is in a graphical or pictorial manner.

Used for data analysis, a histogram can help:

• Present a picture of how the process is operating;

• Compare actual process measurements with an expected 

distribution;

• Observe patterns in the data; and

• Investigate process stability.

frequency

measurable characteristics

X

Y
DAU PROGRAM MANAGERS TOOL KIT

121

Surveys

Surveys are used to collect data from a variable number of items or 

people for a comparative study. They are used when a new project is 

planned to prove the need and the demand of the customer.

Surveys can be used anywhere in the organization to find out specific 

information that is necessary to make improvements in a process.

Surveys:

• Are an inexpensive way to test a system or product

• Can be used with a large number of people or a small group

• Can give you an overall view, determined by the questions you ask

• Show if an organization is meeting its quality goals

• Help identify satisfied and dissatisfied customers or employees

Survey Process

1. Determine the group to be studied.

2. Determine what questions will be asked. 

Note: Train your data collectors thoroughly. Everyone must know how to ask 

the questions, whom to approach, and how to approach them.

3. Compile your results in chart form using a Pareto chart (see page 

126), histogram, and other tools that will give you clarification.

4. Use the compounded data to form a base for improvement.

5. Continue to take data to monitor improvements, and make sure 

the improvements you have made are working.

Caution!

• Data must be collected honestly and consistently.

• An untrained collector can skew the data to reflect personal biases.

• A poor, inconsistent survey will give you invalid data.

• Make sure there is enough time allowed for the collecting process. 

Brainstorming Survey
DAU PROGRAM MANAGERS TOOL KIT

122

Affinity Diagram

An affinity diagram is a technique for organizing verbal information 

into a visual pattern. An affinity diagram starts with specific ideas 

and helps you work toward broad categories. This is the opposite of 

a cause-and-effect diagram, which starts with the broad causes and 

works toward specifics. You can use either technique to explore all 

aspects of an issue. 

Affinity diagrams can help you:

• Organize and give structure to a list of factors that contribute to a 

problem; and

• Identify key areas where improvement is most needed.

How to do it:

1. Identify the problem. Write the problem or issue on a whiteboard or flip chart.

2. Generate ideas. Use an idea-generation technique to identify all 

facets of the problem. Use index cards or Post-it® notes to record 

the ideas.

3. Cluster your ideas (on cards or paper) into related groups.

Use questions like “Which other ideas are similar?” and “Is this 

idea somehow connected to any others?” to help group the ideas 

together.

4. Create affinity cards. For each group, create an affinity card, a 

card that has a short statement describing the entire group of ideas.

5. Cluster related affinity cards. Put all of the individual ideas in 

a group under their affinity card. Now try to group the affinity cards 

under even broader groups. You can continue to group the cards until 

your definition of “group” becomes too broad to have any meaning.

6. Create an affinity diagram. Lay out all of the ideas and affinity 

cards on a single piece of paper or a blackboard. Draw outlines 

of the groups with the affinity cards at the top of each group. The 

resulting hierarchical structure will give you valuable insight into 

the problem.
DAU PROGRAM MANAGERS TOOL KIT

123

Environment

Equipment Ergonomics

Training

Original Document

No Definition of 

Quality

Noise

Lighting

Desk Height

Chair Height

Comfort

Handwriting

Grammar

Punctuation

Spelling

Draft Copy

Final Copy

Distribution

Font

Computers

Printers

Typewriters

Typing Skill

Editing Skill

Computer Skill

Proofreading Skill

No Measurement

No Feedback

Technical Jargon 

Slang

Affinity Diagram

Interruptions

Unreasonable

Deadlines

Time of Day

Typographical Errors

Author Skill Requirements

Affinity Diagram (Example)

A publication team wanted to reduce the number of typographical errors 

in their program’s documentation. As part of a first step, they conducted 

a brainstorming session that produced the following list of factors that 

influenced errors.

Computers

Proofreading Skill

Unreasonable 

Deadlines

Lighting 

Chair Height

Desk Height

Technical Jargon 

Handwriting

Slang 

Spelling

Draft Copy

Distribution

Final Copy

Computer Skill

No Measurements

No Feedback

Printers

Noise

Typewriters

Comfort

Time of Day

Interruptions

Grammar

Punctuation

Font

Editing Skill

Typing Skill

The following affinity diagram helped them to focus on areas for further analysis.
DAU PROGRAM MANAGERS TOOL KIT

124

Pairwise Ranking

Pairwise ranking is a structured method for ranking a small list of items 

in priority order.

Pairwise Ranking can help you:

• Prioritize a small list; and

• Make decisions in a consensus-oriented manner.

How to do it:

2. Rank each pair. For each pair, have the group (using a consensus-oriented discussion) determine which of the two ideas is 

preferred. Then, for each pair, write the number of the preferable 

idea in the appropriate box. Repeat this process until the matrix is 

filled.

1

2 2

3

 4

2

3

4

5

1 and 2 compared

2 is better

1

2 2

1 3

 4

2

3

4

5

1 and 3 compared

1 is better

1

2 2

1 2 3

1 2 3 4

5 5 5 5

2

3

4

5

4 and 5 compared

5 is better

...and so on 

until...

1

 2

3

 4

2

3

4

5

1. Construct a pairwise matrix. Each box in 

the matrix represents the intersection (or 

pairing) of two items. If your list has five 

items, the pairwise matrix would look like 

this, with the top box representing idea 1 

paired with idea 2.
DAU PROGRAM MANAGERS TOOL KIT

125

Alternative 5

ranks 1st overall

Alternative 5 appears

4 times in the matrix

Alternative 1 2 3 4 5 

Count 2 3 1 0 4 

Rank 

Alternative 1 2 3 4 5 

Count 2 3 1 0 4 

Rank 3rd 2nd 4th 5th 1st

Pairwise Ranking (Example)

A program team was asked to recommend a site for testing a unique portion of a system. A feasibility study produced a list of six possible locations. 

The team then used pairwise ranking to determine that Nellis AFB was best 

suited for this particular test.

1. Fort Huachuca 4. Nellis AFB

2. Edwards AFB 5. Eglin AFB

3. Kirtland AFB 6. Hanscom AFB

1

2 2

1 3 3

4 4 4 4

5 5 5 4 5

1 6 6 4 5

2

3

4

5

6

3. Count the number of times each alternative appears in the 

matrix.

4. Rank all items. Rank the alternatives by the total number of 

times they appear in the matrix. To break a tie (where two ideas 

appear the same number of times), look at the box in which those 

two ideas are compared. The idea appearing in that box receives 

the higher ranking.

Site

Count

Rank

1

2

3rd

2

1

6th

3

1

5th

5

4

2nd

6

2

4th

4

5

1st
DAU PROGRAM MANAGERS TOOL KIT

126

Pareto Chart

A bar chart that arranges contributing factors/causes to a problem in 

order with respect to their degree of contribution to the problem.

Used for analyzing problems, a Pareto chart can help:

• Select improvement opportunities;

• Identify root causes with greatest impact from a cause and effect 

diagram;

• Check results of improvement efforts by comparing Pareto charts 

before and after action is taken.

Customer Complaints

(Food Service)

Poor

Service

 Cost Quantity Taste Other

Causes

%

C

o

n

t

r

i

b

u

t

i

o

n

40

30

20

10
DAU PROGRAM MANAGERS TOOL KIT

127

Benchmarking

Benchmarking is the process of measuring products, services, and 

practices against the toughest competitors or those known as leaders 

in their field. Benchmarking can help you:

• Understand how you compare with similar organizations; and

• Identify areas for process improvement.

How to do it:

Identify the process to be benchmarked. Select a process (as opposed to a product) that is important to both your organization and your 

customers. Be sure the process in your organization is similar to, and 

measured in the same manner as the one to which it’s being compared.

Study other organizations. Develop a list of organizations with 

comparable products and services. Determine what specific processes the organization performs. Based on this information, rank the 

organizations from best to worst.

Compare and evaluate. Compare your process to the best and 

worst cases and list the important differences. These differences can 

suggest potential improvements to your process.

Note: Benchmarking is not replicating a process from an organization that excels 

(unless your goal is to be 2nd best). It is studying the process, clearly understanding 

the theory behind the process, and then restudying your own process to determine 

improvements.

Benchmarking Example:

Using inputs their customers provided, the executive leaders at AF 

Product Division B decided that 

their source selection process 

needed improvement. As part of 

the initial analysis, they wanted to 

see how their process compared 

with others. They determined 

that the average number of days 

required for source selection was 

an important process measure. 

As a result of this analysis, 

representatives visited AF Product 

Division A and Navy Division B 

and studied their source selection 

procedures.

Cash

received

Accounts

receivable

Sale

(DD 250) Finished goods

inventory

Raw material

inventory

Contract

award

Cash

disbursed

Cash

disbursed

Wages

payable

Work in process

inventory

Accounts

payable
DAU PROGRAM MANAGERS TOOL KIT

128

Flowcharting 

A flowchart is a graphic representation of the steps of a process. Flowcharts help us understand the process by mapping out the steps in as 

much or as little detail as needed.

Enlisted Accessions and Training

Used for analyzing a process, a flowchart can help:

• Understand the existing process;

• Locate improvement areas in a process;

• Document changes to a process;

• Show relationships between different steps in a process; and

• Identify critical stages of a process.

There are standard flowchart symbols. When you are developing a 

flowchart, especially in a group environment, the goal is to chart the 

process. Don’t waste time debating which shape a symbol should 

be. A flowchart that doesn’t use the standard symbols can be just as 

useful as a chart that does use them.

fIrST DUTY

ASSIGNMENT

HAS

PrIOr

SErVICE

rECrUITING

DISCHArGED

YES ASSIGNED

TO A 

SPECIALTY

rEQUIrES

TrAINING

TECH

TrAINING fAILS

YES

YES

YES

NO

NO

YES

NO

NO

NO

rETAIN

ENTErS

BASIC

TrAINING

PASSES
DAU PROGRAM MANAGERS TOOL KIT

129

Standard Flowchart Symbols

Deployment Flowcharts

A deployment flowchart shows the process flow and the work groups 

involved in each step. It provides a graphic representation of a given 

process or system work groups, or individuals responsible for each 

activity.

A deployment flowchart is used anytime individuals or groups need to 

analyze a process in order to improve a system.

This Symbol... Represents...

Start/Stop

Decision Point

Activity

Document

Connector (to

another page

or part of the

diagram)

Some Examples:

Receive Trouble

Report

Machine operable

Approve/

Disapprove

Accept/Reject

Yes/No

Pass/Fail

Drop off travel

voucher

Open access

panel

Fill out trouble

report

A

B
DAU PROGRAM MANAGERS TOOL KIT

130

Deployment flowcharts are used for various functions such as training 

agenda, daily schedule, meeting analysis, emergency procedures, purchasing process, communication procedures, maintenance process, etc.

Deployment Flowcharts:

• Identify involvement in a process, as it relates to the whole process.

• Define work processes, and identify existing loops through people or 

departments.

• Visualize a process or system.

Deployment Flowcharts (continued)

Time correlation if applicable

Process Identification

Cast of Characters

Task Box

Circle denotes added support

Meeting

Decision Diamond

Decision Line

Shadow box means more

information on another flow chart

Report

End

Source: Dr. Myron Tribus 

Deployment Flow Charting 

Quality & Productivity, Inc., 

Los Angeles, CA 90024

Deployment Process

1. Select a process or system to analyze.

2. Identify the cast of characters (people involved in the process).

3. Document the existing process using the flowchart symbols.

4. Discuss changes to be made in the process with all those 

involved with the process.

5. Update the deployment flowchart with the proposed changes 

and implement the new process

6. Study the effectiveness of the change and return to step 1 above.

NO

YES
DAU PROGRAM MANAGERS TOOL KIT

131

Nominal Group Technique (NGT)

Ranking of consensus

Why?

Allows a team to come to consensus on relative importance of issues, 

problems, or solutions by combining individual importance rankings 

into a team’s final ranking.

What?

• Builds commitment to the team’s choice through equal 

participation in the process

• Allows every team member to rank issues without being 

pressured by others

• Puts quiet team members on an equal footing with more 

dominant members

• Makes a team’s consensus (or lack of it) visible; the major 

causes of disagreement can be discussed.

How to do it:

1. Generate the list of issues, problems, or solutions to be prioritized

In a new team with members who are not accustomed to team participation, it 

may feel safer to do written, silent brainstorming, especially when dealing with 

sensitive topics.

2. Write statements on a flip chart or white board

3. Eliminate duplicate and/or clarify meanings of any of the statements

As a leader, always ask for the team’s permission and guidance 

when changing statements.

4. Record the final list of statements on a flip chart or white board

Example: Why does faculty have inconsistent output?

 A Lack of training 

 B No documented process

 C Unclear quality standards

 D Lack of cooperation with other departments in developing 

 standards

 E High turnover

Use letters rather than numbers to identify each statement so that team members 

do not get confused by the ranking process that follows.
DAU PROGRAM MANAGERS TOOL KIT

132

5. Each team member records the corresponding letters on a piece of

paper and rank orders the statements

 Example: Larry’s sheet of paper looks like this:

 A 4

 B 5

 C 3

 D 1

 E 2

This example uses “5” as the most important ranking and “1” as the least important. Since 

individual rankings will be later combined, this “reverse order” minimizes the effect of team 

members leaving some statements blank. Therefore, a blank (value = 0) would not, in effect, 

increase its importance.

6. Combine the rankings of all team members

John Paul George Ringo Mary Total

A 4 5 2 2 1 = 14

B 5 4 5 3 5 = 22

C 3 1 3 4 4 = 15

D 1 2 1 5 2 = 11

E 2 3 4 1 3 = 13

“No documented process,” B, would be the highest priority. The team 

would work on this first and then move through the rest of the list as 

needed.

Variation:

Weighted Multivoting

Each team member rates, not ranks, the relative importance of choices 

by distributing a value, e.g., 100 points, across the options. Each team 

member can distribute this value among as many or as few choices as 

desired.

Example:

John Paul George Ringo Mary Total

A 20 10 = 30

B 40 80 50 100 45 = 315

C 30 5 10 25 = 70

D 5 10 20 = 35

E 10 10 20 10 = 50

With large numbers of choices, or when the voting for the top choices 

is very close, this process can be repeated for an agreed-upon number 

of items. Stop when the choice is clear.

Nominal Group Technique (NGT) (Continued)
DAU PROGRAM MANAGERS TOOL KIT

133

CREATIVE PROBLEM SOLVING

PROCESS STEPS

1. List perceived problems

2. Gather relevant data

3. Define actual problem

4. Determine alternative solutions

5. Analyze and evaluate alternatives

6. Select solution

7. Validate solution

DIVERGENT THINKING*

1. Accept all ideas and alternatives

2. Defer judgment or evaluation

3. Discuss, combine, hitchhike, improve ideas

4. When exhausted, move to converge

CONVERGENT THINKING*

1. Establish categories of alternatives

2. Develop evaluation criteria

3. Avoid premature closure

4. Keep eye on objective

5. List strengths and weaknesses

6. Select best alternative or idea

*Used sequentially during all problem-solving steps
DAU PROGRAM MANAGERS TOOL KIT

134

KNOT CHART

As you work your way through the problem, everything should move 

into the left column – Know.

The Knot Chart is useful for:

• Initially sorting the wheat from the chaff

• Organizing/coordinating the next steps of the problem-solving 

process

    Think

Know Need to Know Opinion We Know
DAU PROGRAM MANAGERS TOOL KIT

135

QUALITATIVE PROBLEM SOLVING

(Kepner - Tregoe)1

Deviation Statement: (Describe the actual performance vs. should performance)

1. Define deviation.

2. Describe what deviation IS and IS NOT.

3. List distinctions between what deviation IS and IS NOT.

4. Do distinctions indicate or suggest a change?

5. Determine possible causes based on distinctions and changes.

1

Copyright Kepner Tregoe, Inc. (1981). All rights reserved. Reprinted with permission. (Kepner-Tregoe, 

Inc., Research Road, P.O. Box 704, Princeton, N.J. 08542)

Specifying

Question

What?

(Identify)

Where?

(Location)

When?

(Timing)

Extent?

(Magnitude)

Does the distinction

suggest a change?

What is distinctive about

“Is” vs. “Is Not”? Is Is Not

Possible Causes:

Most Likely Cause:
DAU PROGRAM MANAGERS TOOL KIT

136

GANTT CHART

A Gantt Chart is used for planning schedules and managing projects. 

It is a method for basic planning and work instruction.

How to do it:

1. The Gantt Process begins by listing the activities of a project in 

order of execution.

2. Place the number of each activity across the top of your chart. 

Time duration such as days, weeks, years, etc., can replace 

activity numbers if appropriate.

3. Draw vertical lines across the chart for each item.

4. Starting with number 1, begin comparing the activities. Can 

number 1 be done at the same time as number 5 or 6?

5. Draw horizontal lines to indicate which activities can be done 

simultaneously.

6. You now have an overview of your project giving you a starting 

point and time-saving measures to help you complete the project 

on time. 

ACTIVITIES 1 2 3 4 5 6

1. Requirements are written

2. Finances are arranged

3. Bidding takes place

4. Contractor is selected

5. Prototype is built

6. Testing begins

ACTIVITIES

1. Requirements are written

2. Finances are arranged

3. Bidding takes place

4. Contractor is selected

5. Prototype is built

6. Testing begins

Published by the

Defense Acquisition University

Fort Belvoir, Virginia 22060-5565