(QVXULQJ

6XFFHVVIXO

,PSOHPHQWDWLRQ

RI

&RPPHUFLDO

2IIWKH6KHOI

3URGXFWV

&276

LQ

$LU)RUFH6\VWHPV

6WXG\&KDLU-HII*UDQW

)HEUXDU\

7KLV JXLGHERRN GRHV QRW UHSUHVHQW WKH RSLQLRQV RI WKH

'HSDUWPHQW RI 'HIHQVH RU WKH 'HIHQVH 6\VWHPV 0DQDJHPHQW &ROOHJH

DQG DUH VROHO\ WKRVH RI WKH DXWKRU

3XEOLVKHG E\ WKH 'HIHQVH 6\VWHPV 0DQDJHPHQW &ROOHJH 3UHVV

)RUW %HOYRLU 9LUJLQLD
ii
iii

Contents

1 Background ...................................................................................................................... 1

1.1 The Study Initiative................................................................................................ 1

1.2 The Study Chair ..................................................................................................... 1

1.3 Existing Information on COTS.............................................................................. 2

1.4 Scope of the Study ................................................................................................. 2

1.5 Results of the Study ............................................................................................... 3

1.5.1 Quantified Results ................................................................................... 3

1.5.2 Qualitative Results................................................................................... 3

1.6 Goals of the Study.................................................................................................. 4

1.7 Overview of the Study Conclusions....................................................................... 4

2 COTS and New Thinking................................................................................................ 5

2.1 New Thinking ........................................................................................................ 5

2.2 Old Thinking.......................................................................................................... 5

2.3 Transition from Old to New Thinking ................................................................... 5

2.3.1 New Thinking, New Culture, New Organizations .................................. 5

2.3.1.1 Cultural Change ...................................................................... 5

2.3.1.2 Organizational Change ........................................................... 6

2.4 Vehicle for Transition from Old to New Thinking................................................ 6

2.4.1 The Decision Process for Implementing COTS ...................................... 6

2.4.1.1 Historical Context of the COTS Decision Process ................. 6

2.5 Timetable for Transition from Old to New Thinking ............................................ 7

2.5.1 New Thinking Must be Adopted at Project Conception.......................... 7

2.5.1.1 Case Study: F-22 Program...................................................... 8

2.6 Inevitability of the Transition to COTS-Based Systems........................................ 8

2.7 Summary of the Need to Transition from Old to New Thinking........................... 9

3 The Present COTS Environment.................................................................................. 10

3.1 Theory vs. Reality................................................................................................ 10

3.1.1 Naivete Regarding the Complexity of Implementing COTS ................ 10

3.1.2 The Potentially High Cost of Product Integration ................................. 10

3.2 Complexity of COTS Implementation................................................................. 11

3.3 Profitability Issues................................................................................................ 11

3.3.1 Profits Flow from Defense to Commercial Contractors........................ 11

3.3.2 “Turf Protection” and Performance Claims........................................... 11

3.3.3 Imperative to Leverage Commercial Development............................... 12

3.3.4 Suppliers Adapt ..................................................................................... 12

3.4 Change in Buying Practices ................................................................................. 12

3.5 COTS Success in the Present Environment ......................................................... 13

3.5.1 Longevity in the System ........................................................................ 13

3.5.1.1 Requirement for Inside Assessment ..................................... 13

3.5.2 Hardware vs. Software........................................................................... 13

3.5.3 Total Ownership Cost (TOC) ................................................................ 13

3.6 Decision Process for COTS Success.................................................................... 13
iv

4 COTS and Modification ................................................................................................ 15

4.1 Overview of Conclusions on Modification.......................................................... 15

4.2 What is Modification?.......................................................................................... 15

4.2.1 Modification of Software....................................................................... 15

4.2.2 Modification of Hardware ..................................................................... 15

4.2.3 Modification, General Principle ............................................................ 15

4.3 Modification and Nomenclature .......................................................................... 16

4.3.1 Modified COTS is not COTS ................................................................ 16

4.3.2 Accounting and Quality Review............................................................ 16

4.3.3 Modified Non-developmental Items (NDI)........................................... 16

4.4 Case Studies on COTS Modification................................................................... 16

4.4.1 Scope of the Case Studies...................................................................... 16

4.4.2 Conclusions Based on the Case Studies ................................................ 16

4.4.3 Illustrative Findings in the Case Studies ............................................... 17

4.4.3.1 Overview............................................................................... 18

4.4.3.2 Mistakes Made by DMMIS .................................................. 18

4.4.3.3 Causes of DMMIS COTS Implementation Failure .............. 18

4.5 Hughes Electronics’ Experience with COTS Modification................................. 18

4.5.1 Hughes Electronics’ Repeated DMMIS’s Mistakes in COTS

 Implementation ................................................................................... 18

4.5.1.1 Inflexible Requirements........................................................ 18

4.5.1.2 Product Support .................................................................... 19

4.5.2 Specific “Red Flags” Identified in Hughes’ Experience with

 COTS Implementation........................................................................ 19

4.6 Summary Findings Based on the Case Studies.................................................... 19

5 Product Evolution .......................................................................................................... 20

5.1 Evolution and the Buying Process ....................................................................... 20

5.1.1 DoD Does Not Control COTS Evolution.............................................. 20

5.1.2 The Value of Buying Evolution............................................................. 20

5.1.3 Evolution and Product Support.............................................................. 21

5.1.3.1 Planning for Product Support ............................................... 21

5.1.3.2 Pace of the Upgrade Cycle.................................................... 21

5.2 Evolution and Modification ................................................................................. 21

5.2.1 Advantage in Buying Product Evolution............................................... 21

5.2.2 Evaluating Proposed Modifications in Light of Product Evolution ...... 22

5.2.3 Evolution and Integration of Modified Products................................... 22

6 Total Ownership Cost.................................................................................................... 23

6.1 Total Ownership Cost (TOC)............................................................................... 24

6.2 TOC and COTS Implementation Success............................................................ 23

6.2.1 Program-Wide Failure to Focus on TOC .............................................. 23

6.2.2 Flyaway Cost ......................................................................................... 23

6.3 Disincentives to Focus on TOC ........................................................................... 23

6.3.1 Congressional Budget Process............................................................... 23

6.3.1.1 Congressional Oversight....................................................... 24
v

6.3.1.2 Ongoing Funding Requirements........................................... 24

6.4 TOC and Systems Design .................................................................................... 24

6.4.1 Open Architecture.................................................................................. 24

6.5 Case Study: TOC and Navy Attack Submarine................................................... 24

6.5.1 COTS Cost-Effective............................................................................. 25

6.5.1.1 Total Cost-Effectiveness....................................................... 25

6.5.1.2 Other Cost Savings ............................................................... 25

6.5.1.3 Time Savings ........................................................................ 25

6.5.1.4 Performance Improvements.................................................. 25

6.5.1.5 Data Limitations ................................................................... 25

7 COTS and Program Criteria ........................................................................................ 26

7.1 Overview.............................................................................................................. 26

7.2 Program Criteria and Product Integration............................................................ 26

7.3 Program Criteria and SEI Certification................................................................ 26

7.3.1 Software Engineering Institute (SEI) Certification ............................... 26

7.3.1.1 SEI Certification and Risk Modification .............................. 26

7.3.1.2 SEI Certification, Scope ....................................................... 27

7.3.2 SEI Certification and COTS Success .................................................... 27

7.4 Environmental Criteria......................................................................................... 28

7.4.1 Over-Specification of Environmental Criteria....................................... 28

7.4.1.1 Balancing Environmental Criteria ........................................ 28

7.4.1.2 Response ............................................................................... 28

7.4.2 Case Study: Plastic Encapsulated Micro-Electronics (PEM) ................ 28

7.4.2.1 Background........................................................................... 28

7.4.2.2 Environment and Hermeticity............................................... 28

7.4.2.3 Survivability.......................................................................... 29

7.4.2.4 Verifying PEMS for Intended Environment......................... 29

7.4.3 Case Study: AWACS............................................................................. 29

7.4.3.1 Overview............................................................................... 29

7.4.3.2 Environmental Criteria Modified.......................................... 30

7.4.3.3 Result of Criteria Modification............................................. 30

7.4.4 Case Study: JDAM ................................................................................ 30

7.4.4.1 Overview............................................................................... 30

7.4.4.2 Environmental Criterion Modified ....................................... 30

7.4.4.3 Result of Criterion Modification........................................... 30

8 COTS and Best Practices .............................................................................................. 31

8.1 The Need to Identify Best Practices..................................................................... 31

8.2 Methodology for Identifying Best Practices ........................................................ 31

8.3 Best Practices ....................................................................................................... 32

8.3.1 Product Familiarity................................................................................ 32

8.3.1.1 Increased Buyer Participation............................................... 32

8.3.1.2 Decreased Participation From “On High” ............................ 32

8.3.2 Flexible Requirements........................................................................... 32

8.3.3 Software Wrappers ................................................................................ 32
vi

8.3.3.1 Larger Security Issues........................................................... 33

8.3.3.2 Proposed Security Solutions ................................................. 33

8.3.3.3 Recommendations on Security Outside the Scope of

 the Study Group’s Report .................................................. 33

8.3.4 Longevity in the COTS-Based System.................................................. 33

8.3.4.1 Case Study: Navy Attack Submarine.................................... 33

8.3.5 Leadership and a “Burning Platform” ................................................... 33

8.3.5.1 Case Study: Navy Attack Submarine and the

 “Burning Platform”............................................................ 33

8.3.6 Road Maps............................................................................................. 34

9 COTS Implementation .................................................................................................. 35

9.1 Overview.............................................................................................................. 35

9.2 Levels of Implementation Intensity ..................................................................... 35

9.2.1 SEI Categories of COTS Intensity......................................................... 35

9.3 Benefits of COTS Implementation ...................................................................... 35

9.4 Determining Whether COTS is Beneficial in a Particular Application............... 36

9.4.1 Sustainment and/or Logistics Costs....................................................... 36

9.4.2 Maintaining Modernization ................................................................... 37

9.5 Pitfalls of COTS Implementation ........................................................................ 38

9.5.1 Methodology Used to Evaluate Pitfalls ................................................. 38

9.5.2 “Corporate Retirement Plans” ............................................................... 38

9.5.2.1 Potentially Perpetual or Inordinate Corporate

 Involvement ....................................................................... 38

9.5.2.2 Considerations ...................................................................... 38

9.5.2.3 Response ............................................................................... 38

9.5.3 Artificially Quickened Change Cycle.................................................... 39

9.5.3.1 The Impact of the Change Cycle on Operational

 Readiness .......................................................................... 39

9.5.3.2 The Change Cycle and the End User .................................... 39

9.5.3.3 Considerations ...................................................................... 40

9.5.3.4 Response ............................................................................... 40

9.5.3.5 Artificially Quickened Change Cycles and the

 Selection Process ............................................................... 40

9.5.4 Human Skills Acquisition...................................................................... 41

9.5.4.1 Change Rate and Human Skills Acquisition......................... 41

9.5.4.2 Case Study: Software to Detect Man-Made Noise ............... 41

9.5.4.3 Response ............................................................................... 41

9.5.5 Integration.............................................................................................. 41

9.5.5.1 Integration and Unused Functionality................................... 41

9.5.5.2 Highly Integrated vs. Federated Systems ............................. 42

9.5.5.3 Trend Toward Greater Integration........................................ 42

9.5.5.4 Configuration Control........................................................... 42

9.5.6 Wrongway Evolution............................................................................. 42

9.5.6.1 Downsides of Product Evolution .......................................... 42

9.5.6.2 Considerations ...................................................................... 42

9.5.6.3 Response ............................................................................... 43
vii

9.5.7 Performance Claims .............................................................................. 43

9.5.8 Interoperability and Debugging............................................................. 43

9.5.8.1 Performance/Feature Clash................................................... 43

9.5.8.2 Source Code Issues ............................................................... 43

9.5.9 Mandates to Utilize COTS .................................................................... 44

9.5.10 Failure to Understand Costs .................................................................. 44

9.5.11 Static Roles and Responsibilities........................................................... 44

9.5.12 Logistical Volatility............................................................................... 44

9.5.13 Unrealistic Expectations ........................................................................ 45

9.5.14 Internal Use Products............................................................................. 45

10 COTS and Funding Issues............................................................................................. 46

10.1 Overview.............................................................................................................. 46

10.1.1 From Color of Money to TOC............................................................... 46

10.1.2 Funding Issues and Supporting Data ..................................................... 46

10.2 Traditional Funding Curve................................................................................... 46

10.2.1 Life Cycle .............................................................................................. 46

10.2.2 Color of Money...................................................................................... 46

10.3 COTS at Odds with Current Funding Processes.................................................. 47

10.3.1 Traditional Life Cycle Events vs. Commercial Evolution .................... 47

10.3.2 Funding Processes and Product Evolution ............................................ 47

10.3.3 Budget Cycles and Product Evolution................................................... 47

10.3.4 TOC ....................................................................................................... 47

11 Critical Factors in COTS Success................................................................................. 48

11.1 Background .......................................................................................................... 48

11.2 Factors Critical to COTS Implementation Success.............................................. 48

11.2.1 Flexible Requirements........................................................................... 48

11.2.2 Incentives to Contain Sustainment Costs .............................................. 48

11.2.2.1 AWACS................................................................................ 49

11.2.2.2 Navy Attack Submarine........................................................ 49

11.2.2.3 Other Contractual Settings.................................................... 49

11.2.2.4 Goal of Incentives................................................................. 49

11.2.3 Up-Front Awareness of Design Considerations .................................... 49

11.2.4 Matching COTS Functionality with Requirements ............................... 49

11.2.4.1 Evaluating Functionality Against Requirements .................. 50

11.2.5 Changing the Culture............................................................................. 50

11.2.5.1 From Linear to Spiral Processes ........................................... 50

11.2.5.2 Case Study: Lockheed .......................................................... 51

11.2.5.3 Risk-Sharing ......................................................................... 52

11.2.6 Using the Right Tools............................................................................ 52

12 Acquisition Strategy and COTS Success ..................................................................... 54

12.1 Overview.............................................................................................................. 54

12.2 Acquisition Strategy and Flexible Requirements................................................. 55

12.3 Acquisition Strategy and Industry and User Involvement................................... 55
viii

12.4 Acquisition Strategy and Open Architecture ....................................................... 55

12.5 Acquisition Strategy and Incentivizing Contractors ............................................ 55

12.6 Acquisition Strategy and TOC............................................................................. 55

12.7 Acquisition Strategy and Source Selection.......................................................... 56

12.7.1 Contractor Experience with COTS........................................................ 56

12.7.2 Contractor Processes.............................................................................. 57

12.7.3 Contractor Understanding of the Commercial Marketplace.................. 57

12.8 Acquisition Strategy and Product Selection......................................................... 57

12.9 Acquisition Strategy and COTS Modification..................................................... 58

13 The Training Imperative ............................................................................................... 59

13.1 Information Sharing ............................................................................................. 59

13.2 The COTS Training Imperative ........................................................................... 59

13.2.1 Training Emphasis................................................................................. 60

13.2.2 The Role of the Burning Platform ......................................................... 60

13.3 Training and Resource Allocation ....................................................................... 60

13.4 Structuring COTS Training.................................................................................. 61

13.5 Past Training and Information-Sharing Initiatives............................................... 61

13.6 Training Materials................................................................................................ 61

13.7 Training Must be Conducted within a Supportive Policy Infrastructure ............. 62

13.7.1 Case Study: Performance-Based Service Contracting........................... 62

13.7.2 Case Study: Revision of Major Acquisition Directives......................... 62

13.8 Training Must Incorporate Lessons Learned ....................................................... 63

14 Conclusions..................................................................................................................... 64

14.1 Summary .............................................................................................................. 64

14.2 The Road to COTS Success Is Long.................................................................... 64

15 Glossary........................................................................................................................... 66


%DFNJURXQG

7+(678'<,1,7,$7,9(

7KLVVWXG\ZDVVSRQVRUHGE\WKH$LU)RUFH6FLHQWLILF$GYLVRU\%RDUG7KHVWXG\ZDV

FRQGXFWHGODVW\HDU,WVILQDOUHSRUWLVQHDUO\DSSURYHGDQGZLOOEHDYDLODEOHVRRQ

)LJ0HPEHUVRIWKH&2766WXG\*URXS

7+(678'<&+$,5

-HII*UDQW&KDLUSHUVRQRIWKHVWXG\JURXSUHSRUWLQJRQ(QVXULQJ6XFFHVVIXO

,PSOHPHQWDWLRQRI&RPPHUFLDO2IIWKH6KHOI3URGXFWV&276LQ$LU)RUFH6\VWHPV

LVUHWLUHGIURP+XJKHV(OHFWURQLFV

:KHQ*UDQWZDVDVNHGWRFRQGXFWDVWXG\RQWKHVXFFHVVRI&276SXUFKDVLQJLQWKH

$LU)RUFHKHIRXQGWKDWUHTXHVWDPXVLQJ$OWKRXJKKDOIRIKLVFDUHHUKDGEHHQLQ

GHIHQVHV\VWHPVKHVSHQWWKHODWWHUSDUWRIKLVFDUHHULQSXUHO\FRPPHUFLDODFTXLVLWLRQ

*UDQWYLHZHGWKHSURSRVHG&276VWXG\DVGLIILFXOWWRDFFRPSOLVK$WWKHVDPHWLPHKH

FRQFOXGHGWKDWLIGRQHFRUUHFWO\WKHVWXG\ZRXOGKDYHSRWHQWLDOIRUDKXJHSD\RII

/LNHPDQ\SHRSOH*UDQWYLHZVWKH'HSDUWPHQWRI'HIHQVH
V'R'WUDQVLWLRQWR

&276EDVHGV\VWHPVDVODUJHO\LQHYLWDEOH7KHUHIRUHFDSWXULQJOHVVRQVOHDUQHGWKXVIDU

ZRXOGEHYHU\YDOXDEOH

Panel Members

• Mr. Jeff Grant, Chair Hughes Electronics (Ret.)

• Dr. Bob Rankine Hughes Electronics

• Mr. Ken Brown Raytheon Systems

• Mr. Roger Carter Lockheed Martin

• Mr. John Foreman SEI

• Mr. Don Reifer Consultant

• Dr. Will Tracz Lockheed Martin

• Dr. Nick Tredennick SAB (Army)

• Mr. Frank Willis DY 4

• Paul Schubert, Lt Col SAB Secretariat

• Rob Block, Capt USAFA (Tech Editor)
2

(;,67,1*,1)250$7,2121&276

:KHQWKHVWXG\JURXSVXUYH\HGH[LVWLQJGDWDWKHJURXSZDVVXUSULVHGDWWKHDPRXQWRI

LQIRUPDWLRQDYDLODEOHRQ&276

7KHUHDUHUHVHDUFKSURMHFWV7KHUHLVDPDJD]LQHGHYRWHGWR&2767KHUHDUHZKLWH

SDSHUV7KHUHDUH&276FRQIHUHQFHV
WZRRIWKHPLQIDFWRQHRQHDFKFRDVW

,QVKRUWLWLVDVWRXQGLQJKRZPXFKLQIRUPDWLRQLVRXWWKHUH

7KHUHIRUH*UDQW
VILUVWUHDFWLRQZDV:KDWLQDIHZVKRUWPRQWKVFDQDEXQFKRID

6$%$LU)RUFH6FLHQWLILF$GYLVRU\%RDUGSHRSOHFRQWULEXWHWRWKHXQGHUVWDQGLQJRI

&276FRPSDUHGWRDOOWKHVHRWKHUHIIRUWV"

4XLFNO\KRZHYHULWEHFDPHHYLGHQWWKDWYHU\OLWWOHHIIRUWKDGEHHQPDGHWRLQWHJUDWH

WKHFRSLRXVLQIRUPDWLRQRQ&276

6&23(2)7+(678'<

7KHVWXG\JURXSLQWHUYLHZHGSURJUDPVDQGRUJDQL]DWLRQV2IWKDWSURJUDPV

UHSUHVHQWHGWKHIRXUVHUYLFHV$UP\1DY\$LU)RUFHDQG0DULQH&RUSV

(UURU1RWDEOHRIFRQWHQWVHQWULHVIRXQG

)LJ2UJDQL]DWLRQVDQG3URJUDPV$VVHVVHGE\WKH&2766WXG\*URXS

• DCAC/MRM (Boeing

engr/mfg system)

• DY 4 (COTS supplier)

• Earth Sensor

• F117 engine electronics

• F-16 upgrade

• F-22

• GBS

• GPS (Ground Control

Segment)

• GPS Receiver

• Ground Station

• GTE Info Systems

Division

• JASPO (Sigint infrastr)

• JDAM*

• Large ADP system (TRW)

• MRP II**

• Mobility SPO

• NSSN C3I (LM Undersea

Systems)

• OSD Major Policy Initiatives

• OSD/C3I

• PVS/EVS (Boeing info sys)

• SAF/AQX

• T-38C Avionics Upgrade

Program

• T-6A Joint Primary Aircraft

Training System (JPATS)

• Tactech (parts mgt)

Interviews

• AAAV (GD Amphibious Sys)

• AFOTEC

• AFPEO/LI for Logistics Info

SPO at Gunter AFB

• AFRL COTS Initiatives

• AWACS (Computer

Modernization)*

• B-2 Data Storage

• B-2 EFX 99

• Boldstroke (Boeing

commonality initiative)

• Bradley Fighting Vehicle*

• Calce (University of

Maryland)

*COTSCON Conf Presentation

** Teleconference
3

5HYLHZRI&276DSSOLFDWLRQVZDVOLPLWHGWRHQJLQHHULQJGLVFLSOLQHVZKHUHFRPPHUFLDO

PDUNHWVDUHUHDOO\GULYLQJWKHWHFKQRORJ\7KHSURGXFWVRISULPDU\LQWHUHVWWR'R'DUH

HOHFWURQLFVFRPSXWHUVDQGVRIWZDUH

,QWKDWFRQWH[WWKHVWXG\JURXSDOVRUHYLHZHGPDQDJHPHQWLQIRUPDWLRQV\VWHPVVXFK

DV0DQXIDFWXULQJ5HVRXUFHV3ODQQLQJ0537KHJURXSDOVRUHYLHZHG&,DQG

ZHDSRQVV\VWHPV7KHJURXSFRQFOXGHGWKDW&276LVDSSOLFDEOHLQDOOWKUHHDUHDV

$IWHUGHWHUPLQLQJEDVLFDSSOLFDELOLW\WKHVWXG\JURXSORRNHGDWLPSOHPHQWDWLRQ

,PSOHPHQWDWLRQLQWHQVLW\YDULHG

)LJ7KH&2766WXG\3DUDPHWHUV

5(68/762)7+(678'<

4XDQWLILHG5HVXOWV

$OWKRXJKLWLVWHPSWLQJWRTXDQWLI\WKHUHVXOWVRIWKHVWXG\WKHVWXG\JURXSFRQFOXGHG

WKDWWKHVWDWLVWLFVGRQRWMXVWLI\TXDQWLILFDWLRQ

4XDOLWDWLYH5HVXOWV

4XDOLWDWLYHO\WKHVWXG\JURXSFRQFOXGHGWKDWWKHVXFFHVVRI&276LPSOHPHQWDWLRQLV

OLPLWHG0RUHRYHUWKHTXDOLW\RI&276LPSOHPHQWDWLRQUDQJHVIURPWKHVSHFWDFXODUWR

WKHGLVDVWURXVZLWKDZKROHORWLQEHWZHHQ)LQDOO\UHJDUGOHVVRIZKHUHRQWKDW

VSHFWUXPDSURJUDPIHOOLWZDVFOHDUWKDWHYHU\RQHZDVVWUXJJOLQJZLWKWKHQHZ

WHFKQRORJ\DQGWKHSURFHVVHV

Scope of Study

• Applications

− Management information systems

− C3I

− Weapon systems

• Engineering discipline

− Electronics

− Computer systems

− Software

• Implementation extent

− COTS solution systems (One substantial product)

− COTS aggregate systems (Multiple products from multiple

suppliers)

− COTS components (Develop with commercial components)
4

*2$/62)7+(678'<

2QHRIWKHJRDOVIRUWKHVWXG\EHFDPHWRLQWHJUDWHQRWRQO\VRPHRIWKHH[LVWLQJ

LQIRUPDWLRQRQ&276EXWDOVRWKHLQIRUPDWLRQGHYHORSHGE\WKHVWXG\JURXS7KHQRQ

WKDWIRXQGDWLRQWKHJURXSZRXOGGHYHORSUHFRPPHQGDWLRQVWRDFFHOHUDWHWKH

LPSOHPHQWDWLRQRI&276WKURXJKRXWWKHDFTXLVLWLRQFRPPXQLW\,QSDUWLFXODUWKH

JURXSGHWHUPLQHGWKDWWKHUHSRUWVKRXOGHGXFDWHSHRSOHRQWKHFRPSOH[LWLHVRI&276

DQGWKH&276GHFLVLRQPDNLQJSURFHVV

29(59,(:2)7+(678'<&21&/86,216

&276SXUFKDVLQJLVQRWDSDQDFHD0DQ\SHRSOHEHOLHYHLWLVVLPSOHVWUDLJKWIRUZDUG

DQGWKDW'R'RXJKWWRMXVWGRLW

2IWHQPLVVLQJIURPWKHGLVFXVVLRQRQ&276KRZHYHULVDQXQGHUVWDQGLQJRIDOOWKH

FRPSOH[LWLHVRIWKLVW\SHRIDFTXLVLWLRQ7KHUHIRUHRQHRIWKHRYHUDUFKLQJREMHFWLYHVRI

WKHUHSRUWLVWRHGXFDWHSHRSOHRQWKRVHFRPSOH[LWLHV

)LJ2YHUYLHZRIWKH&2766WXG\6FRSHDQG&RQFOXVLRQV

COTS-based solutions are inevitable

Summary

• Interviewed 34 programs or organizations

• Successful exploitation of COTS is limited

− Struggling with new technology, processes and

issues

− Huge learning curve for both Government and

Industry

− Hardware more successful than software

− Total Ownership Cost not considered or

understood

• Applicability of COTS requires careful consideration

• Well defined processes are essential

• Roles and responsibilities change dramatically

• Pervasive changes are needed to harness COTS

• Acquisition policy must address COTS-based systems
5



&276DQG1HZ7KLQNLQJ

1(:7+,1.,1*

,QWKHGHFLVLRQZKHWKHUWRXWLOL]H&276WKHLQLWLDOTXHVWLRQPXVWEH,VZKDWZHQHHG

DOUHDG\DYDLODEOHDQGLVWKHUHVRPHERG\GRLQJWKLVWDVNWKDWLVUHDOO\JRRGDWLW"

,IWKHDQVZHUWRWKDWTXHVWLRQLV\HVWKHVWXG\JURXSIRXQGWKDWWKHZKROHSURGXFW

IDPLO\PXVWEHYLHZHGGLIIHUHQWO\WKDQLQWKHSDVW7KLVQHZWKLQNLQJLVIRFXVHGRQ

EX\DQGLQWHJUDWHUDWKHUWKDQGHVLJQDQGGHYHORS

)LJ%X\DQG,QWHJUDWH&KDUDFWHULVWLFVRI&276

2/'7+,1.,1*

'R'
VDFTXLVLWLRQKLVWRU\KDVWUDLQHGWKHZRUNIRUFHWRWKLQNDERXWGHYHORSLQJD

SURGXFWQRWEX\LQJLWRIIWKHVKHOI

75$16,7,21)5202/'721(:7+,1.,1*

7KHVWXG\JURXSFRQFOXGHGWKDWGHFLVLRQPDNHUVPXVWEHDZDUHWKDWYLVYLV&276

UROHVDQGUHVSRQVLELOLWLHVDUHYHU\GLIIHUHQW)URPWKHSRLQWRIYLHZRIDSULPH

FRQWUDFWRUIRUH[DPSOH&276LVDVZLWFKIURPWKHGHVLJQDQGEXLOGPRGHWRWKHEX\

DQGLQWHJUDWHPRGH7KDWLVGUDPDWLFFKDQJH

1HZ7KLQNLQJ1HZ&XOWXUH1HZ2UJDQL]DWLRQV

&XOWXUDO&KDQJH7KHDYHUDJHDHURVSDFHHQJLQHHULVSUREDEO\VRPHZKHUH

EHWZHHQDQG\HDUVROGZLWKWR\HDUVRIH[SHULHQFHDOORILWLQDGHVLJQ

DQGEXLOGRUGHYHORSPRGH

What is COTS?

• Product sold, leased or licensed to the public

• Supplier is a commercial entity in the business of making a profit

• Integrators use the product without modification

• Supplier retains intellectual property rights

• Supplier provides product support and evolution

• Commercial market drives product evolution
6

:LWKWKHWUDQVLWLRQWR&276LQWHQVLYHV\VWHPV'R'LVDVNLQJWKHVHHQJLQHHUVWRVWRS

GHVLJQLQJVWRSEXLOGLQJDQGOHDUQKRZWREX\DQGLQWHJUDWH7KDWLVDGUDPDWLF

FXOWXUDODQGWKXVSHUVRQDOFKDQJH)UDQNO\SHRSOHDUHJRLQJWRILJKWLW0DQ\SHRSOH

ZLOOIHHOWKUHDWHQHG

)LJ)HDWXUHVRIWKH&XOWXUDO&KDQJH$VVRFLDWHGZLWK&276,PSOHPHQWDWLRQ

2UJDQL]DWLRQDO&KDQJH:LWKWKHWUDQVLWLRQWR&276LQWHQVLYHV\VWHPVWKH

SHUVRQZKREXLOGVEODFNER[HVIRUDQRUJDQL]DWLRQIRUH[DPSOHTXLFNO\UHDOL]HVWKRVH

ER[HVQRORQJHUDUHJRLQJWREHGHVLJQHGDQGEXLOWLQKRXVH5DWKHUPRVWRIWKRVH

ER[HVDUHJRLQJWREHERXJKWRIIWKHVKHOI

)URPDSHUVRQDOSRLQWRIYLHZLWGRHVQRWWDNHDORWRILPDJLQDWLRQWRH[WUDSRODWHD

ZRUVWFDVHVFHQDULRKHUH7KDWFDXVHVDORWRIIHDUERWKLQWKHJRYHUQPHQWDQGLQ

LQGXVWU\
DQGIHDURIWHQLVWKHELJJHVWHQHP\RIFKDQJH

9(+,&/()2575$16,7,21)5202/'721(:7+,1.,1*

7KH'HFLVLRQ3URFHVVIRU,PSOHPHQWLQJ&276

7RWUDQVLWLRQWR&276WKHVWXG\JURXSIRXQGWKDWSHUYDVLYHFKDQJHVDUHQHHGHG
LQ

KRZ'R'RSHUDWHVEXWDOVRLQKRZ'R'WKLQNVDERXWDFTXLVLWLRQ

7KHPRVWLPSRUWDQWFKDQJHWKHVWXG\JURXSUHFRPPHQGHGLVWKDWDFTXLVLWLRQSROLF\

QHHGVWRDGGUHVVWKHLPSOHPHQWDWLRQGHFLVLRQSURFHVVIRU&276EDVHGV\VWHPV7KDWLV

WRPRYHWRZDUGVXFFHVVIXOLPSOHPHQWDWLRQRI&276LQWHQVLYHV\VWHPV'R'PXVW

EHJLQHPSKDVL]LQJWKH&276WKRXJKWSURFHVV
ZKHQZK\DQGKRZWRLQWHJUDWH

&276
UDWKHUWKDQVLPSO\DWWHPSWWRPDQGDWHXVHRI&276

+LVWRULFDO&RQWH[WRIWKH&276'HFLVLRQ3URFHVV,Q&RQJUHVVSDVVHG

OHJLVODWLRQUHTXLULQJ'R'WRJLYHFRQVLGHUDWLRQWRQRQGHYHORSPHQWDOLWHPV1',

7KHQLQWKH)HGHUDO$FTXLVLWLRQV6WUHDPOLQLQJ$FW)$6$UHTXLUHG'R'WR

People Issues

• Unrealistic expectations by leaders

• Dramatically altered roles and responsibilities

• Lack of skills, knowledge, and abilities

− At all levels, both Government &

Industry

• Perceived loss of control and job security
7

JLYHSUHIHUHQFHWRFRPPHUFLDOLWHPVWKHQWR1',DQGRKE\WKHZD\WKHODZVDLG

GRQ
WGRDQ\WKLQJIURPDFRQWUDFWXDOSRLQWRIYLHZWKDWLPSHGHVWKHXVHRIFRPPHUFLDO

LWHPVDQGVRRQDQGVRIRUWK7KHUHZHUHSURYLVLRQVIRUFXVWRPHUVWRSHUIRUPPDUNHW

UHVHDUFKDQGRWKHUIDPLOLDUIXQFWLRQV

7KHUHVXOWLV&276JRYHUQPHQWRIIWKHVKHOIPRGLILHGRIIWKHVKHOIUHVHDUFKRIIWKH

VKHOI7KHUHDUH1',V7KHUHLVUHXVH$QGLQWKHPLGVWRIDOOWKHVHRIIWKHVKHOI

RSWLRQVWKHUHLVFXVWRPGHVLJQ
ZKLFKLVYHU\DSSURSULDWHLQPDQ\FDVHV

&OHDUO\WKHUHDUHFDVHVZKHUHDQRUJDQL]DWLRQFRXOGVSHQGPRUHWLPHDQGGROODUVWU\LQJ

WRLQWHJUDWHFRPPHUFLDOSURGXFWVLQWRDSURJUDPWKDQLIWKHSURGXFWVLPSO\KDGEHHQ

GHVLJQHGIURPVFUDWFK7KLVIDFWLVRQHRIWKHUHDVRQVLWLVVRLPSRUWDQWWKDWUDWKHUWKDQ

DXWRPDWLFDOO\ORRNLQJWR&276SURGXFWVGHWDLOHGWUDGHVWXGLHVEHFRPSOHWHGLQFOXGLQJ

DIRFXVRQ7RWDO2ZQHUVKLS&RVW72&

)LJ+LVWRULFDO&RQWH[WRIWKH&276,QLWLDWLYH

7,0(7$%/()2575$16,7,21)5202/'721(:7+,1.,1*

7KHSDFHRIWKHWUDQVLWLRQIURPROGGHVLJQDQGEXLOGWKLQNLQJWRQHZEX\DQG

LQWHJUDWHWKLQNLQJLVGHWHUPLQHGLQSDUWE\WKHQXPEHURIQHZSURJUDPVWDUWV

1HZ7KLQNLQJ0XVWEH$GRSWHGDW3URMHFW&RQFHSWLRQ

,QJHQHUDOWKHQHHGWRFKDQJHWKLQNLQJIURPWKHGHVLJQDQGEXLOGPRGHWRWKHEX\DQG

LQWHJUDWHPRGHPXVWWDNHSODFHDWWKHFRQFHSWLRQRIWKHSURMHFWQRWLQPLGVWUHDP

Legislation

• 1986 - Congress passed legislation requiring DoD to give preference to NDI

• 1994 - Federal Acquisition Streamlining Act (FASA)

− Contract requirements and market research should facilitate use of

commercial items

− Preference for NDI when commercial not available

− Eliminates contract requirements that impede acquisition of commercial items

− Requires agency to conduct market research prior to development of a new

specification
8

&DVH6WXG\)3URJUDP$QH[DPSOHRIKRZWKHVHWZRYHU\GLIIHUHQW

DFTXLVLWLRQSHUVSHFWLYHVFDQFROOLGHLVWKH)SURJUDP

7KH)RULJLQDOO\ZDVGHVLJQHGDVDW\SLFDOGHVLJQDQGEXLOGSURMHFW7KDWLVLWGLG

QRWLQFRUSRUDWH&276DWDOO6RPHWLPHLQWRWKHSURJUDP&RQJUHVVLPSRVHGDFRVW

FDS7KLVOHGWKHSURJUDPPDQDJHUVWRPDQDJHWKH)SURJUDPOLNHDIL[HGIHH

UDWKHUWKDQDFRVWSOXVFRQWUDFW7KDWLQWXUQOHGWRDQDWWHPSWWRLQFRUSRUDWH&276

,QFRUSRUDWLQJ&276LQWRDSURJUDPWKDWLVDOUHDG\GHVLJQHGLVH[WUHPHO\GLIILFXOW,Q

IDFWDFFRUGLQJWRWKHVWXG\&KDLULWPD\QRWEHWRRVWURQJWRVD\WKDWLWLVLPSRVVLEOH

$OOWKHOHYHUDJHVWKDWKDYHWREHEXLOWLQWRWKHDUFKLWHFWXUH
WKHRSHQV\VWHP

DUFKLWHFWXUHFRPPRQEXVVHVWKLQJVWKDWIDFLOLWDWHSOXJDQGSOD\LQWHJUDWLRQ
DOOWKHVH

WKLQJVDUHPLVVLQJLQDGHVLJQHGSURJUDP$QGLWLVWRRODWHWRJREDFNDQGEXLOGWKHVH

&276IULHQGO\SUHUHTXLVLWHVLQWRWKHGHVLJQRQFHDSURJUDPLVODXQFKHG

,1(9,7$%/,7<2)7+(75$16,7,2172&276%$6('6<67(06

'HVSLWHWKHSUROLIHUDWLRQRIRIIWKHVKHOIWHUPVWKHRYHUDUFKLQJPHVVDJHLVFOHDU

6WRSUHLQYHQWLQJWKHZKHHO/HYHUDJHFRPPHUFLDOSURGXFWVZKHQLWPDNHVVHQVHDQG

DFKLHYHVWKHEHVWYDOXH%HVWYDOXHPHDQVWKHORZHVW72&

6LPLODUO\LQWKHDFTXLVLWLRQFRPPXQLW\WRGD\DFRQVHQVXVLVIRUPLQJWKDW'R'KDVQR

FKRLFHEXWWRPDNHWKHWULS
WKHWULSIURPWKHROGWKLQNLQJRIGHVLJQDQGEXLOGWRWKH

QHZWKLQNLQJRIEX\DQGLQWHJUDWH

7KHWUDQVLWLRQIURPROGWRQHZWKLQNLQJREYLRXVO\LVIUDXJKWZLWKLVVXHVPDQ\RIWKHP

KXJH

,IKRZHYHUWKHWUDQVLWLRQWR&276EDVHGV\VWHPVLVLQHYLWDEOH
DQGWKHVWXG\JURXS

FRQFOXGHVWKDWLWLV
PDQ\SHRSOHIXUWKHUFRQFOXGHWKDWLWLVLPSHUDWLYHWRHQJDJHWKRVH

LVVXHVGHDOZLWKWKHPDQGJHWRQZLWKLW

,QVKRUWWZRFRXUVHVRIDFWLRQDUHSUHVHQWHG

• JHWRXWLQIURQWRIZKDWLVKDSSHQLQJWR'R'DQGLQIOXHQFHWKHWUDQVLWLRQRU

• UHDFWWRWKHFKDQJHDVLWLVLPSRVHGRQ'R'E\H[WHUQDOFLUFXPVWDQFHV
9

)LJ7KH&276,PSHUDWLYH

6800$5<2)7+(1(('7275$16,7,21)5202/'721(:

7+,1.,1*

7KHVWXG\JURXSIRXQGWKDWWKHDFTXLVLWLRQZRUOGKDVSODFHGWRRPXFKHPSKDVLVRQ

&2767KHUHRXJKWWREHPRUHHPSKDVLVRQWKHSURFHVVWRGHWHUPLQHWKHDSSOLFDELOLW\


WKHDSSURSULDWHQHVV
RI&276LQDSDUWLFXODUDFTXLVLWLRQVWUDWHJ\

$QRWKHUZD\RIVWDWLQJWKLVLQFUHDVHGHPSKDVLVRQWKH&276GHFLVLRQSURFHVVLVWKDW

WKHUHPXVWEHDWUDQVLWLRQIURPDGHVLJQDQGEXLOGPHQWDOLW\
ROGWKLQNLQJ
WRDEX\

DQGLQWHJUDWHPHQWDOLW\
QHZWKLQNLQJ

Leverage commercial

products

Achieve best value

Stop reinventing the wheel

The Message

• COTS

• GOTS

• MOTS

• ROTS

• NDI

• Reuse
10



7KH3UHVHQW&276(QYLURQPHQW

7+(25<YV5($/,7<

7KHVLPSOLFLW\RIWKH&276YLVLRQWHQGVWRREVFXUHWKHFRPSOH[LW\RIWKH&276UHDOLW\

0DQDJHPHQWLVYHU\IUXVWUDWHGULJKWQRZEHFDXVH&276RIWHQLVSUHVHQWHGDVYHU\

VWUDLJKWIRUZDUG7KHUHLVDOPRVWDMXVWJRGRLWPHQWDOLW\DPRQJWKHOHDGHUVKLS

:KLOHWKHUHLVJHQHUDODJUHHPHQWWKDW'R'RXJKWWREHLPSOHPHQWLQJ&276WKHGHYLO

LVLQWKHGHWDLO$QGWKLVLVDWHUULEOHGHYLO7KHUHIRUHZKLOHPRVWSHRSOHKDYHWKH

YLVLRQWKHUHDOLW\LVWKDWQRWKLQJLVKDSSHQLQJYHU\IDVWLQWKHUHDOZRUOG

1DLYHWH5HJDUGLQJWKH&RPSOH[LW\RI,PSOHPHQWLQJ&276

7KHVWXG\JURXSIRXQGWKDWXQIRUWXQDWHO\&276OHQGVLWVHOIWRDFHUWDLQDPRXQWRI

QDLYHWH
WKHLGHDWKDW,W
VMXVWDPDWWHURISLFNLQJWKHVHFRPPHUFLDOSURGXFWVDQGXVLQJ

WKHP7KHUHDOLW\LVWKDWLWLVPXFKPRUHFRPSOLFDWHGWKDQWKDW

7KHUHDUHWRROVEHLQJGHYHORSHGWRIDFLOLWDWH&276LPSOHPHQWDWLRQ%XWWKDW
VLQWKH

IXWXUH5LJKWQRZSHRSOHDUHVWUXJJOLQJ

2QHRIWKHELJJHVWIDFWRUVLQWKLVODFNRISURJUHVVLVDIDLOXUHXSDQGGRZQWKH

RUJDQL]DWLRQWRXQGHUVWDQGMXVWKRZFRPSOH[&276LPSOHPHQWDWLRQLVVXHVDUH8QWLO

PDQDJHPHQWIDFHVXSWRWKHFRPSOH[LWLHVRIWKHWUDQVLWLRQDQGVWDUWVWRRUJDQL]HWKDW

WUDQVLWLRQEDVHGRQUHDOLW\SURJUHVVLQLPSOHPHQWLQJ&276ZLOOFRQWLQXHWREHVORZ

7KH3RWHQWLDOO\+LJK&RVWRI3URGXFW,QWHJUDWLRQ

,QUHJDUGWR&276SHRSOHFRPPRQO\WKLQNWKDWWKHFRVWRIGHYHORSPHQWLVORZ%XW

SDUWRIGHYHORSPHQWLVLQWHJUDWLRQ6LQFHLQWHJUDWLRQRI&276SURGXFWVFDQEHYHU\

GLIILFXOWLQWKDWVHQVHWKHULVNRIGHYHORSPHQWFDQEHKLJK

$WOHDVWSDUWRIWKLVGHYHORSPHQWULVNLVDWWULEXWDEOHWRWKHIDFWWKDWWKHUHLVDKXJH

OHDUQLQJFXUYHIRUHYHU\RQHLQYROYHGLQLPSOHPHQWLQJ&276,WFDQQRWEH

RYHUHPSKDVL]HGKRZYDVWO\GLIIHUHQW&276LVIURPRWKHUW\SHVRIDFTXLVLWLRQ
11

&203/(;,7<2)&276,03/(0(17$7,21

7KHVWXG\JURXSIRXQGSURJUDPVIRUH[DPSOHWKDWDUHDWWHPSWLQJWRLQWHJUDWH

RU&276VRIWZDUHSDFNDJHV0RUHRYHUHDFKVRIWZDUHSDFNDJHLVRQDQPRQWK

DV\QFKURQRXVXSJUDGHF\FOH

7KLVVLWXDWLRQLVFRPSDUDEOHWRDSHUIRUPHURQVWDJHWU\LQJWREDODQFHSODWHVRQWKHHQG

RISROHV7KHSODWHVVWDUWVSLQQLQJRQRQHVLGHDQGKHUXQVRYHUKHUH7KHQWKH\VWDUW

VSLQQLQJRQWKHRWKHUVLGHDQGKHUXQVRYHUWKHUH

7KHVWXG\JURXSIRXQGWKDWLQWKH&276LPSOHPHQWDWLRQZRUOGLWRIWHQVHHPVWKDWWKH

SODWHVDUHDERXWWRVSLQRIIWKHSROHV3HRSOHMXVWJHWRQHSURGXFWZRUNLQJDQGWKHQ

WKH\
YHJRWWRUXQRYHUKHUHDQGLQFRUSRUDWHXSJUDGHVDQGWKHQUXQRYHUWKHUHDQG

ZRUNLQWHUIDFHSUREOHPVDQGWKHQUXQEDFNLQWKLVFRUQHUDQGZRUNLQWHURSHUDELOLW\

SUREOHPVDQGRQDQGRQDQGRQ,W
VUHDOO\GLIILFXOW

352),7$%,/,7<,668(6

3URILWV)ORZIURP'HIHQVHWR&RPPHUFLDO&RQWUDFWRUV

7KHWUDQVLWLRQWRD&276EDVHGV\VWHPUDLVHVSURILWDELOLW\LVVXHV,QWKHSUHVHQWV\VWHP

GHIHQVHFRQWUDFWRUVJDUQHUWKHPDMRULW\RISURILWVJHQHUDWHGE\'R'DFTXLVLWLRQ8QGHU

&276PRUHDQGPRUHSURILWVZLOOIORZDZD\IURPGHIHQVHFRQWUDFWRUVWRFRPPHUFLDO

FRQWUDFWRUV7KHFRVWRIWHFKQRORJ\FKDQJHDOVRZLOOFDXVHSURILWVWRIORZIURPGHIHQVH

WRFRPPHUFLDOFRQWUDFWRUV

7XUI3URWHFWLRQDQG3HUIRUPDQFH&ODLPV

&276KDYHWKHSRWHQWLDOWRZLSHRXWZKROHGHSDUWPHQWVVXFKDVPLOLWDU\SURGXFWV

RUJDQL]DWLRQVZLWKLQODUJHVHPLFRQGXFWRUFRPSDQLHV

,QWKHIDFHRIWKLVW\SHRIFKDQJHLWLVQDWXUDOIRUSHRSOHWRZDQWWRVDIHJXDUGWKHLU

GHSDUWPHQWVDQGWKRVHSURILWV,WLVMXVWDVQDWXUDOIRUSHRSOHZKRKDYHGHYRWHG

WKHPVHOYHVWRGHVLJQLQJDQGEXLOGLQJDSDUWLFXODUSURGXFWWRIHHOWKDWLWLVWKHEHVWWKDW

QRRWKHUSURGXFWFDQFRPSHWHZLWKPXFKOHVVVXUSDVVWKHLUSURGXFW

%X\HUVHYDOXDWLQJSURGXFWVVXFKDVWKHFRQWURYHUVLDOSODVWLFHQFDSVXODWHGPLFUR

HOHFWURQLFV3(0PXVWEHDZDUHWKDWWKHUHDUHDUJXPHQWVRQERWKVLGHVRIWKH

SHUIRUPDQFHGHEDWHWKDWFDQEHFRORUHGE\SHUVRQDODQGLQVWLWXWLRQDOELDV
12

,PSHUDWLYHWR/HYHUDJH&RPPHUFLDO'HYHORSPHQW

8VLQJGHIHQVHFRQWUDFWRUVDSHQWLXPFODVVSURFHVVRUIRUH[DPSOHFRVWVEHWZHHQ

0DQG0WRGHYHORS$QGWKDWGHYHORSPHQWFRVWLVJRLQJXSDWDQDQQXDOUDWH

RISHUFHQW

'R'FDQQRWDIIRUGWKHW\SHRIGHYHORSPHQWFRVWVFLWHGLQWKHSHQWLXPH[DPSOH7KH

DQVZHULVJRLQJWREHWKDW'R'PXVWOHYHUDJHFRPPHUFLDOGHYHORSPHQW,I'R'ZDQWV

WRILHOGDGYDQFHGV\VWHPV'R'PXVWWDNHDGYDQWDJHRIFRPPHUFLDOWHFKQRORJ\SHULRG

'R'VLPSO\FDQQRWDIIRUGWRGXSOLFDWHWKDWWHFKQRORJ\

6XSSOLHUV$GDSW

&RQWUDFWRUV
ZKHWKHUGHIHQVHRUFRPPHUFLDO
DUHLQEXVLQHVVWRPDNHDSURILW7KH

JRRGFRPSDQLHVPDNHLWWKHLUEXVLQHVVWRILJXUHRXWZKHUHDPDUNHWWUDQVLWLRQVXFKDV

&276LVJRLQJDQGKRZWKH\FDQRSHUDWHSURILWDEO\ZLWKLQWKDWQHZHQYLURQPHQW

'R'
VFKDOOHQJHLVWRJHWRXWIURQWJHWWKHDWWHQWLRQRIVXSSOLHUVDQGKHOSSXVK

FRPSDQLHVXSWKH&276OHDUQLQJFXUYHDVTXLFNO\DVSRVVLEOH

&+$1*(,1%8<,1*35$&7,&(6

:KHQLPSOHPHQWLQJ&276'R'
VWUDGLWLRQDOZD\RIEX\LQJSURGXFWVPD\FRQIOLFW

ZLWKFRPPHUFLDOSUDFWLFHV

'R'DFTXLVLWLRQPDQDJHUVDUHDFFXVWRPHGWREHLQJLQFKDUJHRIWKHEX\LQJSURFHVV

0DQDJHUVZDQWLQIRUPDWLRQRQDSURGXFWWKHGHVLJQHUJLYHVWKHPWKHLQIRUPDWLRQ

&RPPHUFLDOVXSSOLHUVDUHDFFXVWRPHGWREHLQJLQFKDUJHRIWKHEX\LQJSURFHVV<RX

ZDQWWKHLUSURGXFWWKH\VHOO\RXWKHSURGXFW7KDW
VLW

:KHQ+XJKHVIRUH[DPSOHILUVWEHJDQWREX\FRPPHUFLDOSDUWVIRUVDWHOOLWHV+XJKHV

ZHQWGLUHFWO\WRWKHFRPPHUFLDOVXSSOLHUV7KHVXSSOLHUVSXWXSDWHUULEOHILJKW7KH\

NHSWGLYHUWLQJ+XJKHVWRWKHLUPLOLWDU\SURGXFWVRUJDQL]DWLRQV

&RPPHUFLDOVXSSOLHUVGRQ
WZDQWWRSURYLGHWKHGDWD'R'LVDFFXVWRPHGWRUHFHLYLQJ

7KH\GRQ
WZDQWWRJRWKURXJKWKHKDUDQJXHVDQGQHJRWLDWLRQVWKDWW\SLI\WKHWUDGLWLRQDO

'R'EX\LQJSURFHVV7KHPLQGVHWRIWKHFRPPHUFLDOVXSSOLHULV<RXZDQWLW\RX

RUGHULWRXWRIWKHFDWDORJ\RXGRQ
WDVNTXHVWLRQV
13

&27668&&(66,17+(35(6(17(19,5210(17

/RQJHYLW\LQWKH6\VWHP

7KHJRRGQHZVLVWKDWHYHU\RQHLVOHDUQLQJKRZWRLPSOHPHQW&276EDVHGV\VWHPV

7KHEDGQHZVLVWKDWHYHU\RQHLVOHDUQLQJKRZWRLPSOHPHQW&276EDVHGV\VWHPV

7KHVWXG\JURXSREVHUYHGWKDWWKRVHZKRDUHEHVWDWIROORZLQJJRRG&276SUDFWLFHVDUH

WKRVHZKRKDYHEHHQLPSOHPHQWLQJ&276IRUWKHORQJHVWSHULRGRIWLPH

%\FRQWUDVWWKRVHZKRDUHLPSOHPHQWLQJ&276IRUWKHILUVWWLPHDOPRVWLQYDULDEO\IDLO

7KDWIDLOXUHZDVDVFULEHGSULPDULO\WRZKDWWKHVWXG\JURXSGHVFULEHGDVQDLYHWHDERXW

WKHFRPSOH[LWLHVRI&276V\VWHPV

5HTXLUHPHQWIRU,QVLGH$VVHVVPHQW2IWKHFRPSDQLHVWKHVWXG\JURXS

UHYLHZHGWKHLU&276DELOLWLHVYDULHGIURPYHU\JRRGWRYHU\SRRU%XWLWZRXOGQRWEH

DFFXUDWHWRVD\D/RFNKHHGRUD+XJKHVRUD5D\WKHRQLVJRRGRUQRWJRRGDW&276

7KHRQO\UHOLDEOHZD\WRGHWHUPLQH&276DELOLW\ZDVWRJRLQVLGHWKHFRPSDQ\JHW

LQVLGHWKHVSHFLILFRUJDQL]DWLRQDQGDVVHVVWKDWRUJDQL]DWLRQRUSURJUDPRQDFDVHE\

FDVHEDVLV

+DUGZDUHYV6RIWZDUH

&276VXFFHVVVWRULHVWHQGWREHPRUHDERXWKDUGZDUHWKDQVRIWZDUH+DUGZDUHKDV

PRUHVWDQGDUGVDVVRFLDWHGZLWKLW7KHUHLVOHVVYDULDELOLW\

7RWDO2ZQHUVKLS&RVW72&

6XFFHVVLQ&276LQWHQVLYHV\VWHPVGHSHQGVXSRQDIRFXVRQ72&DVRSSRVHGWR

DFTXLVLWLRQFRVWRUIO\DZD\FRVWRUWKHLQLWLDOUHFXUULQJFRVW2QHRIWKHUHDVRQVIRU

&276SURJUDPV
OLPLWHGVXFFHVVWRGDWHLVWKHIDFWWKDW72&LVUDUHO\FRQVLGHUHG72&

RIWHQLVQRWHYHQXQGHUVWRRG

'(&,6,21352&(66)25&27668&&(66

7KHGHFLVLRQWRXVH&276GHPDQGVFDUHIXOFRQVLGHUDWLRQ

7KHZKHQDQGKRZWRLQWHJUDWH&276DUHQRWGHFLVLRQVWKDWVKRXOGEHPDQGDWHGE\

VHQLRUOHYHOPDQDJHPHQW,WLVLPSRUWDQWWROHWWKHSHRSOHFORVHWRWKHKDUGZDUHDQG

VRIWZDUHPDNHWKHVHFULWLFDOGHFLVLRQV'HFLVLRQVVKRXOGEHEDVHGRQWUDGHVWXGLHVQRW

RQVRPHHGLFWIURPRQKLJK:LWKYHU\IHZH[FHSWLRQVEHFDXVHRIWKHFRPSOH[

G\QDPLFVRI&276LWLVQRWDSSURSULDWHWRPDQGDWH&276SHUFHQWDJHV
14

7RHQKDQFHGHFLVLRQPDNLQJRQZKHQDQGKRZWRXVH&276WKHVWXG\JURXSIRXQG

UHTXLUHPHQWVIRU

• ZHOOGHILQHGZHOOGRFXPHQWHGSURFHVVHV

• SHRSOHWKDWDUHWUDLQHGRQWKRVHSURFHVVHV

• SHRSOHWKDWIROORZWKRVHSURFHVVHVDQGH[HFXWHWKHPZHOO
15



&276DQG0RGLILFDWLRQ

29(59,(:2)&21&/86,2162102',),&$7,21

7KHVWXG\JURXSFRQFOXGHGWKDWWKHFDVHVWXGLHVLWUHYLHZHGGHPRQVWUDWHWKDW

PRGLILFDWLRQRI&276LVWKHPRVWJODULQJVRXUFHRIV\VWHPFRPSOLFDWLRQDQGRXWULJKW

IDLOXUH

3HRSOHZKREX\&276SURGXFWVDQGLQWHJUDWHWKHPLQWRODUJHUV\VWHPVJHQHUDOO\VKRXOG

QRWPRGLI\WKRVH&2767KLVKDVEHHQDPXFKUHSHDWHGPLVWDNH

,QGHHGWKHVWXG\JURXSREVHUYHGWKDWWKRVHZKRDWWHPSWHGWRPRGLI\&276SURGXFWV

ZHUHXQVXFFHVVIXOLQHYHU\VLQJOHLQVWDQFH

7KHERWWRPOLQHLVLIDQ\RQHLVSURSRVLQJWRPRGLI\&276IRUDJRYHUQPHQW

DSSOLFDWLRQUHGIODJVVKRXOGWRJRXS:RXOGWKHVWXG\JURXSVD\QHYHUPRGLI\D

&276"1REXWZKHQDPRGLILFDWLRQLVSURSRVHGHYHU\RQHQHHGVWRIXOO\XQGHUVWDQG

WKHLPSOLFDWLRQV

:+$7,602',),&$7,21"

0RGLILFDWLRQRI6RIWZDUH

0RGLILFDWLRQRIVRIWZDUHPHDQVPRGLI\LQJWKHVRXUFHFRGH,IWKHXVHUFDQSXW

ZUDSSHUVDURXQGWKHVRIWZDUHWRLQWHUIDFHLQWRWKHODUJHUV\VWHPWKDW
VILQHDVORQJDV

WKHEDVLFSURGXFWLWVHOILVQRWWRXFKHG

0RGLILFDWLRQRI+DUGZDUH

0RGLILFDWLRQRIKDUGZDUHPHDQVPRGLI\LQJWKHERDUGRUFRPSRQHQWVLQDQ\ZD\,WLV

DFFHSWDEOHIRUH[DPSOHWREX\DVLQJOHERDUGFRPSXWHUDQGSXWLWLQWRDKDUGHQHG

FDVHDJDLQDVORQJDVWKHEDVLFSURGXFW
WKHERDUGDQGFRPSRQHQWV
LVQRWWRXFKHG

0RGLILFDWLRQ*HQHUDO3ULQFLSOH

'LVWLOOLQJWKHJHQHUDOSULQFLSOHIURPWKHGHILQLWLRQRIPRGLILFDWLRQRIVRIWZDUHDQG

KDUGZDUHPRGLILFDWLRQPHDQVWKDWWKHFKDQJHLQDFRPPHUFLDOLWHPLVQRWPLQRU

UDWKHUWKHLWHPKDVEHHQFKDQJHGLQLWVLQKHUHQWDSSOLFDWLRQ

$QRWKHUZD\WRGHWHUPLQHLID&276KDVEHHQPRGLILHG
WKDWLVWRGHWHUPLQHLIWKH

SURGXFWFKDQJHVZHUHPLQRURUZHQWWRLWVLQKHUHQWDSSOLFDWLRQ
LVWRHYDOXDWHZKHWKHU
16

WKHFKDQJHVOLSSHGWKHSURGXFWRIILWVQDWXUDOHYROXWLRQDU\WUDFN7KHSURGXFW
VQDWXUDO

HYROXWLRQDU\WUDFWLVWKHWUDFNVHWIRULWE\WKHFRPPHUFLDOPDUNHWSODFH3URGXFW

HYROXWLRQLVGLVFXVVHGIXUWKHUEHORZ

02',),&$7,21$1'120(1&/$785(

0RGLILHG&276LV1RW&276

$JDLQWKHVWXG\JURXSIRXQGWKDWPRGLILFDWLRQVJHQHUDOO\MXVWGRQ
WZRUN0RUHRYHU

PRGLILFDWLRQVRIWHQGHIHDWWKHSXUSRVHRIEX\LQJDSURGXFWRIIWKHVKHOI

,QIDFWWKHVWXG\JURXSFRQFOXGHGWKDWLI&276DUHPRGLILHGWKRVHSURGXFWVVKRXOGQR

ORQJHUEHFDOOHG&276&DOOLWVRPHWKLQJHOVH
EHFDXVHLILW
VEHHQPRGLILHGLWLV

VRPHWKLQJHOVH

$FFRXQWLQJDQG4XDOLW\5HYLHZ

:KHQD&276KDVEHHQPRGLILHGLWVKRXOGQRWEHFRXQWHGDVD&2767KLVSULQFLSOH

LVSDUWLFXODUO\LPSRUWDQWIRUDFFRXQWLQJDQGTXDOLW\UHYLHZSXUSRVHV,IVRPHRQHLV

NHHSLQJVFRUHRQWKHSHUFHQWDJHRI&276LQDV\VWHPGRQ
WFRXQWPRGLILHG&276

0RGLILHG1RQGHYHORSPHQWDO,WHPV

,QVRPHDUHDV
1',VDQGPRGLILHG1',VIRUH[DPSOH
WKHUHPD\EHDJHQHUDOO\

DFFHSWHGWHUPIRUPRGLILHGSURGXFWV$VLPLODUWHUPPD\EHFRPLQJLQWRYRJXHIRU

&276WKDWKDYHEHHQPRGLILHG1HYHUWKHOHVVWKDWIDFWLVQRWUHOHYDQWWRWKHVWURQJ

FRQFOXVLRQRIWKHVWXG\JURXSUHJDUGLQJPRGLILFDWLRQRI&276$JDLQWKDWVWURQJ

UHFRPPHQGDWLRQLVWRDYRLGPRGLI\LQJ&276ZKHQHYHUSRVVLEOH

&$6(678',(621&27602',),&$7,21

6FRSHRIWKH&DVH6WXGLHV

7KHVWXG\JURXSLQWHUYLHZHGSURJUDPVDQGRUJDQL]DWLRQV2IWKRVHFDVHV

SURJUDPVKDGPRGLILHGDWOHDVWRQH&276SURGXFW

&RQFOXVLRQV%DVHGRQWKH&DVH6WXGLHV

,QDOOFDVHVZKHUH&276KDGEHHQPRGLILHGWKHSURJUDPUDQLQWRHQRUPRXV

GLIILFXOWLHVZLWK&276PRGLILFDWLRQV

,VFDVHVDKXJHVDPSOH"'RHVWKHVDPSOHLQGLFDWHWKDWWKHUHFRXOGQHYHUEHD

SURJUDPWKDWFRXOGKDYHDSRVLWLYHH[SHULHQFHZLWK&276PRGLILFDWLRQ"

1RRQERWKFRXQWV
17

7KHVWXG\JURXSFRPHVRQVWURQJDJDLQVWPRGLILFDWLRQKRZHYHUEHFDXVHWKHUH

FRQWLQXHVWREHDORWRIQDLYHWHRXWWKHUH
SHRSOHWKLQNLQJ:HOO,FDQPDNHWKLVRQH

OLWWOHFKDQJHDQGHYHU\WKLQJZLOOEHILQH

7KHSUREOHPDVLVZHOONQRZQLVWKDWRQHOLWWOHFKDQJHOHDGVWRDQRWKHUDQGSUHWW\

VRRQWKHSURMHFWLVLQDUHDOIL[7KDWUHVXOWLVVXSSRUWHGE\ZKDWWKHVWXG\REVHUYHG

$QGWKDW
VZK\WKHVWXG\JURXSFRQFOXGHGWKDWDWWKHPLQLPXPHYHU\RQHPXVWEHYHU\

FDUHIXOZKHQFRQVLGHULQJDSURSRVHGPRGLILFDWLRQ

,OOXVWUDWLYH)LQGLQJVLQWKH&DVH6WXGLHV

7KHVWXG\JURXSUHSHDWHGO\HPSKDVL]HGWKDWLWVUHYLHZRIWKHFDVHVWXGLHVOHDGWRD

VWURQJUHFRPPHQGDWLRQDJDLQVWPRGLILFDWLRQRI&276SURGXFWV7KHFRPSDULVRQRIWZR

SURJUDPVLOOXVWUDWHGWKHUHDVRQVIRUWKLVFDXWLRQ

)LJ'00,6DQG053,,&RPSDULVRQRI&276,QWHJUDWLRQ5HVXOWV

7KHILUVWSURJUDPWKH'HSRW0DLQWHQDQFH0DQDJHPHQW,QIRUPDWLRQ6\VWHP'00,6

GLGQRWIROORZJRRG&276SUDFWLFHV7KHVHFRQGSURJUDP0DQXIDFWXULQJ5HVRXUFHV

3ODQQLQJ053GLGIROORZJRRG&276SUDFWLFHV

The Story of Two Real Projects

No pre-selection

DMMIS MRP

Requirements Fixed Flexible

Selection demos “Try before you buy”

User Late involvement Early involvement

Source COTS product COTS product

Modification Allowed Not allowed

Acq Cost $200M+ $40M

1st Delivery Never Delivered 90 days

Result Project terminated Successfully Deployed

COTS
18

'00,6DQG053EDVLFDOO\DUHVLQJOHODUJHLQYHQWRU\PDQDJHPHQWV\VWHPVDOWKRXJK

WKH\GRDORWPRUHWKDQWKDW'00,6SUHFHGHG053

2YHUYLHZ'00,6VSHQWRYHU0DQGWKHSURJUDPHQGHGE\

FDQFHOODWLRQ)RU053WKHDFTXLVLWLRQFRVWZDVTXLWHDELWOHVVWKDQ0DQGWKH

SURJUDPVWLOOLVEHLQJLPSOHPHQWHG%XWWKHLQLWLDOVRIWZDUHLPSOHPHQWDWLRQVWDUWHGWKUHH

PRQWKVDIWHUJRDKHDG

0LVWDNHV0DGHE\'00,6$FFRUGLQJWRWKHVWXG\JURXSDUJXDEO\WKHPRVW

LPSRUWDQWPLVWDNHWKDWOHGWR&276LPSOHPHQWDWLRQIDLOXUHLQ'00,6ZDVWKDWWKH

SURJUDPKDGLQIOH[LEOHUHTXLUHPHQWV

7KHSURJUDPPDQDJHUVKDGSURFHVVHVLQSODFHWKDWWKH\ZHUHXQZLOOLQJWRFKDQJH

7KHUHIRUHUDWKHUWKDQFKDQJLQJWKHZD\WKHPDQDJHUVGLGEXVLQHVVWRDFFRPPRGDWHWKH

H[LVWLQJRIIWKHVKHOIVRIWZDUHWKHPDQDJHUVFKDQJHGWKH&276VRIWZDUHWRILWWKH

SURFHVVHV7RWKDWHQG'00,6PRGLILHGDERXWWZRPLOOLRQOLQHVRIFRGH

 &DXVHVRI'00,6&276,PSOHPHQWDWLRQ)DLOXUH

'00,6
VH[WHQVLYHPRGLILFDWLRQRI&276VRIWZDUHOHGWR

• ORVVRIWKHVRIWZDUH
VQDWXUDOLHPDUNHWGULYHQHYROXWLRQ

• ORVVRIFRQFRPLWDQWVXSSOLHUVXSSRUW

• YDVWO\FRPSOLFDWHGDQGLQWKHHQGDQLQDELOLW\WRPDQDJHV\VWHPLQWHJUDWLRQ

+8*+(6(/(&7521,&6
(;3(5,(1&(:,7+&27602',),&$7,21

,QDGGLWLRQWRWKHFDVHVWXGLHVVRPHPHPEHUVRIWKHVWXG\JURXSDOVRGUHZRQSHUVRQDO

H[SHULHQFHRQWKHLVVXHRI&276PRGLILFDWLRQ

,QWKHVWXG\&KDLU
VFRPSDQ\+XJKHV(OHFWURQLFVWKHUHZHUHQXPHURXVWLPHV+XJKHV

PRGLILHG&276VRIWZDUH5HVXOW",WZDVDGLVDVWHUHYHU\WLPH

+XJKHV(OHFWURQLFV5HSHDWHG'00,6
V0LVWDNHVLQ&276,PSOHPHQWDWLRQ

,QIOH[LEOH5HTXLUHPHQWV+XJKHVPRGLILHGD053SURJUDPWRILWVLWVH[LVWLQJ

SURFHVVHV2YHUWLPH+XJKHVPRGLILHGSHUFHQWRIWKHFRGH,WZDVDQXWWHUGLVDVWHU

7KHFRPSDQ\ZDVWHGDIRUWXQH1RRQHHYHQZDQWHGWRJREDFNDQGFRXQWKRZPXFK

ZDVVSHQW
LWZDVWKDWWHUULEOH
19

3URGXFW6XSSRUW$VVRRQDVWKHSURGXFWZDVPRGLILHG+XJKHVFRXOGIRUJHW

DERXWSURGXFWVXSSRUW7KHVXSSOLHUVLPSO\ZDVQRWJRLQJWRVXSSRUWWKHPRGLILHG

SURGXFW)LUVWRIDOOWKHVXSSOLHUGRHVQRWNQRZKRZWKHSURGXFWKDVEHHQPRGLILHG

(VVHQWLDOO\LW
VQRWWKHLUSURGXFWDQ\PRUH7KHUHIRUHWKRVHZKRPRGLI\&276DUHRQ

WKHLURZQ

6SHFLILF5HG)ODJV,GHQWLILHGLQ+XJKHV
([SHULHQFHZLWK&276

,PSOHPHQWDWLRQ

7KH&KDLULGHQWLILHGDQXPEHURIIDFWRUVWKDWFDQOHDGWRDWWHPSWVWRPRGLI\&276

7KHVHIDFWRUVVKRXOGVHUYHDVUHGIODJVIRUWKRVHLPSOHPHQWLQJ&276EDVHGV\VWHPV

7KHVHUHGIODJVLQFOXGH

• 7KHIDFWWKDWDGHPRZDVQRWUHYLHZHGVXIILFLHQWO\HDUO\LQWKHVHOHFWLRQSURFHVV

• $WHQGHQF\IRUSHRSOHWREHFRPHHQDPRUHGZLWKWKHFODLPVWKDWZHUHPDGHIRUWKH

VRIWZDUHDVRSSRVHGWRREWDLQLQJLQIRUPDWLRQRQWKHVRIWZDUH
VDFWXDORSHUDWLRQ

• )DLOXUHWRXQGHUVWDQGWKHVFRSHRIWKHVRIWZDUH
VSRWHQWLDODSSOLFDWLRQV

• )DLOXUHWRHYDOXDWHWKHVRIWZDUHDJDLQVWSRWHQWLDODSSOLFDWLRQV

• )DLOXUHWRLQYROYHWKHXVHUVXIILFLHQWO\HDUO\LQWKHVHOHFWLRQSURFHVV

6800$5<),1',1*6%$6('217+(&$6(678',(6

0RGLILFDWLRQVWKDWPDLQWDLQDSURGXFW
VFDSDFLW\WRHYROYHZLWKWKHPDUNHWSODFH
LH

PLQRUPRGLILFDWLRQV
KDYHIHZHULPSOLFDWLRQVIRUV\VWHPLQWHJUDWLRQ7KHUHIRUHWKHUH

PD\EHVLWXDWLRQVZKHUHWKRVHW\SHVRIPRGLILFDWLRQVDUHDFFHSWDEOH1HYHUWKHOHVVLWLV

LPSRUWDQWWRHPSKDVL]HWKDWWKHVWXG\JURXSIRXQGQRH[DPSOHVRIVXFFHVVIXO

PRGLILFDWLRQRI&276
20



3URGXFW(YROXWLRQ

(92/87,21$1'7+(%8<,1*352&(66

'R''RHV1RW&RQWURO&276(YROXWLRQ

7KHVXSSOLHU
DQGXOWLPDWHO\WKHPDUNHW
GHWHUPLQHVWKHHYROXWLRQRID&276

SURGXFW

)RUWKRVHZKRDUHDFFXVWRPHGWRZRUNLQJZLWKLQWKHPLOLWDU\V\VWHPWKHWKLQNLQJRI

WKHFRPPHUFLDOPDUNHWLVYHU\YHU\GLIIHUHQW,QPLOLWDU\V\VWHPVWKHPLOLWDU\DQGWKH

SULPHFRQWUDFWRUVGULYHSURGXFWHYROXWLRQ

$JDLQWKDWLVQRWWUXHLQUHJDUGWR&276,Q&276WKHSXUFKDVHULVWKHWDLORQWKH

GRJ7KDWIDFWFDQEHKDUGIRUVRPHSHRSOHWRUHFRJQL]H

7KH9DOXHRI%X\LQJ(YROXWLRQ

:KHQD&276SURGXFWLVSXUFKDVHGWKHSXUFKDVHULVEX\LQJWKHSURGXFWDQG

WKHHYROXWLRQRIWKDWSURGXFW7KLVLVWKHUHDVRQRQHDVSHFWRIWKH&276EX\LQJSURFHVV

PXVWLQFOXGHDQXQGHUVWDQGLQJRIZKHUHWKDWSURGXFWLVKHDGHG
WKDWLVLWVHYROXWLRQDU\

WUDFN

:KHQ'R'EX\VD&276SURGXFW
VHYROXWLRQ'R'EX\VLQWRWKHIDFWWKDWWKHEHVW

SUDFWLFHFRPSDQLHVOLWHUDOO\UHYDPSWKHLUUHVHDUFKDQGGHYHORSPHQW5	'

RUJDQL]DWLRQVRQDFRQWLQXDOEDVLV

7KHVHFRPSDQLHVGHYRWHDSRUWLRQRIWKHLUH[SHQGLWXUHVWRQRWKLQJEXWNHHSLQJWUDFNRI

ZKDW
VJRLQJRQLQWKHFRPPHUFLDOPDUNHWSODFHUHODWLYHWRWKHLUVSHFLILFLQWHUHVWV
LH

WKHLUSURGXFWV

7KHVHFRPSDQLHVFRQVWDQWO\DUHEULQJLQJLQSURGXFWVDQGHYDOXDWLQJWKHPWKRURXJKO\

EHIRUHWKH\PDNHFRQWUDFWXDOFRPPLWPHQWVWRFXVWRPHUV

7KLVLVQRWDSURFHVVWKDWDOORZVDSXUFKDVHUDWWKHODVWPLQXWHWRORRNLQDFDWDORJDQG

VD\:HOOWKDWSURGXFWORRNVOLNHLWZLOOGRWKHMRE(YHQVRWKHVWXG\JURXSIRXQG

QXPHURXVH[DPSOHVZKHUHWKDW
VH[DFWO\ZKDWRFFXUUHG
DQGGLVDVWHUXVXDOO\IROORZHG
21

 (YROXWLRQDQG3URGXFW6XSSRUW

3ODQQLQJIRU3URGXFW6XSSRUW3URGXFWHYROXWLRQJHQHUDOO\LQYROYHVSURGXFW

IHDWXUHVLPSURYHPHQWVQHZDSSOLFDWLRQVDQGVRRQ,WLVLPSRUWDQWKRZHYHUWR

DUWLFXODWHWKHLQWHJUDOUROHWKDWSURGXFWVXSSRUWSOD\VLQWKHHYROXWLRQRIDSDUWLFXODUHQG

LWHP

3URGXFWVXSSRUWLVGULYHQE\WKHVXSSOLHUDVZHOODVLQWKHODUJHUFRQWH[WWKH

FRPPHUFLDOPDUNHW2QHRIWKHDGYDQWDJHVRI&276SURGXFWVLVWKDWWKH\DOORZ

PDQDJHUVWRSODQRQVXSSOLHUVXSSRUW

3DFHRIWKH8SJUDGH&\FOH:KLOH&276DOORZPDQDJHUVWRSODQIRUVXSSOLHU

VXSSRUWZHDOOKDYHH[SHULHQFHGKRZVXSSOLHUVXSSRUWRUWKHODFNRILWFDQSXVK

SURGXFWHYROXWLRQHYHQLIWKHXVHUGRHVQ
WQHHGDPRUHKLJKO\HYROYHGSURGXFWDWWKDW

WLPH

7KHPRVWFRPPRQH[DPSOHRIWKLVSRLQWLVVRIWZDUH,IDSURJUDPGRHVQ
WPDLQWDLQ

UHODWLYHFXUUHQF\RQVRIWZDUHXSGDWHVWKHSURJUDPORVHVVXSSOLHUVXSSRUW7\SLFDOO\

VXSSOLHUVZLOOVXSSRUWRQHRUSHUKDSVWZRVXSHUFHGHGYHUVLRQVRIVRIWZDUHEXW

QRWKLQJEH\RQGWKDW

7KHHQGUHVXOWLVWKDWWKHXVHULVIRUFHGWRXSJUDGHWKHVRIWZDUHLUUHVSHFWLYHRIDQ\

WDVNUHODWHGQHHG$QGWKDWPRUHUDSLGXSJUDGHF\FOHFUHDWHVDQHHGIRUQHZ

SURFHGXUHVUHWUDLQLQJRISHUVRQQHODQGVRRQ%XWLIWKHSURJUDPHOHFWVQRWWR

XSJUDGHSUHWW\VRRQWKHSURJUDPLVRQLWVRZQ

,QHIIHFWWKHQWKHJRRGQHZVDERXW&276SURGXFWVLVWKDWWKH\EULQJZLWKWKHP

VXSSOLHUVXSSRUWDQGWKDWVXSSRUWHYROYHVZLWKWKHSURGXFW7KHRWKHUVLGHRIWKDWFRLQ

LVWKDWHYROXWLRQLQVXSSOLHUVXSSRUWPD\IRUFHDFXVWRPHUWRXSJUDGHDSURGXFWHYHQ

WKRXJKWKHFXVWRPHUGRHVQ
WKDYHWKHQHHGWRXSJUDGH

(92/87,21$1'02',),&$7,21

$GYDQWDJHLQ%X\LQJ3URGXFW(YROXWLRQ

2QHRIWKHDGYDQWDJHVRI&276LVWKDWLI\RXSLFNWKHULJKW&276SURGXFWVDQG&276

VWDQGDUGVWKHSURGXFWZLOOHYROYHRQDWUDFNEDVHGRQZKHUHWKHPDUNHWLVKHDGHG

,QIDFWEHVWSUDFWLFHFRPSDQLHVZLOOVLJQQRQGLVFORVXUHDJUHHPHQWV1'$VZLWK

&276VXSSOLHUV$VGLVFXVVHGWKLVIDFWDOORZVWKHVXSSOLHUDVZHOODVWKHSXUFKDVHUWR

XQGHUVWDQGIXOO\DQGSODQIRUZKHUHWKHPDQXIDFWXUHULVKHDGHGZLWKWKHSURGXFW7KLV

XQGHUVWDQGLQJLVFUXFLDOWRPDLQWDLQLQJPRGHUQL]DWLRQ
22

%\FRQWUDVWLIDSXUFKDVHUKDVDVXSSOLHUPRGLI\D&276SURGXFWIRUDSDUWLFXODU

DSSOLFDWLRQWKHUH
VDJUHDWFKDQFHWKDWWKDWDSSOLFDWLRQLVZLOOEHXQLTXH7KHSURGXFW
V

XQLTXHQHVVWDNHVLWRIIWKHHYROXWLRQDU\WUDFNLWRWKHUZLVHZRXOGKDYHEHHQRQ,QVKRUW

WKHEX\HUORVHVRXWRQWKHQDWXUDOHYROXWLRQRIWKDWSURGXFW

 (YDOXDWLQJ3URSRVHG0RGLILFDWLRQVLQ/LJKWRI3URGXFW(YROXWLRQ

(YDOXDWLQJKRZDSURSRVHGPRGLILFDWLRQZLOORUPLJKWDIIHFWWKHDELOLW\RIWKHSURGXFW

WRHYROYHDORQJWKHPDQXIDFWXUHU
VRUPDUNHWWUDFNLVYHU\LPSRUWDQW

8QGHUVWDQGLQJFKDQJHVLQDSURGXFW
VSRWHQWLDOHYROXWLRQDOVRLVRQHRIWKHNH\ZD\VWR

GHWHUPLQHLILQIDFWD&276KDVEHHQPRGLILHG7RUHSHDWLQWKLVFRQWH[W

PRGLILFDWLRQPHDQVWKDWWKHFKDQJHLQDFRPPHUFLDOLWHPLVQRWPLQRUUDWKHUWKH

LWHPKDVEHHQFKDQJHGLQLWVLQKHUHQWDSSOLFDWLRQ

*HQHUDOO\PRGLILFDWLRQVDUHPLQRULIWKH\GRQRWVOLSDSURGXFWRIIWKHHYROXWLRQDU\

WUDFNLWRWKHUZLVHZRXOGKDYHIROORZHGLQWKHPDUNHWSODFH

0RGLILFDWLRQVWKDWPDLQWDLQDSURGXFW
VFDSDFLW\WRHYROYHZLWKWKHPDUNHWSODFH
LH

PLQRUPRGLILFDWLRQV
KDYHIHZHULPSOLFDWLRQVIRUV\VWHPLQWHJUDWLRQ7KHUHIRUHWKHUH

PD\EHVLWXDWLRQVZKHUHWKRVHW\SHVRIPRGLILFDWLRQVDFFHSWDEOH1HYHUWKHOHVVLWPXVW

EHUHSHDWHGWKDWWKHVWXG\JURXSIRXQGQRH[DPSOHVRIVXFFHVVIXOPRGLILFDWLRQRI&276

 (YROXWLRQDQG,QWHJUDWLRQRI0RGLILHG3URGXFWV

:KHQWKHVWXG\JURXSDGGUHVVHGXQVXFFHVVIXOPRGLILFDWLRQVRQHRIWKHSULPDU\IDFWRUV

LQWKDWIDLOXUHLVWKHIDFWWKDWWKHSURGXFWQRORQJHUFDQEHHYROYHG7KLVODFNRIPDUNHW

GULYHQHYROXWLRQKDVP\ULDGLPSOLFDWLRQVIRUV\VWHPLQWHJUDWLRQ

2IFRXUVHPRGLILFDWLRQVERPELQRWKHUZD\VDVZHOO7KH\PD\IRUH[DPSOHEH

RXWULJKWIDLOXUHVWKH\PD\UXLQSURGXFWVWKH\PD\GHFUHDVHUDWKHUWKDQLQFUHDVHWKH

FDSDFLW\WRDFFRPSOLVKWKHWDVNWKH\PD\ZRUNEXWFDXVHDV\VWHPFUDVK

7KHLQDELOLW\WRWDNHDGYDQWDJHRIDSURGXFW
VHYROXWLRQDOVRFUHDWHVLVVXHVRI

FRQILJXUDWLRQPDQDJHPHQW7KDWLVNHHSLQJWUDFNRIZKDWKDVEHHQPRGLILHGDQGKRZ

WKRVHPRGLILFDWLRQVDIIHFW
RUPD\LQWKHIXWXUHDIIHFW
WKHV\VWHP1RQHRIWKDWLV

HDV\
23



7RWDO2ZQHUVKLS&RVW

727$/2:1(56+,3&26772&

72&LVDOLIHF\FOHFRVW72&LQFOXGHVWKHFRVWRIIXWXUHXSJUDGHVWHFKQRORJ\UHIUHVK

DQGLPSURYHPHQWV

72&$1'&276,03/(0(17$7,2168&&(66

6XFFHVVLQ&276LQWHQVLYHV\VWHPVGHSHQGVXSRQDIRFXVRQ72&DVRSSRVHGWR

DFTXLVLWLRQFRVWRUIO\DZD\FRVWRUWKHLQLWLDOUHFXUULQJFRVW

%DVHGRQWKHGHVLJQRIWKH&276V\VWHPLWLVLPSHUDWLYHWRJLYHSDUWLFXODU

FRQVLGHUDWLRQWRIXWXUHXSJUDGHVWHFKQRORJ\UHIUHVKDQGLPSURYHPHQWV,IWKHVHFRVWV

DUHQRWDGHTXDWHO\FRQVLGHUHGZKHQDVRIWZDUHXSJUDGHFRPHVDORQJIRUH[DPSOH

VXGGHQO\WKHSURGXFWLVJRLQJWRFRVWDORWPRUHWKDQSODQQHG

 3URJUDP:LGH)DLOXUHWR)RFXVRQ72&

7KHVWXG\JURXSIRXQGWKDWRQHRIWKHUHDVRQVIRU&276SURJUDPV
OLPLWHGVXFFHVVWR

GDWHLVWKHIDFWWKDW72&LVUDUHO\FRQVLGHUHG72&RIWHQLVQRWHYHQXQGHUVWRRG7KH

IDLOXUHWRFRQVLGHURUHYHQXQGHUVWDQG72&KDSSHQVDFURVVWKHERDUGDWWKHV\VWHPV

GHVLJQOHYHODQGWKHXVHUOHYHO

 )O\DZD\&RVW

)RUDYDULHW\RIUHDVRQVSXUFKDVHUVWHQGWRIRFXVRQWKHIO\DZD\FRVW%XWLILWLVD

&276LQWHQVLYHV\VWHPRSWLPL]LQJWKHLQLWLDOFRVWOLNHO\ZLOOOHDGWRGLVDVWURXV

VXEVHTXHQWPDLQWHQDQFHFRVWV

',6,1&(17,9(672)2&862172&

7KHUHLVVRPHVHQWLPHQWWKDWLQWKHSUHVHQWDFTXLVLWLRQHQYLURQPHQWQHLWKHUWKHVHOOHU

QRUWKHEX\HUKDVDQLQFHQWLYHWRXVHDOLIHF\FOHFRVWVXFKDV72&

 &RQJUHVVLRQDO%XGJHW3URFHVV

3UHVHQWO\'R'PXVWGHDOZLWKKRZWRLQFRUSRUDWH72&EDVHGWKLQNLQJLQWRKRZLW

KLVWRULFDOO\KDVFRQGXFWHGEXVLQHVVLQFOXGLQJLWVLQWHUDFWLRQZLWK&RQJUHVV
24

&RQJUHVVLRQDO2YHUVLJKW&RQJUHVV
UROHLQVKDSLQJWKH'R'EXGJHW

SURIRXQGO\DIIHFWV
DQGZLOOFRQWLQXHWRDIIHFW
'R'
VDELOLW\WRWUDQVLWLRQWRDIRFXV

RQ72&

&OHDUO\WKHUHDUHVWLOOPRUHROGV\VWHPVLQH[LVWHQFHWKDQQHZV\VWHPV1HYHUWKHOHVV

IRUWKHQHZHUSURJUDPV
DQGDJDLQWKDW
VRQO\DKDQGIXO
WKHPLQGVHWUHDOO\LV

FKDQJLQJLQIDYRURI72&7KHQHZPLQGVHWDIRFXVRQ72&LVQRWSUHYDOHQW

KRZHYHUDQGLWZRQ
WEHSUHYDOHQWVRORQJDVWKHUHDUHVRIHZQHZVWDUWV

$ELJSDUWRIWKHOHDUQLQJFXUYHIRU'R'LVOHDUQLQJKRZWRGULYH72&LQWRWKHH[LVWLQJ

V\VWHPWKHV\VWHPWKDWLVDEVRUELQJPRVWRIWRGD\
V'R'EXGJHW$VSURJUDPVDUH

PRGLILHGDQGXSGDWHGWKHVWXG\JURXSFRQFOXGHGWKDW'R'PXVWVWDUWLQFUHPHQWDOO\

FKDQJLQJWKHZD\LWWKLQNVDERXWWKRVHSURJUDPV,WLVDKXJHFKDOOHQJH

2QJRLQJ)XQGLQJ5HTXLUHPHQWV%HVLGHVWKHLVVXHRIKRZ&RQJUHVVVKDSHVWKH

'R'EXGJHW'R'PXVWGHDOZLWKLWVRZQKLVWRULFDOEXGJHWDU\DQGIXQGLQJSUDFWLFHV

7KRVHSUDFWLFHVFDQQRWEHFKDQJHGRYHUQLJKWIXQGLQJPXVWEHVHFXUHGQRZ%XW

LQWHUQDOO\SURJUDPVFDQVWLOOSXVKIRUDQGZRUNWRZDUGLPSOHPHQWLQJ72&FRQFHSWV

72&$1'6<67(06'(6,*1

,WLVFUXFLDOWKDW72&LVVXHVEHDGGUHVVHGLQ&276EDVHGV\VWHPV)RURQHWKLQJ72&

LQIOXHQFHVWKHV\VWHPGHVLJQ

2SHQ$UFKLWHFWXUH

2SHQV\VWHPDUFKLWHFWXUHVRPHWLPHVLVFDOOHGSOXJDQGSOD\7KLVDUFKLWHFWXUHLVYHU\

LPSRUWDQWZKHQKDQGOLQJVXEVHTXHQWXSJUDGHVDQGWHFKQRORJ\UHIUHVKLQD&276EDVHG

V\VWHP

,IDV\VWHPKDVDXQLTXHDUFKLWHFWXUHLWLVYHU\GLIILFXOWIRUH[DPSOHWRSOXJLQD

PRGHUQFRPSXWHUZLWKRXWVLJQLILFDQWDGGLWLRQDOFRVW7KHUHIRUHWKDWHYHQWXDOLW\QHHGV

WREHDGGUHVVHGXSIURQWQRWGRZQVWUHDP

&$6(678'<72&$1'1$9<$77$&.68%0$5,1(

7KHVWXG\JURXSUHYLHZHGDERXWWKUHH\HDUV
ZRUWKRIILHOGGDWDRQ&276

LPSOHPHQWDWLRQIURPWKH1DY\
VQHZDWWDFNVXEPDULQH7KHJURXSFRPSDUHGWKLVGDWD

ZLWKWKHHDUOLHUGHVLJQHGSURJUDP
25

&276&RVW(IIHFWLYHQHVV

7RWDOFRVWHIIHFWLYHQHVV7KHWRWDOFRVWHIIHFWLYHQHVVRIWKHHDUOLHUSURJUDPYV

WKH&276XSJUDGHSURJUDPLVGHEDWDEOH7KHGDWDGRHVQRWGHILQLWLYHO\DQVZHUWKH

TXHVWLRQRIZKLFKRQHZDVPRUHFRVWHIIHFWLYH

2WKHU&RVW6DYLQJV2QRWKHUUHOHYDQWVFDOHVWKHVWXG\JURXSIRXQGWKDWWKH

FRVWRIWKH&276XSJUDGHFRPSDUHGWRWKHHDUOLHUSURJUDPZDV

• IRUGHYHORSPHQWSHUFHQWOHVV

• IRUDFTXLVLWLRQUHFXUULQJFRVWSHUFHQWOHVV

• IRUVXSSRUWSHUFHQWOHVV

7LPH6DYLQJV,QWKH&276XSJUDGHFRPSDUHGWRWKHHDUOLHUSURJUDPWKH

GHYHORSPHQWWLPHZDVUHGXFHGE\SHUFHQW

3HUIRUPDQFH,PSURYHPHQWV7KHVWXG\JURXSIRXQGWKDWWKHSHUIRUPDQFHRI

WKH&276VXEPDULQHXSJUDGHFRPSDUHGWRWKHHDUOLHUVXEZDV

• SRZHUXVDJHUHGXFHGE\SHUFHQW

• WKURXJKSXWRIWKHFRPSXWHUUHGXFHGE\SHUFHQW

• PRVWVWULNLQJO\WKHPHDQWLPHEHWZHHQFULWLFDOIDLOXUHVLPSURYHG

'DWD/LPLWDWLRQV2QHOLPLWDWLRQRIWKLVGDWDKRZHYHULVWKDWWKHFRPSDULVRQ

ZDVQRWGRQHRYHUDQH[WHQGHGOLIHF\FOHVXFKDV\HDUV,WFRPSDUHGFRVWVRQO\

LQWKHVKRUWHUFRPPHUFLDOOLIHF\FOHRIWKHSURGXFW
26



&276DQG3URJUDP&ULWHULD

29(59,(:

,QWKH'R'DFTXLVLWLRQHQYLURQPHQWROGWKLQNLQJLVWKDWZKLFKIRFXVHVRQGHVLJQDQG

EXLOG1HZWKLQNLQJLVWKDWZKLFKIRFXVHVRQEX\DQGLQWHJUDWH

352*5$0&5,7(5,$$1'352'8&7,17(*5$7,21

2QHRIWKHSRLQWVDWZKLFKROGWKLQNLQJRIWHQFRQIOLFWVZLWK&276LQWHQVLYHV\VWHPVLV

LQWKHDUHDRISURJUDPFULWHULD&ULWHULDFRQWLQXHWRUHIOHFWWKHG\QDPLFVRIGHVLJQDQG

EXLOGUDWKHUWKDQEX\DQGLQWHJUDWHDFTXLVLWLRQPRGHV

352*5$0&5,7(5,$$1'6(,&(57,),&$7,21

6RIWZDUH(QJLQHHULQJ,QVWLWXWH6(,&HUWLILFDWLRQ

,QWKHROGGHVLJQDQGEXLOGWKLQNLQJSROLFLHVIRUH[DPSOHPD\UHTXLUHWKDWRQDQ

$FTXLVLWLRQ&DWHJRU\,$&$7SURJUDPDVRIWZDUHGHYHORSHUPXVWKDYHDWWDLQHG

OHYHOWKUHHRUKLJKHURQWKH6RIWZDUH(QJLQHHULQJ,QVWLWXWH6(,VFDOHRUHTXLYDOHQW

7KLVW\SHRIFULWHULDVLPSO\ZLOOQRWEHPHWLQWKHFRPPHUFLDOZRUOG0DQ\FRPPHUFLDO

VXSSOLHUVKDYHQRWHYHQJRQHWKURXJKWKHFHUWLILFDWLRQSURFHVV7KLVIDFWWKHQ

• FUHDWHVFRQIOLFWEHWZHHQWKHVWDQGDUGVLPSRVHGRQGHYHORSHUVZKRDUHXQGHU'R'

FRQWURODQGVWDQGDUGVDSSOLFDEOHWRFRPPHUFLDOGHYHORSHUV

• HOLPLQDWHVVXSSOLHUVZKRRWKHUZLVHZRXOGKDYHEHHQTXDOLILHGELGGHUVLQD&276

V\VWHP

6(,&HUWLILFDWLRQDQG5LVN0RGLILFDWLRQ:KLOHWKHUHLVVRPHFRUUHODWLRQ

EHWZHHQ6(,FHUWLILFDWLRQOHYHOVDQGTXDOLW\DWEHVWFHUWLILFDWLRQLVDULVNPRGLILHU,WLV

QRWDJXDUDQWHH

%\FRQWUDVW&276VRIWZDUHSUHVXPDEO\LVRXWWKHUHLQTXDQWLW\,WLVEHLQJXVHGE\

PDQ\GLIIHUHQWW\SHVRIXVHUVSRWHQWLDOO\LQPDQ\GLIIHUHQWW\SHVRIDSSOLFDWLRQVDQG

HQYLURQPHQWV7KLVZLGHVSUHDGXVHDQGWKHVXSSOLHU
VGHVLUHWREXLOGDPDUNHWDQG

PDNHDSURILWFDQIXQFWLRQDVEHWWHUTXDOLW\FRQWUROWKDQPDQGDWHGFHUWLILFDWLRQOHYHOV

7KHUHDOULVNUHGXFHUKRZHYHULVIRUEX\HUVWREHFRPHPXFKPRUHNQRZOHGJHDEOH

DERXWZKDWWKH\DUHEX\LQJWKDQWKH\KDYHEHHQLQWKHSDVW
27

6(,&HUWLILFDWLRQ6FRSH2QHRIWKHSUREOHPVLQKHUHQWLQUHO\LQJRQ6(,

FHUWLILFDWLRQLVWKDWLWLVWKHRUJDQL]DWLRQDVDZKROHQRWSDUWLFXODULQGLYLGXDOVRUJURXSV

RILQGLYLGXDOVWKDWLVFHUWLILHG

&HUWLI\LQJWKHRUJDQL]DWLRQGRHVQRWQHFHVVDULO\PHDQWKDWWKHSHRSOHWKHRUJDQL]DWLRQ

SXWVWRZRUNRQDSDUWLFXODUSURJUDPZLOOEULQJWRLWWKHOHYHORITXDOLW\VXSSRVHGO\

JXDUDQWHHGE\WKHFRUSRUDWHVHDO

1HYHUWKHOHVVWKHXWLOLW\RIWKH6(,FHUWLILFDWLRQSURFHVVLVWKDWLWSURYLGHVDPHDQVIRU

UHGXFLQJULVNQRWJXDUDQWHHLQJVXFFHVV7KHKLJKHUWKH6(,FHUWLILFDWLRQOHYHOLQ

JHQHUDOWKHORZHUWKHULVN

• 2QHRIWKHPDMRUVRXUFHVRIULVNDWOHYHORQHLVWKHKHURSKHQRPHQRQ7KHVKRS

FRQVLVWVRIRQHKHURZKRZULWHVWKHVRIWZDUH:KHQWKHKHURJRHVVRGRHVWKH

SURJUDP
ULJKWGRZQWKHWXEHV

• %\FRQWUDVWDWWKHKLJKHUOHYHOVWKUHHDQGXSWKHSURJUDPLVQRWGHSHQGHQWRQD

VLQJOHKHUR5DWKHULWLVVXSSRUWHGE\LQVWLWXWLRQDOL]HGWUDLQLQJE\DNQRZOHGJH

EDVHWKDWLVGLVEXUVHGDPRQJPDQ\SHRSOHLQWKHRUJDQL]DWLRQ7KLVUHGXFHVULVN

6(,&HUWLILFDWLRQDQG&2766XFFHVV

2IWHQWKHUHH[LVWVWKHH[SHFWDWLRQWKDWORZHUOHYHOFRQWUDFWRUVZLOOQRWKDYHWKHVNLOOWR

EHVXFFHVVIXOZLWK&2767KHVWXG\JURXSKRZHYHUGLGQRWILQGDQ\VWULFWFRUUHODWLRQ

EHWZHHQ6(,FHUWLILFDWLRQOHYHO
RUFHUWLILFDWLRQDWDOOIRUWKDWPDWWHU
RIWKHGHYHORSHU

DQGWKHTXDOLW\RIWKHVRIWZDUHSURGXFW

7RWKHFRQWUDU\WKHVWXG\JURXSIRXQGWKDWZRUNLQJZLWKDOHYHOILYHFRQWUDFWRUGRHV

QRWDVVXUHDEHWWHUVRIWZDUHSURGXFWWKDQRQHWKDWFRPHVIURPIRUH[DPSOHDOHYHO

WKUHHFRQWUDFWRURUHYHQDVKRSWKDWLVQRWFHUWLILHGDWDOO

7RLOOXVWUDWHWKDWSRLQWVRPHSHRSOHVD\WKDWYLGHRJDPHVRIWZDUHGHYHORSHUVDUH

FUHDWLQJSDFNDJHVWKDWDUHDVFRPSOLFDWHGDVDQ\WKLQJLQWKHILHOGRIZHDSRQVV\VWHPV

\HWLWLVXQOLNHO\WKDWDQ\RIWKRVHGHYHORSHUVDUHFHUWLILHGHYHQDWOHYHORQH

5HJDUGOHVVRIFHUWLILFDWLRQOHYHOLQVXFFHVVIXOO\LPSOHPHQWLQJ&276WKHFKDOOHQJHLV

WRXQGHUVWDQGZKDWLVEHLQJERXJKWDQGKRZWKDWSURGXFWZLOOEHWDNHQFDUHRI
28

(19,5210(17$/&5,7(5,$

2YHU6SHFLILFDWLRQRI(QYLURQPHQWDO&ULWHULD

%HFDXVHLWLVGLIILFXOWWROLPLWWKHYDULRXVHQYLURQPHQWVLQZKLFKDSURGXFWPLJKWEH

GHSOR\HGWUDGLWLRQDOO\'R'KDVWHQGHGWRRYHUVSHFLI\HQYLURQPHQWDOSDUDPHWHUV

1RWRQO\DUHSURGXFWVGHVLJQHGIRUYDULRXVSRVVLEOHHQYLURQPHQWVGHVLJQHGSURGXFWV

DUHWHVWHGLQYDULRXVHQYLURQPHQWV)HZFRPPHUFLDOSURGXFWVDUHVXEMHFWHGWRWKHW\SH

DQGH[WUHPHVRIHQYLURQPHQWDOWHVWLQJKLVWRULFDOO\UHTXLUHGE\'R'

%DODQFLQJ(QYLURQPHQWDO&ULWHULD,QJHQHUDOWKHPRUHH[WUHPHWKH

HQYLURQPHQWDOFRQGLWLRQVWKHOHVVRSSRUWXQLW\H[LVWVWRXVH&2767KLVPHDQVWKDWWZR

IDFWVDUHLQFRXQWHUEDODQFH

• 'R'FDQQRWFKRRVHRUOLPLWD&276SURGXFW
VLQWHQGHGHQYLURQPHQWEXW

• 'R'FDQQRWDWOHDVWIRUORQJDYRLGXVLQJ&276SURGXFWV

5HVSRQVH:KLOH'R'ORRNVIRUDUHVROXWLRQRIWKLVGLOHPPDWKHLPPHGLDWH

UHVSRQVHPXVWVWDUWZLWKDZDUHQHVVRIWKHVLJQLILFDQWHQYLURQPHQWDOLVVXHV
SURDQG

FRQ
WKDWPD\EHDVVRFLDWHGZLWK&276

 &DVH6WXG\3ODVWLF(QFDSVXODWHG0LFUR(OHFWURQLFV3(0

%DFNJURXQG 2QHRIWKHFRQWURYHUVLHVUHODWHGGLUHFWO\WR&276SHUIRUPDQFH

UDWKHUWKDQLPSOHPHQWDWLRQRULQWHJUDWLRQLQYROYHVSODVWLFHQFDSVXODWHGPLFUR

HOHFWURQLFV3(0

0LFURHOHFWURQLFVDUHXVHGWKURXJKRXWPLOLWDU\V\VWHPV7UDGLWLRQDOO\PLFUR

HOHFWURQLFVZHUHFRQVWUXFWHGZLWKKHUPHWLFDOO\VHDOHGFHUDPLFSDFNDJHV7KHVH

SDFNDJHVSURYLGHH[FHOOHQWLQVXODWLRQIRUHOHFWURQLFV

7KHFRPPHUFLDOPDUNHWSODFHKRZHYHUZRXOGQRWGDUHXVHDFHUDPLFSDFNDJHIRU

HOHFWURQLFV$FHUDPLFSDFNDJHFRVWVWHQWLPHVPRUHWKDQWKHHOHFWURQLFFRPSRQHQWWKDW

LVLQVLGH

&RPPHUFLDOPDQXIDFWXUHUVHQFDSVXODWHPLFURHOHFWURQLFVLQSODVWLF
KHQFH3(07KH\

XVHELJDXWRPDWHGLQMHFWLRQPROGLQJPDFKLQHVWKDWEDQJRXW3(0E\WKH]LOOLRQV

(QYLURQPHQWDQG+HUPHWLFLW\7KHFRQWURYHUV\RYHUWKHDGYLVDELOLW\RIXVLQJ

3(0LQPLOLWDU\V\VWHPVFHQWHUVRQTXHVWLRQVUHJDUGLQJ3(0KHUPHWLFLW\:LOOWKH
29

SODVWLFFDSVXOHNHHSRXWPRLVWXUH":LOOLWZLWKVWDQGSUHVVXUHFKDQJHV":LOOLWNHHSLWV

LQWHJULW\XQGHUWHPSHUDWXUHIOXFWXDWLRQVWKDWQHYHUHYHUHQWHUHGLQWRWKHPDQXIDFWXUHU
V

FDOFXODWLRQV"

6XUYLYDELOLW\)XQGDPHQWDOO\WKH3(0GHEDWHLVDERXWVXUYLYDELOLW\7KLV

GHEDWHKLJKOLJKWVRQHWKLQJWKDWKDVQRWEHHQGLVFXVVHGVRIDUDQGWKDWLVWKHPDMRU

GLVWLQFWLRQEHWZHHQZKDW'R'QHHGVSURGXFWVWRDFFRPSOLVKDQGZKDWWKHPDUNHWSODFH

QHHGVWKHPWRDFFRPSOLVK

$FTXLVLWLRQPDQDJHUVDUHDVNLQJSHRSOHWRGHSHQGRQWKHVHSURGXFWVDVWKH\JRLQWR

KDUP
VZD\)DUIURPWKHGDLO\JULQGRXUKLJKHVWFDOOLQJLVWRPDLQWDLQWKHWUXVWWKDW

RXUZDUILJKWHUVKDYHLQWKHLUHTXLSPHQW7KH\PXVWEHOLHYHWKDWHYHU\WKLQJDERXWWKHLU

HTXLSPHQWLVJRLQJWRSHUIRUPWKHZD\ZHVDLGLWZRXOGZKHQZHVDLGLWZRXOGXQGHU

WKHFRQGLWLRQVZHVDLGLWZRXOG

9HULI\LQJ3(06IRU,QWHQGHG(QYLURQPHQW$VZLWKGHVLJQHGSURGXFWV

XQGHUVWDQGLQJWKHLQWHQGHGHQYLURQPHQWLVFULWLFDOWRDSSURSULDWHXVHRI&276$OVRDV

ZLWKGHVLJQHGSURGXFWVWKHLQWHQGHGHQYLURQPHQWDFWXDOO\LVDZLGHVSHFWUXPRI

GLYHUVHHQYLURQPHQWVHDFKZLWKWKHLURZQVXEVHWRIH[WUHPHV0LOLWDU\XQLWVWRGD\

FRXOGEHLQWKH$UFWLFRQHPRQWKLQDGHVHUWWKHQH[WPRQWKDQGLQDMXQJOHWKH

IROORZLQJPRQWK

• 7HVWLQJDQG6WDII5HVRXUFHV2QHZD\WRUHVROYHRUDWOHDVWDGGUHVVWKH3(0

FRQWURYHUV\LVWRHVWDEOLVKDWKRURXJKWHVWLQJSURJUDP.QRZOHGJHDEOHVHPL

FRQGXFWRUSHRSOHPXVWYHULI\WKDW3(0ZLOOZRUNLQWKHLQWHQGHGHQYLURQPHQW

• (UDVLQJ&RVW%HQHILWV(VWDEOLVKPHQWRIDWHVWLQJSURJUDPKRZHYHUEULQJVXVEDFN

WRWKHLVVXHRIFRVW,WLVWKHWUHPHQGRXVFRVWEHQHILWWKDWLVGULYLQJWKHSURSRVDOWR

XVHRI3(0LQWKHILUVWSODFH7HVWLQJLVYHU\GLIILFXOWVXEWOHDQGH[SHQVLYH

0RUHRYHUDQ\SURJUDPWKDW
VHYHQFRQWHPSODWLQJXVLQJ3(0SUREDEO\RXJKWWR

KDYHDIHZVHPLFRQGXFWRUH[SHUWVRQVWDII6RWHVWLQJDQGVWDIILQJUHTXLUHPHQWVFDQ

HQGXSERRVWLQJWKHFRVWRIXVLQJ3(0WKXVZLSLQJRXWWKHLQLWLDOFRVWEHQHILW

&DVH6WXG\$:$&6

2YHUYLHZ7RDFFRPPRGDWHFRPPHUFLDOFRPSXWHUV$LUERUQH:DUQLQJDQG

&RQWURO6\VWHPV$:$&6WRRNDKDUGORRNDWWKHLUHQYLURQPHQWDOUHTXLUHPHQWV8SRQ

UHYLHZ$:$&6VXEVWDQWLDOO\PRGLILHGWKHLUUHTXLUHPHQWV
30

(QYLURQPHQWDO&ULWHULD0RGLILHG

• 7KHRULJLQDO$:$&6WHPSHUDWXUHUDQJHZDVPLQXVGHJUHHVWRSOXVGHJUHHV

&HQWLJUDGH7KDWUDQJHZDVUHGXFHGWR]HURGHJUHHVWRSOXVGHJUHHV&HQWLJUDGH

• 7KHVKRSUHTXLUHPHQWVUHJDUGLQJ*IRUFHSDUDPHWHUVZHUHUHGXFHG

• 7KHYLEUDWLRQSDUDPHWHUZDVUHGXFHGIURPWKHVWDQGDUGDLUERUQH0LO6SHFQXPEHU

WRZKDWZDVPHDVXUHGSOXVDPDUJLQ

• $VDOWVSUD\SDUDPHWHUZDVHOLPLQDWHG

• 7KHRSHUDWLQJKXPLGLW\SDUDPHWHUZDVUHGXFHGIURPSHUFHQWWRSHUFHQW

5HVXOWRI&ULWHULD0RGLILFDWLRQ0RUHUHDOLVWLFHQYLURQPHQWDOUHTXLUHPHQWV

DOORZHG$:$&6WRXVHFRPPHUFLDOFRPSXWHUVDQGVWLOOHQVXUHVDIHW\DQGPLVVLRQ

SHUIRUPDQFH%DVLFDOO\$:$&6GHWHUPLQHGLWZDVDSSURSULDWHWRPDNHVRPH

FRQFHVVLRQVLQRUGHUWRXVH&276FRPSXWHUV

 &DVH6WXG\-'$0

2YHUYLHZ-RLQW'LUHFW$WWDFN0XQLWLRQV-'$0LVDQRWKHUH[DPSOHRID

SURJUDPWKDWOLNH$:$&6WRRNDKDUGORRNDWLWVHQYLURQPHQWDOUHTXLUHPHQWV

(QYLURQPHQWDO&ULWHULRQ0RGLILHG-'$0KDGDQLQLWLDOUHTXLUHPHQWWKDWWKH

ERPEEHDEOHWRIO\FDSWLYHWRWKHSODQHDWIHHWIRUWKUHHKRXUV7KDWUHTXLUHPHQW

GULOOHGWKHWHPSHUDWXUHZLWKLQWKHERPEGUDPDWLFDOO\

:KHQWKHFRQWUDFWRUDVNHG&DQZHJHWVRPHUHOLHIRQWKDW"-'$0ZHQWEDFNWRLWV

SLORWVDQGGLVFXVVHGWKHUHTXLUHPHQWV7KHSLORWVVDLGWKH\QHYHUIOHZDWIHHWIRU

WKUHHKRXUVDQGIRUDZKROHKRVWRIUHDVRQVLQFOXGLQJVDIHW\

5HVXOWRI&ULWHULRQ0RGLILFDWLRQ%DVHGRQLQSXWIURPWKHXVHUV
SLORWV


-'$0UHYLVHGLWVRYHUO\UHVWULFWLYHFDSWLYHERPEIO\LQJSDUDPHWHU7KLVUHYLVLRQ

DOORZHG-'$0WRLQFRUSRUDWH&276
31



&276DQG%HVW3UDFWLFHV

7+(1(('72,'(17,)<%(6735$&7,&(6

7KHVWXG\JURXSUHYHDOHGWKDWVXFFHVVIXOLPSOHPHQWDWLRQRI&276LQWHQVLYHV\VWHPV

KDVEHHQKDPSHUHGE\WRRPXFKHPSKDVLVRQ&276
DMXVWGRLWDWWLWXGHLIQRW

RXWULJKWSROLF\2QWKHRWKHUKDQGQRWHQRXJKHPSKDVLVKDVEHHQSODFHGRQ

XQGHUVWDQGLQJDQGLPSOHPHQWLQJWKHSURFHVVWRGHWHUPLQHWKHDSSOLFDELOLW\WKDWLVWKH

DSSURSULDWHQHVVRI&276LQDSDUWLFXODUDFTXLVLWLRQVWUDWHJ\

0(7+2'2/2*<)25,'(17,)<,1*%(6735$&7,&(6

7RVXSSRUWDQLQFUHDVHGHPSKDVLVRQWKHSURFHVVRIGHWHUPLQLQJWKHDSSOLFDELOLW\RI

&276WKHVWXG\JURXSIROORZHGWKHWUDGLWLRQDO6$%SURFHVVWRLGHQWLI\H[LVWLQJ

SURFHVVHVDQGVWDQGDUGVLQEHVWSUDFWLFHFRPSDQLHV

:KLOHWKHUHSRUWGRHVQRWOLWHUDOO\VHWRXWSURFHVVHVLWGRHVSURYLGHJXLGHOLQHVDQG

OHVVRQVOHDUQHGVXIILFLHQWIRUVXEVHTXHQWFDSWXUHLQZULWWHQSURFHGXUHV

)LJ2YHUYLHZRI&276%HVW3UDFWLFHV

Key Success

• Acquisition

− Flexible requirements

− Contractor incentivized to consider sustainment

− COTS-experienced contractor

• System Development

− Technology refresh major architectural

consideration

− Open system architecture

− Trade study process

− Understand and influence the marketplace

− Application closely matches COTS features

• Management

− Collocated IPT including field operator

− No modification of COTS products permitted

− Strong leadership
32

%(6735$&7,&(6

3URGXFW)DPLOLDULW\

,QFUHDVHG%X\HU3DUWLFLSDWLRQ:LWKWKHLQFUHDVHGHPSKDVLVRQ&276RQHRI

WKHWKLQJVWKDWJRYHUQPHQWDFTXLVLWLRQRIILFHUVQRZPXVWGRLVEHFRPHIDPLOLDUZLWKWKH

DUUD\RIFRPPHUFLDOO\DYDLODEOHSURGXFWVWKDWPD\EHSURSRVHGWRILOODPLVVLRQQHHG

7KLVUHTXLUHPHQWWREHFRPHIDPLOLDUZLWKDYDLODEOHSURGXFWVDQGXQGHUVWDQGKRZWKRVH

SURGXFWVZLOORUZLOOQRWFRQWULEXWHWRWKHILQDOSURGXFW
DFFRPSOLVKPHQWRIWKH

PLVVLRQ
LVQRWXQOLNHEXLOGLQJDFXVWRPKRXVH7KHEX\HUJRHVWRDQDUFKLWHFWDQGD

EXLOGHUDQGDOODORQJWKHZD\SDUWLFLSDWHVLQWKHVHOHFWLRQRIPDQ\PDQ\&276

SURGXFWVNLWFKHQFDELQHWVIORRULQJIL[WXUHVVLGLQJEULFN7KHUHDUHP\ULDGGHFLVLRQV

WKDWWKHFXVWRPKRPHEX\HUPXVWPDNH,WZRXOGEHFUD]\WRWXUQWKHSURFHVVWRWDOO\

RYHUWRWKHDUFKLWHFWRUWKHEXLOGHUDQGVD\<RXSLFNRXWWKHFDELQHWVDQGWKH

FRXQWHUWRSDQGJLYHPHWKHNH\VZKHQWKHKRXVHLVGRQH

7KHVDPHWKLQJLVWUXHZKHQWKHJRYHUQPHQWEX\HULVEXLOGLQJWKHPLVVLRQ7KHEX\HU

KDVWREHPRUHNQRZOHGJHDEOH
PXFKPRUHNQRZOHGJHDEOH
DERXWWKH&276SURGXFWV

WKDWDUHEHLQJSURSRVHGIRUXVHLQWKHLUKRXVH

'HFUHDVHG3DUWLFLSDWLRQIURP2Q+LJK%HFDXVHWKH&276VHOHFWLRQSURFHVV

UHTXLUHVFORVHLQNQRZOHGJHRIERWKWKHQHHGDQGWKHDYDLODEOHSURGXFWV&276

PDQGDWHVUDUHO\DUHDSSURSULDWH7KHEHVWSUDFWLFHLQWKHYDVWPDMRULW\RIVLWXDWLRQVLV

WROHDYHWKHGHFLVLRQZKHWKHURUQRWWRXVH&276WRWKRVHSHRSOHZKRDUHFORVHVWWRWKH

KDUGZDUHDQGVRIWZDUH

 )OH[LEOH5HTXLUHPHQWV

$FRUROODU\WROHDYLQJWKHGHFLVLRQWRWKHSHRSOHZKRDUHWKHPRVWKDQGVRQLVKDYLQJ

IOH[LEOHUHTXLUHPHQWV,ISURJUDPUHTXLUHPHQWVDUHLQIOH[LEOHWKHUHLVOHVVRSSRUWXQLW\

WRXVH&2760RUHRYHULQIOH[LEOHUHTXLUHPHQWVPDNHLWOHVVOLNHO\WKDW&276

LPSOHPHQWDWLRQZLOOEHVXFFHVVIXO

,QGHHGWKHVWXG\JURXSIRXQGWKDWIOH[LELOLW\WKDWLVWKHDELOLW\WRFRQIRUPWKHSURFHVV

WR&276UDWKHUWKDQDWWHPSWLQJWRPRGLI\&276WRILWWKHSURFHVVZDVWKHPRVW

LPSRUWDQWSDUDPHWHULQVXFFHVVIXOLPSOHPHQWDWLRQRI&276

6RIWZDUH:UDSSHUV

7KRVHZKRH[HFXWHWKHEHVWVHFXULW\SUDFWLFHVZLOOSURYLGHZUDSSHUVIRU&276

VRIWZDUH

:UDSSHUVOLPLWZKDWWKHVRIWZDUHFDQGR)RUH[DPSOHLIWKHVRIWZDUHLVQRWVXSSRVHG

WRZULWHDILOHDZUDSSHUFDQSUHYHQWWKHVRIWZDUHIURPZULWLQJDILOH:UDSSHUVZLOO
33

VHWUHVWULFWLRQVRQSDUDPHWHUV7KHVHUHVWULFWLRQVFDQSUHYHQWWKHVRIWZDUHIURPJRLQJ

KD\ZLUH:UDSSHUVDOVRFDQSUHYHQWDEXJWKDWLVKLGLQJLQWKHVRIWZDUHIURPGDPDJLQJ

WKHV\VWHPDOWKRXJKVXFFHVVDWWKLVWDVNLVOLPLWHG

/DUJHU6HFXULW\,VVXHV6HFXULW\LVIDUDQGDZD\WKHPRVWFRPSOLFDWHGDVSHFW

RIWKH&276EX\LQJSURFHVV,WLVREYLRXVWKDWVHFXULW\LVDQHQRUPRXVLVVXHLQXVLQJ

&276SURGXFWVLQ'R'DQGVXERUGLQDWHGHSDUWPHQWPLVVLRQV

3URSRVHG6HFXULW\6ROXWLRQV7RDGGUHVVWKHLVVXHRIVDERWDJH7URMDQ

KRUVHVDQGRWKHUSRWHQWLDOO\GHOHWHULRXVDFWVWKHVWXG\JURXSKHDUGDSURSRVDOWR

DOORZWKLUGSDUW\DFFHVVWRWKHVRXUFHFRGH7KLVWKLUGSDUW\ZRXOGWKHQVFRXUWKHFRGH

IRUSRWHQWLDOVHFXULW\WKUHDWV

5HFRPPHQGDWLRQVRQ6HFXULW\2XWVLGHWKH6FRSHRIWKH6WXG\*URXS
V5HSRUW

7KHVHFXULW\LVVXHVLQKHUHQWLQEX\LQJ&276SURGXFWVDUHKLJKO\FRPSOH[7KHUHSRUW

DGGUHVVHVVHFXULW\VRPHZKDW$WERWWRPKRZHYHUWKHVWXG\JURXSFRQFOXGHGWKDW

VHFXULW\LVVXFKDFRPSOH[LVVXHWKDWLWGHVHUYHVDVHSDUDWHVWXG\

 /RQJHYLW\LQWKH&276%DVHG6\VWHP

7KHVWXG\JURXSIRXQGWKDWWKRVHSURJUDPVDQGRUJDQL]DWLRQVIROORZLQJWKHEHVW

SUDFWLFHVIRU&276EDVHGV\VWHPVZHUHWKRVHWKDWKDGEHHQRSHUDWLQJD&276V\VWHP

IRUWKHORQJHVWSHULRGRIWLPH

&DVH6WXG\1DY\$WWDFN6XEPDULQH$QH[DPSOHRIWKLVSD\RIIRIVKHHU

H[SHULHQFHDQGORQJHYLW\LVWKH/RFNKHHGSURJUDPLQ0DQDVVDV9LUJLQLDIRUWKHQHZ

1DY\DWWDFNVXEPDULQH

7KLVSURJUDP
V&276SURFHVVZDVYHU\LQGHSWK7KH\KDYHEHHQDWLWIRUWHQ\HDUV

7KH\IUHHO\DGPLWWKDWWKH\KDYHPDGHYLUWXDOO\HYHU\PLVWDNHLQWKHERRN%XWWKHLU

ORQJHYLW\LQWKHV\VWHPKDVDOORZHGWKHPWRHYROYHDQGGHYHORSYHU\VSHFLILFYHU\

GHWDLOHG&276SURFHVVHVDVZHOODVWRGHYHORSWKHSHUVRQQHOWRLPSOHPHQWWKRVH

SURFHVVHVH[SHUWO\

/HDGHUVKLSDQGD%XUQLQJ3ODWIRUP

6LPSO\GHYHORSLQJ&276SURFHVVHVKRZHYHUGRHVQRWDOZD\VIXUWKHUWKHUHTXLUHG

WUDQVIRUPDWLRQIURPROGWKLQNLQJGHVLJQDQGEXLOGWRQHZWKLQNLQJEX\DQG

LQWHJUDWH7KDWWUDQVIRUPDWLRQUHTXLUHVOHDGHUVKLSVRPHWLPHVHYHQIURPWKHFXVWRPHU

&DVH6WXG\1DY\DWWDFN6XEPDULQHDQGWKHEXUQLQJSODWIRUP$VLWWXUQHG

RXWZLWKWKH/RFNKHHG1DY\DWWDFNVXEPDULQHSURJUDPLWZDVWKHFXVWRPHUWKDWSXW

/RFNKHHGRQDEXUQLQJSODWIRUP7KHFXVWRPHUVDLGHVVHQWLDOO\'RWKLV
34

LPSOHPHQW&276RU,
PJRLQJWRVKXW\RXGRZQ+RWIHHWWHQGWRKHOSSHRSOH

RYHUFRPHWKHLUQDWXUDOUHVLVWDQFHWRWKLVW\SHRIFKDQJH

7RUHYLHZWKH/RFNKHHGSURJUDPWKHVWXG\JURXSDWWHQGHGZKROHGD\VRIEULHILQJV\HW

QRRQHPHQWLRQHGWKHEXUQLQJSODWIRUP1HYHUWKHOHVVLWZDVREYLRXVWRWKHVWXG\JURXS

WKDWVRPHWKLQJKDGMXPSVWDUWHGWKHSURJUDP
VPRYHPHQWWRZDUG&276)LQDOO\

/RFNKHHGSHUVRQQHODFNQRZOHGJHGWKHPRVWLPSRUWDQWWKLQJWKDWKDSSHQHGLQWKH

SURJUDPZDVWKHIDFWWKDWWKH\EHFDPHFRQYLQFHGWKDWWKH\PXVWXVH&276:KDW

FRQYLQFHGWKHPZDVWKHOHDGHUVKLSLQWKLVFDVHRIWKHFXVWRPHUSXWWLQJWKHPRQD

EXUQLQJSODWIRUP

5RDG0DSV

%HVWSUDFWLFHVXSSOLHUVKDYHDWHFKQRORJLFDOURDGPDS
DORQJWHUPJDPHSODQ
IRU

WKHLUSURGXFWV

%X\HUVPXVWXQGHUVWDQGWKDWURDGPDSEHIRUHWKH\EX\DQGWKHQWKURXJKRXWWKH

SURGXFW
VOLIHF\FOH8VLQJWKHURDGPDSEX\HUVPXVWWDNHLWXSRQWKHPVHOYHVVRIDUDV

SRVVLEOHWR

• PLQLPL]HWKHSRWHQWLDOIRUDQDUWLILFLDOO\TXLFNHQHGWHFKQRORJ\FKDQJHUDWHE\

HQVXULQJIRUH[DPSOHWKDWWKHFDSDELOLW\WRIXOO\DXWRPDWHXSJUDGHVLVGHVLJQHGLQ

• DYRLGPLQLPL]HRUUHGLUHFWZURQJZD\HYROXWLRQ
WKDWLVHYROXWLRQWKDWWDNHVWKH

SURGXFWDZD\IURPWKHWUDFNWKDWHQVXUHVWKHSURGXFWZLOOFRQWLQXHWRFRQWULEXWHWR

WKHPLVVLRQ
35



&276,PSOHPHQWDWLRQ

29(59,(:

7KHVWXG\JURXSUHYLHZHGDSSOLFDELOLW\LHDSSURSULDWHQHVVRI&276LQYDULRXV

V\VWHPV7KHQLWH[DPLQHGWKHLQWHQVLW\RI&276LPSOHPHQWDWLRQLQWKHSURJUDPVDQG

RUJDQL]DWLRQVVWXGLHG

/(9(/62),03/(0(17$7,21,17(16,7<

6(,&DWHJRULHVRI&276,QWHQVLW\

7KHVWXG\JURXSDSSOLHGWKHIROORZLQJ6(,FDWHJRULHVLQUHYLHZLQJ&276V\VWHP

LQWHQVLW\

• ,PSOHPHQWDWLRQRI&276VROXWLRQV\VWHPV&276VROXWLRQV\VWHPVHQWDLORQH

VXEVWDQWLDOSURGXFWVXFKDVD0DQDJHPHQW,QIRUPDWLRQ6\VWHP0,6SURJUDP

• ,PSOHPHQWDWLRQRIDQDJJUHJDWHV\VWHP7KLVLQYROYHVLQWHJUDWLRQRIPXOWLSOH&276

V\VWHPVDOWKRXJKQRWQHFHVVDULO\H[FOXVLYHO\&276

• 7KHOHDVWLQWHQVLYHOHYHORILPSOHPHQWDWLRQLVWDNLQJ&276HOHFWURQLFFRPSRQHQWV

VXFKDVSURFHVVRUVRUPLFURHOHFWURQLFVDQGGHVLJQLQJXSIURPWKDW

%(1(),762)&276,03/(0(17$7,21

0RVWRIWKHEHQHILWVRI&276DUHREYLRXVDQGKDYHEHHQWRXWHGIRUVRPHWLPH7KHVH

EHQHILWVFDQLQFOXGH

• ORZHUFRVWERWKLQLWLDODQGORJLVWLF

• IDVWHUWLPHWRPDUNHW

• LPSURYHGTXDOLW\DQGUHOLDELOLW\

• WKHDELOLW\WROHYHUDJHIDVWSDFHGFRPPHUFLDOWHFKQRORJ\

• UHGXFHGULVNRIGHYHORSPHQWDOWKRXJKDVGLVFXVVHGSUHYLRXVO\YDULRXVULVNV

DVVRFLDWHGZLWKLQWHJUDWLQJ&276VRIWZDUHSURJUDPVPD\EHRIIVHWWLQJ
36

• DSURGXFWVXSSRUWV\VWHPLVLQSODFHLIWKHSURGXFWLVOHIWRQLWVHYROXWLRQDU\WUDFW

XSJUDGHVDQGRWKHUHQKDQFHPHQWVEHFRPHDYDLODEOHWUDFNLQJWKHSURGXFW
V

HYROXWLRQFDQWKHQSURYLGHVHFRQGDU\RUIROORZRQEHQHILWVWRWKHRUJDQL]DWLRQRU

SURJUDP

7DNHQWRJHWKHUWKHVHSRWHQWLDOEHQHILWVRIXVLQJ&276JLYHPDQDJHUVWKHDELOLW\WR

PDLQWDLQPRGHUQL]DWLRQ,QWKHMXGJPHQWRIWKHVWXG\JURXSLWLVFRPSOHWHO\SRVVLEOH

WKDWFRUUHFWO\LPSOHPHQWHG&276FDQDFKLHYHHDFKRIWKHVHEHQHILWVVLPXOWDQHRXVO\

)LJ%HQHILWVRI&276,PSOHPHQWDWLRQ

'(7(50,1,1*:+(7+(5&276,6%(1(),&,$/,1$3$57,&8/$5

$33/,&$7,21

6XVWDLQPHQWDQGRU/RJLVWLFV&RVWV

7KHUHLVGDWDWKDWVXSSRUWVWKHFRQFOXVLRQWKDWLPSOHPHQWHGFRUUHFWO\&276V\VWHPV


VXVWDLQPHQWDQGRUORJLVWLFVFRVWVVKRXOGEHQRKLJKHUDQGSHUKDSVORZHUWKDQD

SURIHVVLRQDOLHDGHVLJQHGV\VWHP

• 'DWDGHPRQVWUDWLQJFRQVLGHUDEO\ORZHUVXSSRUWFRVWVIRU&276FRPHVIURPWKH

VXPPDU\SURJUDP7KDWSURJUDPKDVEHHQILHOGHGIRUDERXWWKUHH\HDUV$Q

LPSRUWDQWFRXQWHUSRLQWWKRXJKLVWKDWWKHDELOLW\WRPDLQWDLQPRGHUQL]DWLRQFDQ

OHDGWRDPRUHUDSLGXSJUDGHF\FOHIRUVRIWZDUHIRUH[DPSOHDQPRQWKF\FOH

DQGWKHGHVLUHWRUHIUHVKWHFKQRORJ\PRUHUHJXODUO\0RUHUDSLGXSJUDGLQJDQG

PRUHIUHTXHQWUHIUHVKLQJZLOOGULYHFRVWVXSDJDLQ

• 7KHVWXG\JURXSFRPSDUHGHIIRUWYV\HDUVLQDWUDGLWLRQDO0LO6SHF
WKDWLVD

FXVWRPGHVLJQHGV\VWHP
WRDZHOOUXQ&276V\VWHP:KDWEHFDPHFOHDUDWOHDVW

WRWKHVWXG\JURXSZDVWKDWWKH&276V\VWHPDFFRPSOLVKHGWKHIURQWHQGRIWKH

SURJUDPFKHDSHUDQGVRRQHU:KDWLVQRWVRFOHDULVIXWXUHVXVWDLQPHQWRUORJLVWLFV

FRVWV

Potential Benefits

Primary

• Lower cost (initial procurement and

logistics support)

• Faster deployment (time to market)

• Improved quality and reliability

• Leverage fast-paced commercial

product evolution

• Reduced development risk

• Support system in place

• Upgrades provided

Secondary

• More stable industrial base

• Decreased reliance on sole

providers

• Improved surge capability

• Facilitates innovation from small

businesses/academia

• Easier to attract new employees

− Mainstream development

environment
37

0DLQWDLQLQJ0RGHUQL]DWLRQ

7KHUHDODQGVXEVWDQWLDODGYDQWDJHWR&276LVWKHDELOLW\LWJLYHVPDQDJHUVWRPDLQWDLQ

PRGHUQL]DWLRQ

'HVSLWHWKHFRVWSRWHQWLDORIPRUHUDSLGXSJUDGHDQGUHIUHVKF\FOHVWKHVWXG\JURXS

FRQFOXGHGWKDWWKHDELOLW\WRPDLQWDLQPRGHUQL]DWLRQZDVDYHU\ELJDGYDQWDJHIRU

&2767KHUHLVQRW\HWHQRXJKGDWDRQ&276LQWHQVLYHV\VWHPVWRPDNHTXDQWLWDWLYH

MXGJPHQWVDERXWLWVFRVWV2QHWKLQJWKDWLVFOHDUKRZHYHULVWKDWFRVWVDUH

LQIOXHQFHG
XSRUGRZQ
E\KRZZHOOPRGHUQL]DWLRQLVPDLQWDLQHG

)LJ3LWIDOOVRI&276,PSOHPHQWDWLRQ

Process Issues - Acquisition

• Inappropriate application

• Inflexible requirements

• Inadequate treatment of Total Ownership Cost

• Lack of thorough COTS products search & evaluation

• Inadequate product volatility consideration

− Open system architecture

− Spiral development

• Difficulty of maintaining configuration control

• Complexity of integrating multiple products

• Misunderstanding of commercial practices

• Inability to incrementally perform operational test and

evaluation (OT&E) on an evolving product

Product Issues

• Trend towards integrated vs. federated

• Evolving industry standards

• Claims and performance don’t always match

• Interoperability

• Performance/feature clash

• Frequent upgrades

• Questionable trust (Trojan Horse)

• Right to use, not own

• Source code not available

• Lack of support for niche features

• Environmental performance

• Plastic encapsulated microcircuits (PEMs)
38

3,7)$//62)&276,03/(0(17$7,21

7KHSLWIDOOVRI&276SUREDEO\DUHRIPRUHLQWHUHVWWKDQWKHEHQHILWV

0HWKRGRORJ\8VHGWR(YDOXDWH3LWIDOOV

7RVHHWKHELJSLFWXUHRISRVVLEOHSLWIDOOVLQ&276LPSOHPHQWDWLRQWKHVWXG\JURXS

FRPSDUHGHIIRUWYV\HDUV

(IIRUWLVDFRQFHSWVLPLODUWRPDQSRZHURUH[SHQGLWXUHUDWH

<HDUVLVWKHWLPHSHULRGIURPSURJUDPVWDUWWRPLOHVWRQHRUSURJUDPFRPSOHWLRQ

&RUSRUDWH5HWLUHPHQW3ODQV

3RWHQWLDOO\3HUSHWXDORU,QRUGLQDWH&RUSRUDWH,QYROYHPHQW2QHRIWKH

REMHFWLRQVWR&276EDVHGV\VWHPVLVWKDWWKH\PD\FUHDWHDFRUSRUDWHZRUOG

UHWLUHPHQWSODQ

7KHLGHDLVWKDWLIDFRUSRUDWLRQFDQFRQYLQFHWKHJRYHUQPHQWWREX\D&276V\VWHP

DQGWKHFRUSRUDWLRQFRQWUROVWKHWHFKQRORJ\DQGLWVHYROXWLRQWKHQWKHFRUSRUDWLRQFDQ

JHQHUDWHDQHZF\FOHDVVRRQDVLWKDVWKHWHFKQRORJ\LQKDQG7KHFRUSRUDWLRQFDQDV

SUHYLRXVO\GLVFXVVHGLQWKHFRQWH[WRIVXSSOLHUVXSSRUWLQHVVHQFHIRUFHWKHJRYHUQPHQW

WREX\DFRQWLQXRXVF\FOHRIQHZWHFKQRORJ\DVUDSLGO\DVWKDWWHFKQRORJ\FDQEHSXWRQ

WKHVWUHHW

,QVKRUWWKHREMHFWLRQLVWKDW&276JLYHVWKHFRPPHUFLDOYHQGRUFRQWURORYHUZKHQ

ZKHUHDQGKRZDQRUJDQL]DWLRQPXVWUHLVVXHKDUGZDUHRUVRIWZDUH

&RQVLGHUDWLRQV7KHREMHFWLRQKDVVRPHPHULW7KHNH\SRLQWVWRFRQVLGHULQ

UHVSRQVHWRWKLVREMHFWLRQDUH

• 7KHFRPPHUFLDOPDUNHWSODFHQRWDQ\JLYHQYHQGRUFRQWUROVWKHWHFKQRORJ\7KDW

LVDVXEWOHEXWYHU\LPSRUWDQWGLIIHUHQFH

• ,QPRVWFDVHV&276SURGXFWVDUHDYDLODEOHIURPPXOWLSOHVRXUFHV7KLVSUHYHQWVD

SDUWLFXODUYHQGRUIURPGULYLQJWKHXSJUDGHDQGUHIUHVKSDFHIDVWHUWKDQWKHSDFHVHW

E\WKHFRPPHUFLDOPDUNHWSODFH

5HVSRQVH%HFDXVHWKHUHLVWKHSRWHQWLDOIRUSHUSHWXDO
RUSRWHQWLDOO\

LQRUGLQDWH
FRUSRUDWHLQYROYHPHQWLQ&276EDVHGV\VWHPVLWLVYHU\LPSRUWDQWWR

FDUHIXOO\VHOHFWVXSSOLHUVDQGEHDZDUHRIWKHGDQJHURIEHLQJORFNHGLQ
39

0RUHRYHUWKHVWXG\JURXSVWURQJO\DGYRFDWHVWKDWDFRUHWHDPRISHRSOHIURPWKH

SULPHFRQWUDFWRUEHNHSWLQSODFHGXULQJWKHVXVWDLQPHQWRIWKHV\VWHP7KLVWHDPLV

QHFHVVDU\WRUHVSRQGIRUH[DPSOHWRLVVXHVRIREVROHWHSDUWVDQGGLPLQLVKHG

PDQXIDFWXULQJ7KHVHSUREOHPVZKLFKSODJXHSURJUDPVQRZFDQEHH[DFHUEDWHGLQWKH

&276HQYLURQPHQW

)LJ7HFKQRORJ\&KDQJH&\FOH,VVXHV

$UWLILFLDOO\4XLFNHQHG&KDQJH&\FOH

7KH,PSDFWRIWKH&KDQJH&\FOHRQ2SHUDWLRQDO5HDGLQHVV$QRWKHU

REMHFWLRQVRPHWLPHVUDLVHGWR&276EDVHGV\VWHPVLVWKDWWKHSRWHQWLDOIRUWKHYHQGRU

WRTXLFNHQWKHXSJUDGHDQGUHIUHVKUDWHXOWLPDWHO\LPSDFWVQHJDWLYHO\RQRSHUDWLRQDO

UHDGLQHVV

7KHIDVWSDFHRIWHFKQRORJ\FKDQJHLQVRPHLQVWDQFHVFDQFDXVHFRQVLGHUDEOH

KHDGDFKHVLWVHHPVWRFUHDWHFRQVWDQWUROORYHUDQGUHF\FOLQJLQFUHDVHGWUDLQLQJ

UHTXLUHPHQWVFKDQJHVLQFRQILJXUDWLRQPDQDJHPHQWWKHGHYHORSPHQWDQG

GLVVHPLQDWLRQRIQHZGRFXPHQWDWLRQ

7RWKHH[WHQWWKDWWKHYHQGRULVSXVKLQJWHFKQRORJ\FKDQJHIDVWHUWKDQWKHPDUNHWSODFH

LHDUWLILFLDOO\TXLFNHQHGFKDQJHVRPHPDQDJHUVPD\IHHOWKH\GRQRWKDYHWKH

RSWLRQWRVD\,
PJRLQJWRVNLSDJHQHUDWLRQRUWZRDQGVLPSOLI\P\SUREOHPRI

UHWUDLQLQJWKHWURRSVUHLVVXLQJWKHGRFXPHQWDWLRQKDYLQJPXOWLSOHYDULDWLRQVRIWKH

V\VWHPLQWKHILHOG2WKHUZLVHLWFDQVHHPWKDWE\WKHWLPHRQHYHUVLRQLVILHOGHGLWLV

WLPHWRWXUQDURXQGDQGGRLWDOORYHUDJDLQ

7KH&KDQJH&\FOHDQGWKH(QG8VHU%HFDXVHRIWKHUDWHRIWHFKQRORJ\

FKDQJH&276EDVHGV\VWHPVRIWHQORRNXJO\WRHQGXVHUV)URPWKHLUSHUVSHFWLYH

WKHUHVLPSO\LVQ
WHQRXJKWLPHWRJHWXSWRVSHHGDQGUHDOO\SURILFLHQWRQRQHV\VWHP

EHIRUHLWLVWLPHWRUROORYHUDQGVWDUWOHDUQLQJDQHZRQH

Process Issues —

Logistics and Support

• Short commercial product life cycle

• Multiple configurations

• Software upgrades

• Technology refresh

• Frequent procedure updates and retraining

• Diminishing manufacturing supply/obsolete parts
40

$FKDQJHUHTXLUHVVZDSSLQJRXWHOHFWURQLFVER[HVIRUH[DPSOHDQGDWHFKQLFLDQFRPHV

LQDQGVD\V7KHUH
VDQHZER[:KHUHLVP\QHZDXWRPDWHGWHVWSURJUDP":KHUH
V

P\QHZLQWHUIDFHV\VWHP":KHUHLVP\QHZGRFXPHQWDWLRQ"7KLVLVZKDWWKHSRWHQWLDO

IRUDQHJDWLYHLPSDFWRQRSHUDWLRQDOUHDGLQHVVLQFOXGLQJPRUDOHORRNVOLNHWRWKHXVHU

&RQVLGHUDWLRQV

• 0DUNHWUDWHFKDQJH7KHLPSRUWDQWSRLQWKHUHLVWRUHFRJQL]HWKDWZKLOHWKHUDWHRI

WHFKQRORJLFDOFKDQJHRIWHQLVGLVUXSWLYHVRPHRIWKDWGLVUXSWLRQLVXQDYRLGDEOH7KH

UDWHRIFKDQJH
WKDWLVWKHUDWHVHWE\WKHFRPPHUFLDOPDUNHWSODFH
LVVLPSO\WKH

UDWHRIFKDQJH,WLVYHU\LPSRUWDQWXSDQGGRZQWKHRUJDQL]DWLRQWRSDUVHWKH

HIIHFWVRIWHFKQRORJ\FKDQJHWKDWEHFDXVHRIWKHSDFHRIHYROXWLRQLQWKH

FRPPHUFLDOPDUNHWSODFHDUHXQDYRLGDEOH:HDVWHFKQRORJ\XVHUVPXVWDGDSWWR

WKDWFKDQJHRUGLH

• $UWLILFLDOO\TXLFNHQHGUDWHRIFKDQJH$SUREOHPPD\DULVHKRZHYHULIE\

ZKDWHYHUPHFKDQLVPDYHQGRULVDEOHWRSXVKWKHUDWHRIWHFKQRORJ\FKDQJHEH\RQG

ZKDWWKHPDUNHWRWKHUZLVHZRXOGUHTXLUH6XFKDFKDQJHUDWHLVDUWLILFLDOLQWKH

VHQVHWKDWLWLVQRWSURSHOOHGE\PDUNHWG\QDPLFV

• 6FRSHRIWKHFKDQJH,QHYDOXDWLQJDQ\SRWHQWLDOIRUWKHUDWHRIWHFKQRORJ\FKDQJHWR

LPSDFWQHJDWLYHO\RQRSHUDWLRQDOUHDGLQHVVWKHW\SHRIFKDQJHPXVWEHDUWLFXODWHG

,WLVQRWVDIHWRDVVXPHWKDWHYHU\FKDQJHLVDVGLVUXSWLYHDVUHPRYHDQGUHSODFH

7RWKHFRQWUDU\VRIWZDUHPRGLILFDWLRQVRIWHQDGGRQO\IXQFWLRQDOLW\,QWKDWFDVH

WKHIROORZRQLPSDFW
WUDLQLQJDQGVRRQ
LVOLPLWHGWRWKRVHQHZIXQFWLRQV

5HVSRQVH$VSUHYLRXVO\HPSKDVL]HGVRPHFULWLFDODVSHFWVRIWKH&276

GHFLVLRQPDNLQJSURFHVVDUH

• DYRLGLQJWKHSRWHQWLDOWREHORFNHGLQWRDSDUWLFXODUYHQGRU

• PDLQWDLQLQJDVXVWDLQPHQWWHDPIURPWKHSULPHFRQWUDFWRU

• XQGHUVWDQGLQJWKDWDFHUWDLQDPRXQWRIWHFKQRORJ\FKDQJHLVXQDYRLGDEOH

$UWLILFLDOO\4XLFNHQHG&KDQJH&\FOHVDQGWKH6HOHFWLRQ3URFHVV7KHFKDOOHQJH

DVZHOODVWKHLPSHUDWLYHIRUPDQDJHUVLVWROLPLWWHFKQRORJ\FKDQJHDQGLWVIROORZRQ

LPSDFWVRQRSHUDWLRQDOUHDGLQHVVWRWKHEDUHPLQLPXPFRQVLVWHQWZLWKPDUNHWSODFH

G\QDPLFV
41

7KDWFKDOOHQJHRQFHDJDLQIRFXVHVRQWKHVHOHFWLRQSURFHVV,WLVXSWRWKHDFTXLVLWLRQ

ZRUNIRUFHWRHQVXUHWKDWDWWKHRXWVHWZHEX\V\VWHPVWKDWDUHGHVLJQHGVRZHOOWKDW

IROORZRQLPSDFWVRIXSJUDGHVDQGVRRQZHUHDGGUHVVHGIXOO\LQWKHGHVLJQSKDVH$Q

H[DPSOHRIJRRGGHVLJQWKDWSUHHPSWVUHDGLQHVVSUREOHPVODWHULVRQHWKDWEXLOGVLQWKH

FDSDFLW\WRIXOO\DXWRPDWHXSJUDGHV

'DWDVXSSRUWVWKHDFKLHYDELOLW\RIVXFKGHVLJQV7KHSUREOHPXVXDOO\LVRQWKHEX\HU
V

SDUW7UDGLWLRQDOO\GXULQJGHYHORSPHQW'R'GRHVQRWSD\DORWRIDWWHQWLRQWRKRZ

ZHOODSURGXFWZLOOPLQLPL]HWKHGLVUXSWLRQRIFKDQJHLQWKHIXWXUH%XWRQHRIWKHPRVW

HIILFLHQWZD\VWRPLQLPL]HFKDQJHUHODWHGGLVUXSWLRQLVWRGHVLJQDQGEX\SURJUDPVZLWK

WKDWJRDOLQPLQG

+XPDQ6NLOOV$FTXLVLWLRQ

&KDQJH5DWHDQG+XPDQ6NLOOV$FTXLVLWLRQ$QRWKHUSLWIDOOLQKHUHQWLQXVLQJ

&276
DQGWKXVEHFRPLQJVXEMHFWWRWKHUDWHRIWHFKQRORJ\FKDQJHVHWE\WKH

PDUNHWSODFHZKDWHYHUWKDWUDWHLV
DSSHDUVZKHQVRIWZDUHFDSDELOLWLHVDQGKXPDQ

FDSDELOLWLHVPXVWLQWHUVHFWWRDFFRPSOLVKDWDVN

&DVH6WXG\6RIWZDUHWR'HWHFW0DQ0DGH1RLVH$QH[DPSOHRIWKH

LQWHUVHFWLRQRIVRIWZDUHDQGKXPDQFDSDELOLWLHVRFFXUVLQWKHXVHRIVRIWZDUHWRGHWHFW

PDQPDGHQRLVHV

:KLOHWKHVRIWZDUHHQKDQFHVGHWHFWLRQWKHDFWXDOGHWHFWLRQDQGLGHQWLILFDWLRQRIYDULRXV

PDQPDGHQRLVHVUHTXLUHVKXPDQVNLOODQGH[SHULHQFH7KLVWDVNLQYROYHVVXEWOH

GLIIHUHQFHVDQGQXDQFHVWKDWDUHUHFRJQL]DEOHRQO\WKURXJKVXEVWDQWLDOKDQGVRQ

WUDLQLQJ

,QWKLVVLWXDWLRQDW\SLFDOUHIUHVKUDWHRIPRQWKVFDQQHJDWLYHO\LPSDFWKXPDQVNLOOV

DFTXLVLWLRQ

5HVSRQVH7KHSRWHQWLDOWKDWWKHWHFKQRORJ\FKDQJHUDWH
HYHQLIWKDWUDWHLV

QRWDUWLILFLDOO\TXLFNHQHG
ZLOOQHJDWLYHO\LPSDFWKXPDQVNLOOVDFTXLVLWLRQDQG

WKHUHIRUHRSHUDWLRQDOUHDGLQHVVPXVWEHDGGUHVVHGDWWKHIURQWHQGEHIRUHD&276

SURGXFWLVVHOHFWHG

 ,QWHJUDWLRQ

,QWHJUDWLRQDQG8QXVHG)XQFWLRQDOLW\8QOLNHDGHVLJQHGSURGXFWD&276

SURGXFWDWOHDVWLQWKHRU\RIWHQGHOLYHUVPXFKPRUHIXQFWLRQDOLW\WKDQLVUHTXLUHG

7KHVHXQXVHGIXQFWLRQVKDYHDZD\RILQWHUDFWLQJZLWKWKHUHVWRIWKHV\VWHPLQDKLJKO\

XQSUHGLFWDEOHZD\

7KHLQWHUDFWLRQRIXQXVHGIXQFWLRQDOLW\PXVWEHWKRURXJKO\ZUXQJRXWGXULQJWKH

LQWHJUDWLRQSURFHVV
42

:KHQWKH&276SURGXFWLVXSJUDGHGWKHVDPHVLWXDWLRQSUHVHQWVLWVHOI6RPHXQXVHG

IXQFWLRQHLWKHUKDVEHHQFKDQJHGRUDGGHG7KLVQHFHVVLWDWHVJRLQJEDFNWKURXJKWKH

LQWHJUDWLRQSURFHVVDQGDJDLQZULQJLQJRXWDQ\QHJDWLYHLQWHUDFWLRQVEHWZHHQXQXVHG

IXQFWLRQVDQGWKHUHVWRIWKHV\VWHP

+LJKO\,QWHJUDWHGYV)HGHUDWHG6\VWHPV&RPPRQO\SUREOHPVZLWK&276

LPSOHPHQWDWLRQRFFXUUHGLQKLJKO\LQWHJUDWHGV\VWHPV,WLVHDVLHUWRXVH&276LQD

IHGHUDWHGV\VWHP

$IHGHUDWHGV\VWHPLVDV\VWHPZKHUHWKHDUFKLWHFWXUHLVSDUWLWLRQHGE\IXQFWLRQ7KHVH

SDUWLWLRQVPDNHLWHDVLHUWRSOXJLQD&276SURGXFW

7UHQG7RZDUG*UHDWHU,QWHJUDWLRQ7KHUHLVDWUHQGWRZDUGJUHDWHUV\VWHPV

LQWHJUDWLRQ7KHPRUHLQWHJUDWHGWKHV\VWHPLVKRZHYHUVXFKDVZKHQIXQFWLRQVVXFK

DVIOLJKWFULWLFDOQRQFULWLFDOIOLJKWIXQFWLRQVDUHLQWHJUDWHGWKHPRUHGLIILFXOWLWLVWR

WDNHDGYDQWDJHRI&276SURGXFWV7KHUHIRUHWKLVWUHQGLVFRXQWHUSURGXFWLYHWR&276

LPSOHPHQWDWLRQ

&RQILJXUDWLRQ&RQWURO3DUWLFXODUO\EHFDXVHRIWKHSHUIRUPDQFHIHDWXUHFODVK

DULVLQJIURPXQXVHGIXQFWLRQDOLW\LQ&276V\VWHPVLWFDQEHGLIILFXOWWRPDLQWDLQ

FRQILJXUDWLRQFRQWURO

 :URQJZD\(YROXWLRQ

'RZQVLGHVRI3URGXFW(YROXWLRQ2QHRIWKHDGYDQWDJHVRI&276LVWKHLU

LQKHUHQWHYROXWLRQ2EYLRXVO\KRZHYHULQVRPHFDVHVWKLVHYROXWLRQFDQKDYHD

GRZQVLGHRISXVKLQJWHFKQRORJ\FKDQJHIDVWHUWKDQWKHXVHUUHTXLUHVRUFDQDEVRUE<HW

DQRWKHUIDFWRUWRFRQVLGHUUHJDUGLQJDSURGXFW
VHYROXWLRQLVZURQJZD\HYROXWLRQ

&RQVLGHUDWLRQV:URQJZD\HYROXWLRQRFFXUVZKHQWKHFRPPHUFLDO

PDUNHWSODFHHYROYHVDSURGXFWLQWKHZURQJGLUHFWLRQ
WKDWLVDZD\IURPWKHWUDFN

'R'QHHGVWKHSURGXFWWRIROORZLQRUGHUIRUWKHSURGXFWWRFRQWLQXHWRPHHW'R'
V

UHTXLUHPHQWV

&OHDUO\ZURQJZD\HYROXWLRQFDQOHDYHDXVHUKDQJLQJ$EX\HUIRUH[DPSOHPD\

SXUFKDVHDVRIWZDUHSDFNDJHVSHFLILFDOO\WRREWDLQDQLFKHIHDWXUH2YHUWLPHWKH

FRPPHUFLDOPDUNHWSODFHEHJLQVWRGLVOLNHWKDWQLFKHIHDWXUH6RVXSSRUWJRHVGRZQ
43

5HVSRQVH7KHDQVZHUKHUHDJDLQFRPHVEDFNWRWKHLQLWLDOEX\LQJSURFHVVDQG

RQJRLQJLQWHUDFWLRQZLWKVXSSOLHUV

%HVWSUDFWLFHFRPSDQLHVSDUWLFLSDWHLQVWDQGDUGVFRPPLWWHHVQRWZLWKWKHLGHDWKDWWKH

FRPPLWWHHVFDQEHLQIOXHQFHGYHU\PXFKDOWKRXJKFRPSDQLHVFHUWDLQO\WU\%HVW

SUDFWLFHFRPSDQLHVKDYHDWHFKQRORJLFDOURDGPDS
DORQJWHUPJDPHSODQ
IRUWKHLU

SURGXFWV

%X\HUVQHHGWRXQGHUVWDQGWKDWURDGPDSEHIRUHWKH\EX\DQGWKHQWKURXJKRXWWKH

SURGXFW
VOLIHF\FOH%X\HUVPXVWWDNHLWXSRQWKHPVHOYHVVRIDUDVSRVVLEOHWRDYRLG

PLQLPL]HRUUHGLUHFWZURQJZD\HYROXWLRQ

3HUIRUPDQFH&ODLPV

7KHVWXG\JURXSIRXQGWKDWPDQ\SUREOHPVDURVHIURPWKHIDFWWKDWFODLPVIRUSURGXFWV

DQGSHUIRUPDQFHRIWKRVHSURGXFWVRIWHQGRQRWPDWFK7KLVZDVDFDXVHRIWKHIDLOXUH

RIWKH&276V\VWHPDW'00,6

,W
VFRPPRQVHQVH3URJUDPVDQGRUJDQL]DWLRQVFDQQRWEX\WKLQJVEDVHGVROHO\RQDQ

DGYHUWLVLQJEURFKXUH

 ,QWHURSHUDELOLW\DQG'HEXJJLQJ

3HUIRUPDQFH)HDWXUH&ODVK3DUWLFXODUO\ZLWKVRIWZDUH&276FDQFUHDWH

LQWHURSHUDELOLW\SUREOHPV7KHVHSUREOHPVSULPDULO\ZHUHPDQLIHVWHGDVFODVKHV

EHWZHHQSHUIRUPDQFHDQGIHDWXUHVHVSHFLDOO\LQUHJDUGWRXQXVHGIXQFWLRQDOLW\

FRQIOLFWLQJZLWKWKHUHVWRIWKHV\VWHP7KLVLVDQLQWHJUDWLRQLVVXHWKDWUHTXLUHVFDUHIXO

DWWHQWLRQ

6RXUFH&RGH,VVXHV2QHFKDQJH&276FUHDWHLVWKHIDFWWKDWLQD&276

V\VWHP'R'EX\VWKHULJKWWRXVHVRIWZDUHDQGKDUGZDUH'R'GRHVQ
WRZQLW7KDW

ORFNV'R'HQJLQHHUVRXWRIWKHVRXUFHFRGHWKDW
VKDUGIRUHQJLQHHUVWRDFFHSW

(QJLQHHUVZDQWWRJHWLQVLGHWKHVRIWZDUHDQGILGGOHDQGWZHDNDQGWKH\FDQ
WGRWKDW

ZLWK&276

%HFDXVHHQJLQHHUVFDQQRWJHWLQVLGH&276VRIWZDUH&276PDNHGHEXJJLQJGLIILFXOW

,QRZQHGSURGXFWVVRIWZDUHHQJLQHHUVDUHDFFXVWRPHGWRSHQHWUDWLQJWKHVRXUFHFRGH

LQVWDOOLQJIODJVDQGRYHUVHHLQJWKHGHEXJJLQJSURFHVV:LWK&276WKDWFDQQRWEH

GRQH,QWKLVUHJDUG&276VRIWZDUHLVPRUHOLNHKDUGZDUH7KDWIDFWUHTXLUHVDQHZ

PHQWDOLW\DQGDSSURDFKWRGHEXJJLQJ
44

0DQGDWHVWR8WLOL]H&276

7KHRYHUDUFKLQJFRQFOXVLRQRIWKHVWXG\JURXSZDVWKDW&276EDVHGV\VWHPVFRXOGEH

VXFFHVVIXORQO\WRWKHH[WHQWWKHUHZDVOHVVHPSKDVLVRQXVLQJ&276DQGPRUH

HPSKDVLVRQWKHSURFHVVRIGHWHUPLQLQJZKHQLWLVDSSURSULDWHWRXVH&276

7KHUHDUHVRPHVLWXDWLRQVWKDWVLPSO\DUHQRWJRRGFDQGLGDWHVIRUD&276LQWHQVLYH

V\VWHP)LJKWHUDLUFUDIWDYLRQLFVXSJUDGHLVRQHVXFKVLWXDWLRQ1XFOHDUUHDFWRUFRQWUROV

LVDQRWKHUVLWXDWLRQZKHUH&276ZRQ
WILW7KHVDPHJRHVIRUFRQWUROVIRUDQ\QXFOHDU

ZHDSRQ

:KLOHWKHUHDUHVRPHVLWXDWLRQVZKHUHLWLVREYLRXV&276DUHDQRJRLQWKHYDVW

PDMRULW\RIFDVHVWKHGHFLVLRQWRXVHRUQRWXVH&276VKRXOGEHOHIWXSWRWKHSHRSOH

ZKRDUHFORVHVWWRWKHKDUGZDUHDQGVRIWZDUH

)DLOXUHWR8QGHUVWDQG&RVWV

$OWKRXJK72&LVLPSRUWDQWWRWKHORQJWHUPXQGHUVWDQGLQJDQGHYDOXDWLRQRI&276

V\VWHPVSHRSOHGRQ
WHYHQWDONDERXW72&

7KH8QLYHUVLW\RI6RXWKHUQ&DOLIRUQLDKDVEHHQZRUNLQJIRU\HDUVWRIRUPDOL]HWKH

FRQFHSWRI72&7KH\KDYHLGHQWLILHGDOOWKHFRVWIDFWRUVDQGFUHDWHGFRVWPRGHOV

$SDUWIURPD72&PRGHOFRVWSURMHFWLRQVIRU&276ZLOOEHLQDFFXUDWH

6WDWLF5ROHVDQG5HVSRQVLELOLWLHV

,PSOHPHQWDWLRQRI&276EDVHGV\VWHPVFKDQJHVUROHVDQGUHVSRQVLELOLWLHVGUDPDWLFDOO\

7KLVLVDQGVKRXOGEHDSRLQWRIJUHDWFRQFHUQ

7RVXFFHHGDQ\PDMRUFKDQJHLQLWLDWLYH
DQGFHUWDLQO\WKHWUDQVLWLRQWR&276LVD

PDMRUFKDQJHLQLWLDWLYH
UHTXLUHVDFRPSUHKHQVLYHPXOWLIDFHWHGJDPHSODQ3DUWRIWKDW

JDPHSODQKDVWREHVRPHRQHPDNLQJLWFOHDUWRHYHU\RQHLQYROYHGWKDWLI\RXGRQ
W

JHWRQERDUG\RX
UHRXW
OLWHUDOO\

,QVXPLIDPDMRUFKDQJHLQLWLDWLYHHYHULVJRLQJWRJHWRIIWKHJURXQGLWUHTXLUHVD

EXUQLQJSODWIRUP,WWDNHVOHDGHUVKLSWRFRQVWUXFWWKDWSODWIRUP

/RJLVWLFDO9RODWLOLW\

$VWRORJLVWLFVZHDOONQRZWKHFRPPHUFLDOPDUNHWSODFHLVYHU\YRODWLOH7RWKHH[WHQW

'R'KDV&276LQWHQVLYHV\VWHPVLQSODFHWKDWYRODWLOLW\KDVDULSSOHHIIHFWLQWKHILHOG
45

8QUHDOLVWLF([SHFWDWLRQV

7KHSHRSOHLVVXHLQ&276FRPHVGRZQWRWKHIDFWWKDWOHDGHUVKDYHXQUHDOLVWLF

H[SHFWDWLRQV/HDGHUVFRQFOXGHWKDW&276LVVWUDLJKWIRUZDUGEXWLWLVQRW&276LVWKH

RSSRVLWHRIVWUDLJKWIRUZDUG

,QWHUQDO8VH3URGXFWV

7KHGDQJHUH[LVWVWKDWSHRSOHFDQEX\ZKDWWKH\WKLQNLVD&276SURGXFWIURPDSXEOLF

FRPSDQ\RQO\WRGLVFRYHUODWHUWKDWWKHSURGXFWZDVGHYHORSHGIRULQWHUQDOXVH

,QWHUQDOXVHSURGXFWVDVRSSRVHGWRIRUSURILWFRPPHUFLDOO\DYDLODEOHSURGXFWVDUH

KLJKO\XQSUHGLFWDEOHDVWRWKHDYDLODELOLW\DQGFRVWRIWHFKQLFDOVXSSRUWDQGXSJUDGHV

7KHUHIRUHWKHVWXG\JURXSVWURQJO\DGYLVHVWKDWWKHSURGXFWEHFRPPHUFLDOO\DYDLODEOH

DQGVROGIRUDSURILWUDWKHUWKDQIRULQWHUQDOXVH7KDWLVWKHSURGXFWVKRXOGEHWUXO\

RIIWKHVKHOI
46



&276DQG)XQGLQJ,VVXHV

29(59,(:

&276EDVHGV\VWHPVUHTXLUHQHZWKLQNLQJLQIXQGLQJSURFHGXUHV0RUHRYHUEXGJHWDU\

F\FOHVPXVWEHPRUHIOH[LEOHDQGPRUHV\QFKURQRXVZLWKWHFKQRORJ\F\FOHV

)URP&RORURI0RQH\WR72&

1HZFRQFHSWVDUHQHFHVVDU\WRWUDQVLWLRQWKHFXUUHQWIXQGLQJV\VWHPIURPDFRORURI

PRQH\IRFXVWRDIRFXVEHWWHUDGDSWHGWRWKHUHDOLWLHVRI&276V\VWHPV7KDWQHZIRFXV

PD\EHRQ72&

)XQGLQJ,VVXHVDQG6XSSRUWLQJ'DWD

3UHVHQWO\WKHVXSSRUWIRUD72&IRFXVHGIXQGLQJV\VWHPLVPDLQO\SKLORVRSKLFDO
WKH

VWXG\JURXSIXQGDPHQWDOO\EHOLHYHVWKDWXVLQJFRPPHUFLDOSURFHVVHVSUDFWLFHVDQG

SURGXFWVZLOOGULYHGRZQFRVWV

:KLOHWKHUHH[LVWVVRPHGDWDWKDWWHQGVWRVXSSRUWWKHFRQFOXVLRQWKDW&276UHVXOWLQ

ORZHUFRVWVPRUHGDWDPXVWEHGHYHORSHG

75$',7,21$/)81',1*&859(

$WUDGLWLRQDOPLOLWDU\VSHFLILFDWLRQ0LO6SHFWKDWLVGHVLJQFRQWUDFWKDVDSDUWLFXODU

OLIHF\FOH

/LIH&\FOH

)LUVWLQDWUDGLWLRQDO0LO6SHFVHWWLQJWKHGHVLJQJRHVLQWRUHVHDUFKGHYHORSPHQWWHVW

DQGHYDOXDWLRQ5'7	($IWHUVXFFHVVIXOFRPSOHWLRQRIWKDWSKDVHWKHSURGXFWLV

SURFXUHG$WWKDWSRLQWWKHSURGXFWLVDQ2SHUDWLRQDODQG0DLQWHQDQFH2	0LWHP

XVXDOO\ORQJWHUP2YHUWKHOLIHF\FOHRIWKHSURGXFW2	0IXQGVDUHXVHGIRUXSJUDGHV

DQGPRGLILFDWLRQV

&RORURI0RQH\

7KLVWUDGLWLRQDOIXQGLQJFXUYH
5'7	(WRSURFXUHPHQWWR2	0
GHPRQVWUDWHVWKH

KLVWRULFDOVHSDUDWLRQRIIXQGVRIWHQFDOOHGWKHFRORURIPRQH\WKDWFKDUDFWHUL]HV

'R'
VSUHVHQWIXQGLQJV\VWHP
47

&276$72''6:,7+&855(17)81',1*352&(66(6

7KHFXUUHQWIXQGLQJV\VWHPZKLFKUHTXLUHVVWULFWVHSDUDWLRQRIWKHYDULRXVFRORUVRI

PRQH\WHQGVWREHDWRGGVZLWKD&276V\VWHP

7UDGLWLRQDO/LIH&\FOH(YHQWVYV&RPPHUFLDO(YROXWLRQ

,QWKHWUDGLWLRQDO0LO6SHFVHWWLQJLWLVHDVLHUWREXGJHWIRUOLIHF\FOHHYHQWV
SUHFLVHO\

EHFDXVH'R'FRQWUROVWKHGHVLJQDQGWKHUHIRUH'R'H[HUFLVHVPRUHFRQWURORYHU

GHVLJQFKDQJHV

%\FRQWUDVW&276V\VWHPVWUDFNFKDQJHVGLFWDWHGE\WKHFRPPHUFLDOPDUNHWSODFH

,QGHHGRQHRIWKHDGYDQWDJHVRI&276LVWKDWSXUFKDVHRIDSURGXFWDOVRLQHVVHQFH

SXUFKDVHVWKDWSURGXFW
VFRPPHUFLDOHYROXWLRQ

)XQGLQJ3URFHVVHVDQG3URGXFW(YROXWLRQ

:KLOHSURGXFWHYROXWLRQDQGWKHFRQFXUUHQWDELOLW\WRPDLQWDLQPRGHUQL]DWLRQLVRQHRI

WKHFKLHIDGYDQWDJHVRI&276WKHEX\HUUHFHLYHVWKHEHQHILWRIWKDWHYROXWLRQRQO\WR

WKHH[WHQWIXQGLQJV\VWHPVSHUPLW7KHPDUNHWSODFHFDQGULYHDOONLQGVRIWHFKQRORJ\

FKDQJHEXWLIIXQGVDUHQRWDYDLODEOHWREX\WKDWFKDQJHXSJUDGHVDQGVRRQDVD

SUDFWLFDOPDWWHUSURGXFWHYROXWLRQLVORVW

%XGJHW&\FOHVDQG3URGXFW(YROXWLRQ

%HVLGHVFRORURIPRQH\LVVXHVUDLVHGE\WKHSUHVHQWIXQGLQJV\VWHPWKHUHRIWHQH[LVWVD

WLPLQJGLVFRQQHFWEHWZHHQEXGJHWF\FOHVDQGPDUNHWGULYHQWHFKQRORJ\FKDQJHF\FOHV


LHSURGXFWHYROXWLRQ

3UHVHQWO\LWFDQWDNHPRQWKVWZR\HDUVHYHQWKUHH\HDUVWRSXWLQSODFHDQHZ

EXGJHWLQWKHVRIWZDUHWHFKQRORJ\FRPPXQLW\,QWKDWWLPHIUDPHWHFKQRORJ\LQWKH

PDUNHWSODFHFRXOGKDYHFKDQJHGWZRWLPHVRUPRUH

72&

$OWKRXJK72&LVLPSRUWDQWWRWKHORQJWHUPXQGHUVWDQGLQJDQGHYDOXDWLRQRI&276

V\VWHPVSHRSOHGRQ
WHYHQWDONDERXW72&
PXFKOHVVKDV72&EHHQGULYHQLQWR'R'

IXQGLQJSUDFWLFHV7KHUHDUHQRWHYHQWRROVWRIDFLOLWDWHDGLVFXVVLRQRI72&DQGKRZWR

LQFRUSRUDWH72&LQWKHWUDQVLWLRQWR&276

7KH8QLYHUVLW\RI6RXWKHUQ&DOLIRUQLDKDVLGHQWLILHGDOOWKHFRVWIDFWRUVDQGFUHDWHG

72&PRGHOV0RUHXVHPXVWEHPDGHRI72&PRGHOVLQHYDOXDWLQJELGGHUVRU

VXSSOLHUV
48



&ULWLFDO)DFWRUVLQ&2766XFFHVV

%$&.*5281'

7KHVWXG\JURXSUHYLHZHGWKHWRSILYH&276VXFFHVVVWRULHVWRGHWHUPLQHWKHWHQRU

WZHOYHIDFWRUVFULWLFDOWRWKDWVXFFHVV

)LJ7RS)LYH&2766XFFHVV6WRULHV

)$&7256&5,7,&$/72&276,03/(0(17$7,2168&&(66

)OH[LEOH5HTXLUHPHQWV

7KHILUVWDQGPRVWLPSRUWDQWIDFWRULQ&276VXFFHVVLVIOH[LEOHUHTXLUHPHQWV7KH

UHSRUWHPSKDVL]HVWKLVSRLQWUHSHDWHGO\)OH[LEOHUHTXLUHPHQWVDUHFULWLFDOIRUPXOWLSOH

UHDVRQV

([DPSOHVRIIOH[LEOHUHTXLUHPHQWVOHDGLQJWRVXFFHVVIXOLPSOHPHQWDWLRQRI&276LVWKH

$:$&6DQG-'$0SURJUDPV7KH$:$&6DQG-'$0H[SHULHQFHVZLWKIOH[LQJ

HQYLURQPHQWDOFULWHULDDUHVHWRXWDERYH

7KH$:$&6DQG-'$0H[DPSOHVGHPRQVWUDWHWKDWVRPHWLPHVHQYLURQPHQWDO

VSHFLILFDWLRQVDUHQRWEDFNHGXSZLWKDORWRIWKRXJKW7KRVHW\SHVRIVSHFVVKRXOGQRW

VWDQGLQWKHZD\RI&276

,QFHQWLYHVWR&RQWDLQ6XVWDLQPHQW&RVWV

+RZ'R'KDQGOHVVXVWDLQPHQWFRVWVLQD&276V\VWHPLVDQRWKHUDUHDWKDWUHTXLUHVD

FKDQJHLQSKLORVRSK\

Exemplary Programs

• Advanced Amphibious Assault Vehicle (AAAV)

• New Attack Submarine and Rapid COTS Insertion

(NSSN/RCI)

• Airborne Warning and Control System (AWACS) upgrade

• Joint Direct Attack Munitions (JDAM)

• Manufacturing Resource Planning II (MRP II)
49

$:$&6$:$&6VXFFHHGHGLQLPSOHPHQWLQJ&276EHFDXVHWKHFRQWUDFWRU

ZDVLQFHQWLYLVHG
ELJWLPH
WRFRQVLGHUVXVWDLQPHQW0DMRUILQDQFLDOSHQDOWLHVZHUH

LPSRVHGLIWKHSURGXFWH[FHHGHGLWVSURMHFWHGVXVWDLQPHQWFRVW

1DY\$WWDFN6XEPDULQH,QWKHFDVHRIWKH1DY\DWWDFNVXEPDULQHSURJUDP

/RFNKHHGLVRQWKHKRRNIRUVXVWDLQPHQWIRUWKHILUVWILYH\HDUV,QIDFWWKHVKLSLVQRW

''
G0DWHULDO,QVSHFWLRQDQG5HFHLYLQJ5HSRUWRUVROGRIIIRUILYH\HDUV

/RFNKHHGLVDUJXLQJIRUDFRQWUDFWWKDWSURYLGHVWKHPZLWKVXVWDLQPHQWIRUWKHORQJ

WHUP7KH1DY\KRZHYHULVWDNLQJDZDLWDQGVHHDQGUHDVVHVVDSSURDFK

2WKHU&RQWUDFWXDO6HWWLQJV

• ,QWKHFDVHRIDQDZDUGIHHEDVHGFRQWUDFWRUDQGGHYHORSHUVXVWDLQPHQWFRVWVZHUH

KDQGOHGE\VKDULQJDSRUWLRQRIWKHDZDUGIHHZLWKHPSOR\HHVEDVHGRQVXVWDLQPHQW

FRVWV7KLVLQFHQWLYL]HGWKHHPSOR\HHVWRGHVLJQWKHV\VWHPIRUVXEVHTXHQWVXSSRUW

DQGPDLQWDLQDELOLW\

• $VLPLODUDSSURDFKDOVRFRXOGEHXVHGLQDIL[HGSULFHFRQWUDFWVHWWLQJ

*RDORI,QFHQWLYHV 7KHWHDFKLQJSRLQWLVWKDWLQFHQWLYHVPXVWEHWLHGWR

ILQDQFLDOSHUIRUPDQFH7KHFRQWUDFWRUVPXVWEHPDGHWRVXIIHULIWKH\GRQ
WFRQWDLQ

VXVWDLQPHQWFRVWV:LWKRXWLQFHQWLYHVVXVWDLQPHQWFRVWVDUHQRWJRLQJWRUHFHLYHWKH

DWWHQWLRQWKH\GHVHUYHRQWKHIURQWHQGRIWKHGHVLJQ

8SIURQW$ZDUHQHVVRI'HVLJQ&RQVLGHUDWLRQV

8SIURQWGHVLJQFRQVLGHUDWLRQVUHPDLQLPSRUWDQWLQWKHQHZWKLQNLQJRIEX\DQG

LQWHJUDWH,QV\VWHPVGHYHORSPHQWIRUH[DPSOHWHFKQRORJ\UHIUHVKDQGXSJUDGHVDUHD

PDMRUDUFKLWHFWXUDOGHVLJQFRQVLGHUDWLRQIURPWKHEHJLQQLQJ

0RVWUHIUHVKDQGXSJUDGHFKDQJHVDUHVXSSRUWHGE\YHU\GHWDLOHGWUDGHVWXGLHV7KDWLV

WKHSURSRVHGFKDQJHVZHUHVXEMHFWHGWRULJRURXVVWXG\WRGHWHUPLQHWKHWUDGHRII

EHWZHHQDQHQKDQFHGSURGXFWDQG72&)RUEHVWSUDFWLFHFRPSDQLHVWKHLPSDFWRI

WHFKQRORJ\UHIUHVKDQGXSJUDGHVRQ72&LVNQRZDEOHEHFDXVHWKHVHDUHGHVLJQ

FRQVLGHUDWLRQVIURPWKHEHJLQQLQJ

0DWFKLQJ&276)XQFWLRQDOLW\ZLWK5HTXLUHPHQWV

:KLOHIOH[LEOHUHTXLUHPHQWVDUHRQHRIWKHPRVWLPSRUWDQWIDFWRUVLQVXFFHVVRI&276

EDVHGV\VWHPVFOHDUO\IOH[LELOLW\KDVOLPLWV

&276SURGXFWVWKDWGHYLDWHWRRIDUIURPUHTXLUHPHQWVVKRXOGEHDYRLGHG6XFK

SURGXFWVPD\VHHPXVDEOHDWILUVWEXWWKH\OHDGWRPRGLILFDWLRQVXFKDVVRIWZDUH

ZUDSSHUVDQGRWKHUMXU\ULJJLQJ&276VKRXOGQRWEHPRGLILHG5DWKHULQWKHEX\LQJ

SURFHVVWKH&276IXQFWLRQDOLW\PXVWEHPDWFKHGFORVHO\ZLWKUHTXLUHPHQWV
50

(YDOXDWLQJ)XQFWLRQDOLW\$JDLQVW5HTXLUHPHQWV)URPPDQDJHPHQW
V

SHUVSHFWLYHHQVXULQJ&276DUHDJRRGILWZLWKUHTXLUHPHQWVPXVWLQYROYHLQSXW
LQDOO

FDVHV
IURP

• WKHLQWHJUDWHGSURGXFWWHDPV,37

• XVHUV

• RSHUDWLRQDOWHVWHUVDQG

• WKHDFTXLVLWLRQFRPPXQLW\

,QVKRUWHYHU\RQHZKRKDVDVWDNHLQWKHV\VWHPVKRXOGEHLQYROYHGLQFDOLEUDWLQJ

&276WHVWLQJ7KLVLVQHFHVVDU\VRWKDWDQDSSURSULDWHGHFLVLRQRQZKHWKHURUQRWWR

XVHWKH&276LVPDGHEHIRUHQRWDIWHUWKHSURGXFWLVSXUFKDVHG

&KDQJLQJWKH&XOWXUH

7KHWUDQVLWLRQIURPWKHROGWKLQNLQJRIGHVLJQDQGEXLOGWRWKHQHZWKLQNLQJRIEX\DQG

LQWHJUDWHQHFHVVLWDWHVDPDMRUFXOWXUDOVKLIW

7KHVSLUDOSURFHVVRIQRWRQO\JRLQJZLWKEXWDOVRXQGHUVWDQGLQJWKHPDUNHWIORZLV

YHU\GLIIHUHQWIURP'R'
VKLVWRULFDOOLQHDUSURFHVVRIWDNLQJDSURGXFWIURPVWDUWWR

ILQLVKLUUHVSHFWLYHRIZKDWLVKDSSHQLQJLQWKHPDUNHWSODFH

)URP/LQHDUWR6SLUDO3URFHVVHV

• 7KHROGOLQHDUSURFHVV'HVLJQDQGEXLOGZDVDSURFHVVWKDWZDVQRWYHU\YXOQHUDEOH

WRWKHYRODWLOLW\RIWKHPDUNHWSODFH,WZDVDYHU\OLQHDUSURFHVV,WGLGQRWOHQG

LWVHOIWRHYROXWLRQRUPXOWLSOHVLPXOWDQHRXVFKDQJHVWRDFFRPPRGDWHWKHYRODWLOLW\RI

WKHPDUNHWSODFH7KHUHIRUHE\WKHWLPH'R'ILQLVKHGGHYHORSLQJV\VWHPV

UHTXLUHPHQWVDQGVSHFLILFDWLRQVWKHWHFKQRORJ\DYDLODEOHLQWKHPDUNHWSODFHKDG

FRPSOHWHO\FKDQJHG%X\DQGLQWHJUDWHRQWKHRWKHUKDQGWLHV'R'WRWKH

PDUNHWSODFH
DPDUNHWSODFHWKDW'R'GRHVQRWFRQWURODQGZKLFKFKDQJHVYHU\

UDSLGO\

• 7KHQHZVSLUDOSURFHVV%X\DQGLQWHJUDWHLVOLNHDVSLUDOUDWKHUWKDQOLQHDU,WV

GLVWLQJXLVKLQJIHDWXUHLVLWVFRQFXUUHQF\
WKDWLVLWGRHVHYROYHDQGJRWKURXJK

PXOWLSOHVLPXOWDQHRXVFKDQJHVWRDFFRPPRGDWHWKHYRODWLOLW\RIWKHPDUNHWSODFH%\

GHILQLWLRQDVSLUDOSURFHVVLVDSURFHVVFKDUDFWHUL]HGE\VLPXOWDQHLW\
51

)LJ/LQHDUYV6SLUDO3URFHVVHV

&DVH6WXG\/RFNKHHG$QH[DPSOHRIKRZDVSLUDODVRSSRVHGWROLQHDU

SURFHVVORRNVLQSUDFWLFHRFFXUUHGDW/RFNKHHG

$UHYLHZRIDSDUWLFXODUSURJUDPUHYHDOHGDKROHLQWKHV\VWHP,QRWKHUZRUGV

FRPSOHWLRQRIWKHSURMHFWZRXOGUHTXLUHDSDUWLFXODUW\SHRIFRPPHUFLDOSURGXFWEXW

WKDWSURGXFWVLPSO\ZDVQRWDYDLODEOHDWWKHWLPH

,QWKHROGWKLQNLQJWKLVKROHPLJKWKDYHEHHQDVKRZVWRSSHU%XWLQWKHQHZ

WKLQNLQJ
WKLQNLQJPRUHDGDSWHGWRWKHJ\UDWLRQVRIWKHPDUNHW
SURMHFWPDQDJHUVZHUH

H[WUHPHO\IDPLOLDUZLWKWKHFRPPHUFLDOPDUNHWSODFH

,QWKHYLHZRIWKHSURMHFWPDQDJHUVWKH&276WKH\QHHGHGWRFRPSOHWHWKHSURMHFW

ZRXOGEHDYDLODEOHLQWKHPDUNHWSODFHZLWKLQD\HDU7KHFXVWRPHUDJUHHG7KH\

ZRUNHGWKDWDVVXPSWLRQLQWRWKHLUSODQQLQJ

,QIDFWWKH&276ZDVDYDLODEOHZLWKLQWKH\HDU/RFNKHHGSOXJJHGLWLQDQGLW

ZRUNHG

2IFRXUVHWKHRXWFRPHRIDVSLUDOSURFHVVLVQRWDOZD\VVRVXFFHVVIXODVLQWKH

/RFNKHHGH[DPSOH%\WKHWLPHDV\VWHPLVSURSRVHGWKHSURGXFWFRXOGEHQRORQJHU

DYDLODEOH

Required COTS

Approach

(Spiral Development) Traditional Approach

(Waterfall Development)

System

Context

Architecture

& Design

Implementation

Build from Scratch

Fundamental Cultural Shift

Simultaneous

Definition

and Tradeoffs

Marketplace

System

Context

Architecture,

Design &

Test

Buy, Integrate, Continuously Refresh
52

• 6XPPDU\&OHDUO\WKHYRODWLOLW\RIWKHPDUNHWSODFHPDNHVWKHVSLUDOSURFHVVRI

EX\DQGLQWHJUDWHGLIILFXOWWRPDQDJH%XWLWLVWKHYRODWLOLW\RIWKHPDUNHWSODFH
WKH

SDFHDQGVFRSHRIWHFKQRORJ\FKDQJH
WKDWPDNHVWKHWUDQVLWLRQWREX\DQGLQWHJUDWH

QHFHVVDU\LQWKHILUVWSODFH

5LVN6KDULQJ&KDQJLQJWKHDFTXLVLWLRQFXOWXUHIURPGHVLJQDQGEXLOGWREX\

DQGLQWHJUDWHLPSDFWVRQKRZDFTXLVLWLRQPDQDJHUVWUDGLWLRQDOO\WKLQNDERXWULVN

VKDULQJ

7KHVWXG\JURXSFRQFOXGHGWKDWLWLVSRVVLEOHWRUHGXFHGHYHORSPHQWULVNE\

WUDQVLWLRQLQJWRWKH&276HQYLURQPHQWEHFDXVHH[SHULHQFHGFRQWUDFWRUVKDYH

• SLFNHGVWDQGDUGV

• GHYHORSHGV\VWHPV

• NQRZOHGJHRIZKLFKFRPPHUFLDOSURGXFWVZRUN

• GHILQHGVWDQGDUGEXVVHVDQGRSHQDUFKLWHFWXUH

:KHQDOOWKDWH[SHULHQFHLVEURXJKWWREHDURQGHFLVLRQVRQEX\LQJDQGLQWHJUDWLQJ

&276WKHVHFRQWUDFWRUVVKRXOGKDYHDEHWWHUKDQGOHRQGHYHORSPHQWFRVWVWKDQLQWKH

GHVLJQDQGEXLOGHQYLURQPHQW

8VLQJWKH5LJKW7RROV

7KHIROORZLQJDUHVRPHRIWKHWRROVUHTXLUHGWRVXFFHVVIXOO\LPSOHPHQW&276LQWHQVLYH

V\VWHPV

• 7RWDO2ZQHUVKLS&RVW72&PRGHOVDUHUHTXLUHGWRHYDOXDWHSURSRVDOVWRXVH

&2767KHVHPRGHOVSUHVHQWO\DUHEHLQJHYROYHG

• 6\VWHPVWRPDQDJHREVROHWHSDUWV7KHUHH[LVWH[FHOOHQW&276SURGXFWVWRKHOS

PDQDJHLVVXHVRIGLPLQLVKLQJPDQXIDFWXULQJLHVRXUFHVDQGREVROHWHSDUWV

'HPDQGOLNHO\ZLOOFDXVHWKHVH&276WRFRQWLQXHWRHYROYH

• 3URFHVVHVPXVWEHZULWWHQ$QDSSURSULDWHVWDUWLQJSRLQWIRUWKLVWDVNLVWKH

JXLGHOLQHVSURGXFHGE\WKHVWXG\JURXS3URFHVVZULWHUVVKRXOGSD\SDUWLFXODU

DWWHQWLRQWRWKHSLWIDOOVLGHQWLILHGE\WKHJURXS

• 7UDLQLQJLVHVVHQWLDO6(,KDVEHHQZRUNLQJRQWUDLQLQJPRGXOHVIRUVHYHUDO\HDUV

7KH\KDYHVRPHYHU\JRRGRQHVWKDWDUHGHVLJQHGIRUSDUWLFXODUDXGLHQFHV

• 0DQDJHUVDWDOOOHYHOVPXVWUHGHILQHUROHVDQGUHVSRQVLELOLWLHV
53

• (IIHFWLYHLQFHQWLYHVPXVWEHHVWDEOLVKHG,QWKHYLHZRIWKHVWXG\JURXSWKH$LU

)RUFHIRUH[DPSOHKDVQRWXVHGLQFHQWLYHVHIIHFWLYHO\
DQGE\LQFHQWLYHVWKH

UHSRUWLQGLFDWHVWRXJKFRQWUDFWXDOWHUPVWKDWHQVXUHWKHFRQWUDFWRUSD\VLI

VXVWDLQPHQWFRVWVDUHQRWFRQWDLQHG,WLVJRLQJWRWDNHWRXJKLQFHQWLYHVWRJHW

SHRSOH
VDWWHQWLRQLQVXFKDZD\WKDWHYHU\RQHJHWVRQERDUGVXSSRUWLQJWKH&276

HQYLURQPHQW6RPHH[DPSOHVRIHIIHFWLYHLQFHQWLYL]LQJDUHWKH0DWHULDO,QVSHFWLRQ

DQG5HFHLYLQJ5HSRUW''DQGWKHQWKH1DY\
VVXEPDULQHSURJUDPILYH\HDUV

ODWHU$QRWKHUH[DPSOHLVWKH-'$0ZDUUDQW\WKDWSXWWKHFRQWUDFWRURQWKHKRRN

IRU\HDUV

• 7KHDFTXLVLWLRQFRPPXQLW\PXVWHVWDEOLVKSROLF\WKDWZLOOEHFRPHDIUDPHZRUNIRU

&276VXFFHVV2QFHDIUDPHZRUNLVLQSODFHWKHDFTXLVLWLRQFRPPXQLW\FDQGULYH

LQGXVWU\WRVXSSRUWZKDW'R'ZDQWVWRGRZLWK&276PXFKIDVWHUWKDQ'R'KDVLQ

WKHSDVW

• 3HUVRQQHOHYDOXDWLRQWHFKQLTXHVPXVWEHGHYHORSHGVSHFLILFDOO\WRLGHQWLI\WKHVNLOOV

DQGH[SHULHQFHWKHDFTXLVLWLRQFRPPXQLW\ZDQWVSHRSOHWREULQJWR&276LQWHQVLYH

V\VWHPV

• (GXFDWLRQDQGWUDLQLQJRQ&276V\VWHPVVKRXOGEHLQFRUSRUDWHGLQWRWKH'HIHQVH

6\VWHPV0DQDJHPHQW&ROOHJH'60&FXUULFXOXP

• /HVVRQVOHDUQHGPXVWEHFDSWXUHGUHJXODUO\$V&276EDVHGV\VWHPVDUHVWLOO

HYROXWLRQDU\WKHVWXG\JURXSUHFRPPHQGVSHULRGLFDWOHDVWDQQXDOUHYLHZDQGDV

QHZOHVVRQVDUHOHDUQHGUHYLVLRQ
54



$FTXLVLWLRQ6WUDWHJ\DQG&2766XFFHVV

29(59,(:

$FTXLVLWLRQODZJHQHUDOO\LVDGHTXDWHWRVXSSRUW&276DOWKRXJKWKHUHVWLOOH[LVWFRORU

RIPRQH\DQGGHSRWLVVXHV7KHVWXG\JURXSEHOLHYHVWKHVHLVVXHVFDQEHUHVROYHGLQWKH

QHDUWHUP

&276LPSOHPHQWDWLRQSROLF\PXVWGULYHWKHDFTXLVLWLRQVWUDWHJ\LQFOXGLQJVXEVHTXHQW

VRXUFHVHOHFWLRQDQGSURJUDPPDQDJHPHQW

)LJ&276DQG$FTXLVLWLRQ6WUDWHJ\

Key Elements of COTS Policy

Requirements Tools

• Flexible Req’ts & Specs

• Open System Architecture &

Spiral Dev & Test

• Incentivize Contractor to

address support

• Cost criterion: Lowest TOC

• Assess Contractor’s COTS exp

• Contractor COTS Mgt process

(Assess, Select, Refresh, etc)

• Analysis of COTS<=>Custom Apps

• Understand Marketplace

• Application Matches COTS

• No Mod of COTS

• Trade everything

• Collocated IPT

Education &

Training

Policy

• Annual review of COTS experience

• Web site for COTS-based systems information

Recommendation: Prepare & implement a policy for acquisition and

sustainment of COTS-based systems

Systems

• Dev COTS TOC Model

• Dev contractor/SPO

evaluation techniques

• Consortia Participation

Review

Acq Strategy Source

Selection

Program

Mgt
55

$&48,6,7,21675$7(*<$1')/(;,%/(5(48,5(0(176

7KHVWXG\JURXSUHSHDWHGO\HPSKDVL]HVKRZLPSRUWDQWIOH[LEOHUHTXLUHPHQWVDUHWR

&276V\VWHPVXFFHVV7KLVIDFWPXVWEHLQFRUSRUDWHGLQWRDFTXLVLWLRQVWUDWHJ\

6WUDWHJLVWVDQGWKRVHZKRUHYLHZWKHVWUDWHJ\PXVWDVNDUHWKHUHTXLUHPHQWV

DSSURSULDWHO\IOH[LEOH":KLFKUHTXLUHPHQWVLIDQ\DUHQHJRWLDEOH"

7KHFUXFLDOSRLQWLVWRDVNWKLVTXHVWLRQWRFRQGXFWWKLVUHYLHZ:LWKRXWWKLVW\SHRI

UHYLHZVRPHRQHPLJKWSXOODUHTXLUHPHQWRXWRIERLOHUSODWHVRPHZKHUHIDLOWRWDLORULW

WRWKHWDVNDQGHQGXSGULYLQJWKHSURMHFWDZD\IURPDSSURSULDWH&276XWLOL]DWLRQ

$&48,6,7,21675$7(*<$1',1'8675<$1'86(5

,192/9(0(17

7RPDNHD&276V\VWHPHIIHFWLYHWKHDFTXLVLWLRQVWUDWHJ\DOVRPXVWLQYROYHLQGXVWU\

,QGXVWU\LQYROYHPHQWPXVWWDNHSODFHLQWKHFRQFHSWXDOSODQQLQJSKDVHLQFRQFHUWZLWK

XVHUV
LQRWKHUZRUGVEHIRUHWKH2SHUDWLRQDO5HTXLUHPHQWV'RFXPHQW25'DQGWKH

5HTXHVWIRU'HYLDWLRQ5)'

7KHXVHUVEULQJWRWKHVWUDWHJ\WDEOHWKHLUQHHGDQGWKHLULGHDVDQGFRQFHSWVRQKRZWR

PHHWWKDWQHHG,QGXVWU\FDQEULQJWRWKHWDEOHLQIRUPDWLRQRQKRZWKDWQHHGFDQEH

DGGUHVVHGE\&276

$&48,6,7,21675$7(*<$1'23(1$5&+,7(&785(

2SHQV\VWHPDUFKLWHFWXUHLVFULWLFDOWRVXFFHVVIXOLPSOHPHQWDWLRQRI&2763OXJDQG

SOD\DOORZV&276EDVHGV\VWHPVWRHYROYHZLWKWKHWHFKQRORJ\DQGVHDPOHVVO\PDNH

XSJUDGHV2QHDVSHFWRIDFTXLVLWLRQVWUDWHJ\UHYLHZPXVWEHWRHQVXUHRSHQV\VWHP

DUFKLWHFWXUHLVLQSODFH

$&48,6,7,21675$7(*<$1',1&(17,9,=,1*&2175$&7256

$FTXLVLWLRQVWUDWHJ\UHYLHZPXVWLQFOXGHDQHYDOXDWLRQRILQFHQWLYHV,QSDUWLFXODUZKDW

LQFHQWLYHVH[LVWIRUWKHVXSSOLHUWRSURYLGHFUHGLEOHHVWLPDWHVRIVXVWDLQPHQWFRVWVDQG

WRFRQWDLQVXVWDLQPHQWFRVWVRQFHWKHSURGXFWLVVHOHFWHG"

$&48,6,7,21675$7(*<$1'72&

72&QRWDFTXLVLWLRQRUIO\DZD\FRVWVPXVWEHWKHVRXUFHVHOHFWLRQFULWHULD2QH

VXSSRUWWRROWKDWLVUHTXLUHGIRUVXFFHVVIXO&276LPSOHPHQWDWLRQLVWKH72&PRGHO

0RGHOVDUHFRPPHUFLDOO\DYDLODEOH

72&PXVWGHWHUPLQHWKHVXLWDELOLW\RI&2763HUKDSVDFXVWRPGHVLJQLVDSSURSULDWH

%XWWKLVGHFLVLRQVKRXOGEHPDGHEDVHGRQWUDGHDQDO\VLVQRWE\HGLFW
56

3URJUDPPDQDJHUV30VPXVWVXSSRUW&276E\FRQGXFWLQJDSSURSULDWHWUDGHRIIVWKDW

LPSDFW72&30VDOVRPXVWHQIRUFHRQJRLQJLQWHUDFWLRQEHWZHHQWKHJRYHUQPHQWDQG

WKHFRQWUDFWRUWKURXJK,37VSUHIHUDEO\FROORFDWHG,37FROORFDWLRQFDQEHGLIILFXOWDW

WKHPLQLPXP,37VVKRXOGPHHWLQFRPPRQIDFLOLWLHVZKHQWKH\JHWWRJHWKHU

)LJ$FTXLVLWLRQ6WUDWHJ\DQG7RWDO2ZQHUVKLS&RVW

$&48,6,7,21675$7(*<$1'6285&(6(/(&7,21

$FTXLVLWLRQVWUDWHJLVWVPXVWGHYHORSWHFKQLTXHVWRHYDOXDWHWKHSRWHQWLDOFRQWUDFWRUDQG

WKHJRYHUQPHQWSURJUDPRIILFHRQWKHLUDELOLW\WRGHYHORSDQGPDQDJHDWRXJK&276

LQWHQVLYHV\VWHP

&RQWUDFWRU([SHULHQFHZLWK&276

7KHDFTXLVLWLRQVWUDWHJ\UHYLHZPXVWDVVHVVWKHFRQWUDFWRU
VSDVWH[SHULHQFHZLWK

&276,Q&276V\VWHPVWKHUHLVDYHU\VWURQJFRUUHODWLRQEHWZHHQH[SHULHQFH

LQFOXGLQJORQJHYLW\DQGVXFFHVV

$VZLWKDQ\QHZV\VWHPLWPD\EHWKDWLQLWLDOO\QRWWKDWPDQ\FRQWUDFWRUVKDYH

H[SHULHQFHLQIDFWHYHU\ELGGHUPLJKWEHHOLPLQDWHGRQWKDWJURXQG1HYHUWKHOHVV

PDNLQJH[SHULHQFHDIDFWRULQDFTXLVLWLRQVWUDWHJ\WHOOVSRWHQWLDOELGGHUVWKDW'R'LV

VHULRXVDERXW&276%LGGHUVZLOOILJXUHRXWDZD\WRREWDLQH[SHULHQFHDQGWR

GHPRQVWUDWHWKDWWKH\KDYHSURFHVVHVLQSODFHWREHVXFFHVVIXO

Key Elements of COTS Policy -

The Program Management

• Conduct trade-off analyses of all changes vs TOC

• Enforce ongoing interaction between government personnel (ops &

acquisition) and prime contractor in Integrated Product Teams

− Facilitates trades between requirements & TOC

− Collocate is preferable
57

&RQWUDFWRU3URFHVVHV

)RU&276WKHVRXUFHVHOHFWLRQHYDOXDWLRQPXVWGHWHUPLQHLISURFHVVHVDUHLQSODFHIRU

VHOHFWLQJLQWHJUDWLQJVXSSRUWLQJDQGUHIUHVKLQJ&276

&RQWUDFWRU8QGHUVWDQGLQJRIWKH&RPPHUFLDO0DUNHWSODFH

7KHJRYHUQPHQWPXVWHYDOXDWHZKHWKHUFRQWUDFWRUVXQGHUVWDQGWKHFRPPHUFLDO

PDUNHWSODFH:KDWLVWKHFRQWUDFWRUGRLQJIRUH[DPSOHLQWKHDUHDRI5	'"7KLV

HYDOXDWLRQLQFLGHQWDOO\LVH[DFWO\ZKDWKDSSHQVZKHQFRPPHUFLDOFRPSDQLHVHYDOXDWH

SRWHQWLDOVXSSOLHUV

$QHOHFWURQLFVFRPSDQ\IRUH[DPSOHPD\JRLQWRDSRWHQWLDOVXSSOLHU
VVHPL

FRQGXFWRURSHUDWLRQDQGVFUXELWIRUWZRZHHNV,WLVQRWHQRXJKIRUWKHHOHFWURQLFV

FRPSDQ\WRHYDOXDWHWKHSURGXFW7KH\ZDQWWRNQRZZKDWSURFHVVHVDUHLQSODFH"&DQ

WKHVHPLFRQGXFWRUGHSDUWPHQWPDNHLWLQWKHPDQXIDFWXULQJUDFH"$UHWKHHTXLSPHQW

DQGSHUVRQQHOVXIILFLHQWDQGXSWRVSHHG"

7KHVWXG\JURXSVWURQJO\FRQFOXGHGWKDWWKHDFTXLVLWLRQZRUNIRUFHQRORQJHUFDQOHWWKH

SURGXFWGRDOOWKHWDONLQJ5DWKHUDFTXLVLWLRQSHUVRQQHOPXVWGLJLQWRWKHVXSSOLHU
V

ZKROHVLWXDWLRQ7RRPDQ\WLPHV&276KDYHIDLOHGEHFDXVHQRRQHORRNHGEHKLQGWKH

SURGXFWIRUH[DPSOHWRWKHSURFHVVHVRUSHUVRQQHORU5	'GHSDUWPHQW

)LJ5ROHRIWKH&RQWUDFWRULQ&276,PSOHPHQWDWLRQ6XFFHVV

$&48,6,7,21675$7(*<$1'352'8&76(/(&7,21

)XQFWLRQDOO\&276SURGXFWVPXVWILWUHTXLUHPHQWVIDLUO\FORVHO\7RPDNHWKDWFDOO

GXULQJWKHDFTXLVLWLRQVWUDWHJ\UHYLHZWKHDFTXLVLWLRQFRPPXQLW\DQGRWKHUVPXVW

EHFRPHYHU\IDPLOLDUZLWKWKHVHSURGXFWV

Key Elements of COTS Policy -

Source Selection Evaluation

• Evaluate contractor’s COTS processes for assessing, selecting, integrating,

supporting and refreshing

• Use TOC to determine suitability of COTS vs. custom products

• Assess contractor’s understanding of the commercial marketplace and

relevant COTS products

• Ensure system application matches the COTS product functionality

• Verify contractor proposes to use COTS products without modification
58

7RIDFLOLWDWHSURGXFWNQRZOHGJH'R'VKRXOGVHHNFRPPRQDOLW\DFURVVV\VWHPV

%HFDXVHRIWKHHFRQRPLHV
NQRZOHGJHDQGRWKHUZLVH
EHVWSUDFWLFHFRPSDQLHVDOVRDUH

ZRUNLQJWRHQKDQFHFRPPRQDOLWLHVDFURVVSURGXFWOLQHV

$&48,6,7,21675$7(*<$1'&27602',),&$7,21

7KHVWXG\JURXSPDGHDVWURQJVWDWHPHQWWKDWPRGLILFDWLRQVRI&276DUHLQDGYLVDEOH

$JDLQLIDPRGLILFDWLRQLVSURSRVHGLWLVFULWLFDOWKDWHYHU\RQHXQGHUVWDQGDOOWKH

LPSOLFDWLRQV

VOLGH

)LJ6WUDWHJLF*RDOVIRU&276,PSOHPHQWDWLRQ6XFFHVV

Key Elements of COTS Policy -

Acquisition Strategy Review

• All operational requirements and procurement specifications should be

negotiable

− Focus on the output rather than the input

− Involve industry before ORD and RFP are finalized

• Open system architecture & spiral development essential

• Incentivize prime contractor to provide a credible estimate of support costs

• Use Total Ownership Cost as source selection cost criterion

• Assess the contractor’s past experience employing COTS products
59



7KH7UDLQLQJ,PSHUDWLYH

,1)250$7,216+$5,1*

$VDEDFNGURSWRIRUPDOWUDLQLQJSURJUDPVWKHVWXG\JURXSUHFRPPHQGVSDUWLFLSDWLRQ

LQFRQVRUWLDDVDZD\WREHFRPHHGXFDWHGRQERWKVRIWZDUHDQGKDUGZDUH&276

&RPSXWHU$LGHG/LIH&\FOH(YDOXDWLRQ&$/&(IRUH[DPSOHDWWKH8QLYHUVLW\RI

0DU\ODQGLVGRLQJDORWRIJRRGZRUNLQREWDLQLQJJRYHUQPHQWIXQGLQJ

$QRWKHUFRQVRUWLXPLVWKH,QWHURSHUDELOLW\&OHDULQJKRXVH7KH&OHDULQJKRXVHLVMXVW

JHWWLQJRIIWKHJURXQGVRWKHUHLVQRWPXFKLQIRUPDWLRQDERXWLW3UHVXPDEO\WKH

&OHDULQJKRXVHZLOOEHDUHSRVLWRU\RIH[SHULHQFHVZLWK&276,GHDOO\LIVRPHRQH

SURSRVHVD&276DFTXLVLWLRQPDQDJHUVVKRXOGEHDEOHWRFKHFNZLWKWKH&OHDULQJKRXVH

WRVHHKRZWKDWSURGXFWKDVZRUNHGIRURWKHUVZKDWSUREOHPVDURVHZKDWTXHVWLRQV

VKRXOGEHDVNHG

)LJ7DVNVWR6XSSRUW&2767UDLQLQJ

7+(&27675$,1,1*,03(5$7,9(

'R'PXVWFRQGXFWDVXEVWDQWLDODPRXQWRIWUDLQLQJWRVXSSRUWHIIHFWLYHLPSOHPHQWDWLRQ

RI&276V\VWHPV

7KHVWXG\JURXSHQFRXUDJHG'60&WKH$UPHG)RUFHV,QVWLWXWHRI7HFKQRORJ\

$),7DQGRWKHUVWRLQFRUSRUDWH&276V\VWHPVLQWRWKHFXUULFXOXP6(,DOUHDG\KDV

GHYHORSHGDQXPEHURIWUDLQLQJPRGXOHV

7KHUHPXVWEHDSURFHVVRIFHUWLILFDWLRQDWDOOOHYHOV

Key Elements of COTS Policy -

 Source Selection Support Tools

• Develop TOC analysis Model/Tools with COTS cost factors

− Refresh, rewrite procedures, retrain, etc.

• Develop evaluation techniques that focus on contractor & program office

capability to produce COTS-Based systems

• Participate in consortia that share COTS product experiences

− CALCE for electronic components

− Consider Interoperability Clearinghouse for software, based on progress

assessment of this emerging organization
60

)LJ&2767UDLQLQJ*RDOV

7UDLQLQJ(PSKDVLV

7RVXSSRUWWKHWUDQVLWLRQWR&276LQWHQVLYHV\VWHPVHGXFDWLRQPXVWWDNHRQDQ

HPSKDVLVGLIIHUHQWIURPLQWKHSDVWLWPXVWWHDFKSHRSOHKRZWRDVNWKHFRUUHFW

TXHVWLRQV

7KHFKDOOHQJHLQ&276HGXFDWLRQLVWRWUDQVODWHWKHLPSHUDWLYH
WHDFKSHRSOHWRDVNWKH

FRUUHFWTXHVWLRQV
LQWRDFXUULFXOXPWKDWIRFXVHVRQKRZWRUDWKHUWKDQWKHWKHRU\RI

&276V\VWHPV

$KRZWRIRFXVHQFRPSDVVHVWKHZD\SHRSOHWKLQNDQGEHKDYH,WLVQRWVRPHWKLQJ

WKDWFDQEHDFFRPSOLVKHGIRUH[DPSOHLQDWZRGD\EULHILQJ7KHUHPXVWEHVXIILFLHQW

WLPHDQGRSSRUWXQLW\IRUSHRSOHWRGRH[HUFLVHVDQGLQWHUQDOL]HWKHDQDO\WLFDOSURFHVV

 7KH5ROHRIWKH%XUQLQJ3ODWIRUP

3DUWRIWKHNQRZOHGJHRIKRZWRDVNWKHFRUUHFWTXHVWLRQVFRPHVIURPWUDLQLQJ%XW

DQRWKHUSDUWRIWKLVNQRZOHGJHLVVSXUUHGE\DEXUQLQJSODWIRUP
SHRSOHPXVWSHUFHLYH

WKDWWKH\PXVWJHWZLWKWKHSURJUDPRUWKH\ZLOOEHOHIWEHKLQG2QO\WKHQZLOOWUDLQLQJ

WXUQSHRSOHIURPWKHLQWHOOHFWXDOO\FXULRXVLQWRUHDOZRUOG&276V\VWHPDQDO\VWV

75$,1,1*$1'5(6285&($//2&$7,21

7LPH
ODFNRILWRIFRXUVH
RIWHQLVWKHSUREOHPLQFRQGXFWLQJDGHTXDWHWUDLQLQJ

7KHUHDUHPDQ\SUHVVLQJWUDLQLQJQHHGVWKDWPXVWEHDGGUHVVHGIRUH[DPSOHLQD

ZHHNORQJFXUULFXOXP+RZGRZHSULRULWL]HWUDLQLQJ"+RZGRZHMXVWLI\WKHDPRXQWRI

WLPHZHUHTXLUHSHUVRQQHOWRVSHQGDZD\IURPWKHLUSURJUDPVDQGLQWUDLQLQJ"

Key Elements of COTS Policy -

Education and Training

• Increase COTS competency at all levels for workforce

− Identify audiences, required skills, knowledge and abilities

− Develop required education and training courses

− Require training at front end of programs assigned to Program

Executive Officer (PEO) and Designated Acquisition

Commander (DAC)

• Encourage DSMC, AFIT to incorporate COTS topics into their

curriculum

• Include COTS material in contracting officer and acquisition officer

certification programs

• Make COTS guidelines available via Defense Acquisition Deskbook
61

1LQHW\SHUFHQWRISURJUDPVDUHGHILFLHQWLQKRZWKH\PDQDJHVRIWZDUH'R'PXVWGR

PRUHWUDLQLQJRQVRIWZDUHPDQDJHPHQW,VWKDWWKHELJJHVWQHHGIRUWKHSHUVRQJRLQJ

EDFNWRWKHILHOG"2ULV&276DKLJKHUSULRULW\"

7KLVGLOHPPDLOOXVWUDWHVKRZFULWLFDOLWLVWKDWWKH&276WUDLQLQJWKDWLVFRQGXFWHGEH

HIIHFWLYH'R'FDQQRWDIIRUGWRSXWWLPHLQWRWUDLQLQJWKDWGRHVQ
WJHWWKHMREGRQH

WKDWZKHQLW
VRYHUWKHWUDLQLQJHQGVXSLQWKHWKDWZDVLQWHUHVWLQJEXWQRWSUDFWLFDO

ILOH

6758&785,1*&27675$,1,1*

7KHPDMRULW\RI'R'HGXFDWLRQDOUHTXLUHPHQWVDUHGHWHUPLQHGE\IXQFWLRQDODFTXLVLWLRQ

ERDUGV&276KRZHYHUDUHQRWZLWKLQWKHSXUYLHZRIDGHIHQVHIXQFWLRQDODFTXLVLWLRQ

ERDUG7KLVODFNRIERDUGLQSXWPDNHVLWGLIILFXOWWRNQRZKRZWRVWUXFWXUH&276

HGXFDWLRQ'R'PXVWUHYLHZWUDLQLQJDFURVVDOOWHQIXQFWLRQDODUHDV

• :KDWNQRZOHGJHGRHV'R'QHHGVLWVSHRSOHWRDFTXLUHWRVXFFHVVIXOO\LPSOHPHQW

&276"

• +RZGRZHPHDVXUHZKHQSHRSOHKDYHDFTXLUHGVXIILFLHQW&276NQRZOHGJH"

• 'RHVWKHWUDLQLQJOLQNDQGLQWHJUDWHDOOWKHFRXUVHVLQDZD\WKDWSURYLGHVWKH

DFTXLVLWLRQZRUNIRUFHZLWKDFRPSOHWHWRRONLW"

3$6775$,1,1*$1',1)250$7,216+$5,1*,1,7,$7,9(6

,QDQDFTXLVLWLRQFRXQFLOLQWKH3HQWDJRQFUHDWHGDFRPSUHKHQVLYHFRUHERG\RI

NQRZOHGJHRQVLPXODWLRQ7KHLQWHQWZDVWRJLYHWKLVFRUHERG\RINQRZOHGJHWRHDFK

IXQFWLRQDOERDUG(DFKERDUGWKHQZDVWRUHYLHZKRZWKDWFRUHERG\RINQRZOHGJH

VKRXOGEHLQFRUSRUDWHGLQWRWKHLUSLHFHRIWKHSX]]OH:KLOHWKDWJRDOZDVQHYHU

UHDFKHGWKDWLVWKHVRUWRISURFHVVQHHGHGIRU&276

75$,1,1*0$7(5,$/6

&276JXLGHOLQHVPXVWEHDYDLODEOH7KH6'KDQGERRN
'R'6WDQGDUGL]DWLRQ

'LUHFWRU\DKDQGERRNIRUEX\LQJFRPPHUFLDO	1',V
FRYHUV&276DQG1',

DFTXLVLWLRQ,WKDVEHHQRXWTXLWHDZKLOH%XWWKHDFTXLVLWLRQFRPPXQLW\PXVWJR

EH\RQGH[LVWLQJJXLGHOLQHVWRORRNDWWKHEHVWSUDFWLFHFRPSDQLHVWDNHWKHLUH[SHULHQFH

DQGXVHLWWRJHWDEHWWHUXQGHUVWDQGLQJRIKRZWR

:KLOHDORWRILQIRUPDWLRQRQ&276H[LVWVUHODWLYHO\OLWWOHRILWLVTXDOLW\JXLGDQFH$

JRRGSODFHWRVWDUWLVWKH6(,ZKLWHSDSHU'R'WUDLQHUVFRXOGWDNHVRPHWKLQJOLNHWKDW

DQGXSJUDGHLWZLWKVRPHJRRGFDVHVWXGLHV
62

)LJ&2767UDLQLQJ0DWHULDOV&RQWHQW

75$,1,1*0867%(&21'8&7(':,7+,1$6833257,9(32/,&<

,1)5$6758&785(

'R'PXVWHQVXUHWKDWLWVSROLF\VWUXFWXUHEDFNVXSWKHV\VWHPVLWLVWU\LQJWR

LPSOHPHQW

1LQHW\SHUFHQWRIWKHWLPHZKHQWKHGHVLUHGUHVXOWVDUHQRWEHLQJDFKLHYHGLWLVEHFDXVH

HLWKHUWKHUHLVDVWUXFWXUDOOLPLWDWLRQRUSHRSOHKDYHQRWEHHQSURYLGHGWKHULJKWWRROVWR

DFFRPSOLVKWKHFKDQJH7KXV&276JXLGHOLQHVDQG&276WUDLQLQJDUHQRWWKHZKROH

SLFWXUH'R'PXVWHQVXUHWKDWWKRVHJXLGHOLQHVDUHRSHUDWLQJDQGWKDW&276WUDLQLQJLV

EHLQJFRQGXFWHGZLWKLQDVXSSRUWLYHSROLF\LQIUDVWUXFWXUH

 &DVHVWXG\3HUIRUPDQFH%DVHG6HUYLFH&RQWUDFWLQJ

3HUIRUPDQFHEDVHGVHUYLFHVFRQWUDFWLQJKDVEHHQSXVKHGVLQFH,WLVQRWDGLIILFXOW

VW\OHRIFRQWUDFWLQJEXWWKHSUREOHPZDVWKDWWKHLQIUDVWUXFWXUHDQGSROLF\VWUXFWXUH

ZDVVHWXSWRGRVRPHWKLQJHOVH7KHUHJXODWLRQVKDGQRWEHHQFKDQJHGWRVXSSRUW

SHUIRUPDQFHEDVHGFRQWUDFWLQJ1RPDWWHUZKDWV\VWHPZHVD\ZHZDQWWRVHH

LPSOHPHQWHGLWLVUHJXODWLRQVWKDWGHWHUPLQHZKDWKDSSHQVLQWKHILHOG

 &DVH6WXG\5HYLVLRQRI0DMRU$FTXLVLWLRQ'LUHFWLYHV

7KH'R'LVLQWKHSURFHVVRIUHYLVLQJLWV0DMRU$FTXLVLWLRQ'LUHFWLYHVDQG

57KHGLUHFWLYHVSUHVHQWO\DUHLQGUDIWIRUP

7KLVLVDQH[DPSOHRIZKHUHWKHSROLF\LQIUDVWUXFWXUHPXVWVXSSRUW&276LI&276LVWR

EHVXFFHVVIXO6RPHSHRSOHKRZHYHUDOUHDG\VHHDQLQWXLWLYHGLVFRQQHFWEHWZHHQ

ZKLFKSXWVDSULRULW\RQ&276DQG5ZKLFKVHWVRXWWKHWUDGLWLRQDO

GHFLVLRQPDNLQJKLHUDUFK\DQGWKXVSXWVDSULRULW\RQPRGLILFDWLRQVDQGXSJUDGHV

EHIRUHFRQVLGHULQJ&2767KHGLUHFWLYHVPXVWPDNHWKHSULRULWLHVFOHDU

Terms of Reference

• Case studies

• Identify existing processes and standards

• Provide guidelines

− Applicability

− Processes and standards

− Field support

− Metrics

− Government oversight

− Security
63

75$,1,1*0867,1&25325$7(/(66216/($51('

7KHVWXG\JURXSUHFRPPHQGHGSHULRGLFDWOHDVWDQQXDOUHYLHZDQGDVDSSURSULDWH

UHYLVLRQFRQVLVWHQWZLWKQHZOHVVRQVOHDUQHGRQ&276LQWHQVLYHV\VWHPV

7RWKDWHQGWKH1DY\KDVDZHEVLWHZKHUHWKH\FROOHFWLQIRUPDWLRQRQ&276

$FTXLVLWLRQ5HYLHZKDVDZHEIRUXPIRUFRPPHUFLDODGYRFDWHV

7KHNH\SRLQWLVWKDW'R'PXVWHVWDEOLVKUHSRVLWRULHVWRVXSSRUWWKH&276OHDUQLQJ

FXUYH

)LJ&2767UDLQLQJ3HULRGLF5HYLHZDQG,QIRUPDWLRQ6KDULQJ

Key Elements of COTS Policy -

 Review

• Sponsor annual review of COTS experience on AF systems

− Focus on experiences and lessons learned; target

policy /infrastructure improvements, updates to

education & training

• Sponsor web site for COTS-based systems information

(lessons learned, case studies, references, etc.)
64



&RQFOXVLRQV

6800$5<

7KHVWXG\JURXS
VRYHUDUFKLQJFRQFOXVLRQZDVWKDW&276LVUDGLFDOO\GLIIHUHQWIURP

WUDGLWLRQDODFTXLVLWLRQLQHYHU\GLPHQVLRQ7KLVUDGLFDOGLIIHUHQFHLVQRWZHOO

DSSUHFLDWHGDQGQHLWKHUDUHWKHFRPSOH[LWLHVRI&276V\VWHPV7KHVHDUHWKHSHUYDVLYH

UHDVRQVWKDWSURJUHVVWRZDUG&276LPSOHPHQWDWLRQLVVORZ,WDOVRLVZK\SHRSOHDUH

VWUXJJOLQJ

)LJ6XPPDU\'LIIHUHQFHV%HWZHHQ0LO6SHFDQG&276

7+(52$'72&27668&&(66,6/21*

7KHUHLVDGLUHFWFRUUHODWLRQEHWZHHQORQJHYLW\LQ&276LQWHQVLYHV\VWHPVDQGVXFFHVV

ZLWK&2767KLVLVEHFDXVH&276V\VWHPVDUHGLIILFXOWWRLPSOHPHQW0RUHRYHU

&276DUHQRWDSDQDFHD7KDWLQLWVHOILVDVXUSULVHWRVRPH

1HYHUWKHOHVVWKHFRQVHQVXVYLHZLVWKDWWKHWUDQVLWLRQIURPROGGHVLJQDQGEXLOG

SUDFWLFHVWRQHZEX\DQGLQWHJUDWHSUDFWLFHVLVJRLQJWRKDSSHQZKHWKHU'R'OLNHVLWRU

QRW7KHNH\LVWRPDNHWKHWUDQVLWLRQKDSSHQPRUHTXLFNO\DQGWRDYRLGRUDWOHDVW

PLQLPL]HDOOWKHSDLQDQGDJRQ\

COTS is Radically Different

Mil Spec COTS

• Requirements driven Market driven

• Spec focus Business plan focus

• Rigid requirements Flexible requirements

• Unique architecture Open system architecture

• Owner controls evolution Market controls evolution

• Stable design Constant changes

• Ignore evolution Design for evolution (tech refresh)

• Cost emphasis TOC emphasis

• Make custom hardware Buy from catalog

• Develop software License software

• Obsolescence Earlier obsolescence

• Waterfall-style development Spiral development
65

)LJ&RQFOXVLRQVRIWKH&2766WXG\*URXS

“Every aspect of acquisition planning, system engineering processes, test planning,

etc. must be explicitly crafted to account for COTS issues” - Dr. Mike Borky

Embed

in

Policy

Conclusion

• COTS is very difficult to implement

• COTS is not a panacea

• COTS is inevitable

• To be successful

− Everyone needs training

− New processes need to be established

− Roles and responsibilities need to be

redefined

− Contractor

− Government
66

*ORVVDU\

A

$FFRXQWLQJ&276  

$FTXLVLWLRQVWUDWHJ\

)OH[LEOHUHTXLUHPHQWV 

,QGXVWU\LQYROYHPHQW  

2SHQV\VWHPDUFKLWHFWXUH  

8VHULQYROYHPHQW 

$FTXLVWLRQ&DWHJRU\,$&$7SURJUDPV  

$GYDQFHG$PSKLELRXV$VVDXOW9HKLFOH$$$9  

$LU)RUFH  

$LU)RUFH6FLHQWLILF$GYLVRU\%RDUG  

$LUERUQH:DUQLQJDQG&RQWURO6\VWHP$:$&6  

$SSOLFDELOLW\&276  

$UPHG)RUFHV,QVWLWXWHRI7HFKQRORJ\$),7  

$UP\  

B

%HQHILWV&276  

%XVVHV 

C

&,  

&DVHVWXGLHV

$LUERUQH:DUQLQJDQG&RQWURO6\VWHP$:$&6  

)SURJUDP  

-RLQW'LUHFW$WWDFN0XQLWLRQV-'$0 

/RFNKHHG
VVSLUDOSURFHVV 

0DMRU$FTXLVLWLRQ'LUHFWLYHV 

0DQPDGHQRLVHGHWHFWLRQ  

1DY\DWWDFNVXEPDULQH  

3HUIRUPDQFHEDVHGVHUYLFHVFRQWUDFWLQJ 

3ODVWLFHQFDSVXODWHGPLFURHOHFWURQLFV 

&KDQJHF\FOH 

$UWLILFLDOO\TXLFNHQHG 

&RQWURO  

&RVW  

(QGXVHU 

+XPDQVNLOOVDFTXLVLWLRQ  

0DUNHWUDWH  
67

2SHQV\VWHPDUFKLWHFWXUH  

2SHUDWLRQDOUHDGLQHVV  

3DFH  

6FRSH  

6RIWZDUH 

6\VWHPGHVLJQ  

&RPPHUFLDO2IIWKH6KHOI3URGXFWV&276 

&RPPHUFLDOSUDFWLFHV  

&RPPRQDOLW\SURGXFW  

&RPSXWHUV

$LUERUQH:DUQLQJDQG&RQWURO6\VWHP$:$&6  

3HQWLXPSURFHVVRUV 

&RPSXWHU$LGHG/LIH&\FOH(YDOXDWLRQ&$/&(  

&RQILJXUDWLRQPDQDJHPHQW 

&RQJUHVVLRQDOEXGJHWSURFHVV  

&RVW

$FTXLVLWLRQ  

'HYHORSPHQW  

)O\DZD\  

,QLWLDOUHFXUULQJ  

/RJLVWLFV  

3URGXFWLQWHJUDWLRQ  

6XVWDLQPHQW 

7HFKQRORJ\FKDQJH 

&RVWHIIHFWLYHQHVV&276  

&XVWRPGHVLJQ 

D

'HEXJJLQJ 

'HFLVLRQSURFHVV&276 

7LPHWDEOH  

Defense Systems Management College (DSMC) .......................................................53, 59

Department of Defense (DoD) ............................................................................................1

'HSRW 

'HSRW0DLQWHQDQFH0DQDJHPHQW,QIRUPDWLRQ6\VWHP'00,6  

'HVLJQSKDVH&276  

'HYHORSPHQW

&RVW  

5LVN  

7LPH  

'R'6WDQGDUGL]DWLRQ'LUHFWRU\6'+DQGERRN 
68

E

(OHFWURQLFV 6HH3ODVWLFHQFDSVXODWHGPLFURHOHFWURQLFV

(QYLURQPHQWDOFULWHULD

$LUERUQH:DUQLQJDQG&RQWURO6\VWHP$:$&6  

&DSWLYHIOLJKWERPE  

*IRUFH  

+XPLGLW\RSHUDWLQJ  

2YHUVSHFLILFDWLRQ  

6DOWVSUD\  

6WDIILQJ  

7HPSHUDWXUH  

7HVWLQJ  

9LEUDWLRQ  

(YROXWLRQSURGXFW

%XGJHWF\FOHV  

&RQWURO  

)XQGLQJ  

0RGLILFDWLRQ 

5RDGPDSV  

9DOXH  

:URQJZD\  

F

)SURJUDP  

)HGHUDO$FTXLVLWLRQV6WUHDPOLQLQJ$FW)$6$  

)HGHUDWHGV\VWHPV  

)XQFWLRQDOLW\  

)XQGLQJ

%XGJHWF\FOHV  

&RORURIPRQH\  

(YROXWLRQSURGXFW  

/LIHF\FOH  

H

+DUGZDUH  

+LVWRULFDOFRQWH[W&276 

+XJKHV(OHFWURQLFV 

I

,QFHQWLYL]DWLRQ  

*RDO 
69

,QWHJUDWHG3URGXFW7HDPV,37 

,QWHQVLW\&276  

,QWHUQDOXVHSURGXFWV  

,QWHURSHUDELOLW\  

,QWHURSHUDELOLW\&OHDULQJKRXVH 

J

-RLQW'LUHFW$WWDFN0XQLWLRQV-'$0  

L

/HDGHUVKLS  

/RFNKHHG  

/RQJHYLW\ 

M

0DMRU$FTXLVLWLRQ'LUHFWLYH

  

5  

0DQDJHPHQW,QIRUPDWLRQ6\VWHP0,6  

'HSRW0DLQWHQDQFH0DQDJHPHQW,QIRUPDWLRQ6\VWHP'00,6 

0DQGDWHV&276

0DQXIDFWXULQJ

'LPLQLVKLQJVRXUFHV  

0DQXIDFWXULQJ5HVRXUFHV3ODQQLQJ053

0DULQH&RUSV  

0LOLWDU\6SHFLILFDWLRQV0LO6SHF  

0RGHUQL]DWLRQ  

0RGLILFDWLRQ

&DVHVWXGLHV  

'HILQLWLRQ  

(YROXWLRQSURGXFW  

)DLOXUH 

*HQHUDOSULQFLSOH  

+DUGZDUH  

1RPHQFODWXUH  

3URGXFWLQWHJUDWLRQ  

6RIWZDUH 

6XSSOLHUVXSSRUW  

0RGLILHGQRQGHYHORSPHQWDOLWHPV  
70

N

1DY\  

$WWDFNVXEPDULQH  

:HEIRUXP  

1RQ'HYHORSPHQWDO,WHPV1',  

1RQGLVFORVXUHDJUHHPHQWV 

O

2SHQV\VWHPDUFKLWHFWXUH 

6HOHFWLRQSURFHVV  

2SHUDWLRQDODQG0DLQWHQDQFH2	0 

2SHUDWLRQDOUHDGLQHVV  

2SHUDWLRQDO5HTXLUHPHQWV'RFXPHQW25'  

P

3DUWVREVROHWH  

3HUIRUPDQFHFODLPV 

3HUIRUPDQFHLPSURYHPHQWV&276  

3HUVRQQHO  

(YDOXDWLRQWHFKQLTXHV 

3LWIDOOV&276 

/RJLVWLFDOYRODWLOLW\ 

3HUSHWXDOFRUSRUDWHLQYROYHPHQW  

3ODVWLFHQFDSVXODWHGPLFURHOHFWURQLFV  

3OXJDQGSOD\ 

3ROLF\LQIUDVWUXFWXUH  

3UHUHTXLVLWHV&27V  

3URFHVVHV&276 

3URGXFWIDPLOLDULW\ 

3URGXFWLQWHJUDWLRQ  

)HGHUDWHGV\VWHPV  

+LJKO\LQWHJUDWHGV\VWHPV 

0RGLILFDWLRQ 

6RIWZDUH 

7UHQG  

3URILWDELOLW\LVVXHV  

3URJUDPUHTXLUHPHQWV 

Q

4XDOLW\FRQWURO&276 

4XDOLW\UHYLHZ&276  
71

R

5HTXHVWIRU'HYLDWLRQ5)'  

5HVHDUFKDQG'HYHORSPHQW5	'  

5HVHDUFK'HYHORSPHQW7HVWDQG(YDOXDWLRQ5'7	(  

5LVNUHGXFWLRQ  

5LVNVKDULQJ 

5ROHVDQGUHVSRQVLELOLWLHV

S

6HFXULW\

6RIWZDUH 

7KLUGSDUW\DFFHVV  

6HOHFWLRQSURFHVV  

&KDQJHF\FOH  

0XOWLSOHVRXUFHV  

6RIWZDUH  6HH&KDQJHF\FOH

$SSOLFDWLRQV  

$XWRPDWHGXSJUDGHV  

'HEXJJLQJ  

'HYHORSHUFHUWLILFDWLRQ  

,QWHURSHUDELOLW\ 

0RGLILFDWLRQ 

4XDOLW\FRQWURO  

6RXUFHFRGH 

6XSSOLHUVXSSRUW  

8SJUDGH  VHHFKDQJHF\FOH

9LGHRJDPHV 

:UDSSHUV 

6RIWZDUH(QJLQHHULQJ,QVWLWXWH6(,

&HUWLILFDWLRQ  

,QWHQVLW\FDWHJRULHV 

6RXUFHVHOHFWLRQ 

6WXG\&276

*RDO 

4XDOLWDWLYHUHVXOWV  

4XDQWLWDWLYHUHVXOWV

6FRSH  

6XSSOLHUVXSSRUW 

6XVWDLQPHQWFRVW  

1DY\DWWDFNVXEPDULQH  

6XVWDLQPHQWWHDP  
72

T

7RWDO2ZQHUVKLS&RVW72&  

%HVWYDOXH  

'HILQLWLRQ  

'LVLQFHQWLYHV  

)RFXVRQ 

0RGHOV  

6RXUFHVHOHFWLRQ  

6\VWHPGHVLJQ  

7UDGHRIIV 

7UDGHVWXGLHV

7UDLQLQJ

&HUWLILFDWLRQ  

&XUULFXOXP  

(PSKDVLV 

)XQFWLRQDODFTXLVLWLRQERDUGV  

/HVVRQVOHDUQHG  

0DWHULDOV  

5HVRXUFHDOORFDWLRQ  

7UDQVLWLRQWR&276

U

8SJUDGHVDXWRPDWHG  

8VHUV  

W

:HDSRQVV\VWHPVJHQHUDO