ARMY RESEARCH LABORATORY

Vulnerability and Lethality Assessment:

The Role of Full-Up System-Level

Live-Fire Testing and Evaluation

wJ^tä^ifflSMtmmiüi

ARL-CR-450 April 2000

by Martha K. Nelson

Department of Business Administration

Franklin and Marshall College

Lancaster, Pennsylvania 17604-3003

under contract

DAAH04-96-C-0086

Approved for public release; distribution is unlimited.

DTIG QUALITY IKSKSGTED 3 20000511 023
The findings in this report are not to be construed as an official

Department of the Army position unless so designated by other

authorized documents.

Citation of manufacturer's or trade names does not constitute an

official endorsement or approval of the use thereof.

DESTRUCTION NOTICE—For classified documents, follow

the procedures in DOD 5200.22-M, Industrial Security Manual,

Section 11-19 or DOD 5200.1-R. Information Security Program

Regulation, Chapter IX. For unclassified, limited documents,

destroy by any method that will prevent disclosure of contents or

reconstruction of the document.
Army Research Laboratory

Aberdeen Proving Ground, MD 21005-5068

ARL-CR-450 April 2000

Vulnerability and Lethality Assessment:

The Role of Full-Up System-Level

Live-Fire Testing and Evaluation

by Martha K. Nelson

Department of Business Administration

Franklin and Marshall College

Lancaster, Pennsylvania 17604-3003

under contract

DAAH04-96-C-0086

Approved for public release; distribution is unlimited.
Abstract .

In this era of decreased defense budgets and limited resources, it is important for

decision-makers to determine the optimal strategy for assessing the vulnerability orlethality (V/L)

of a weapon system and the role ofFull-Up System-Level Live-Fire Test and Evaluation (FU SL

LFT&E) in that strategy. This report presents a foundation for a methodology to (1) identify,

measure, and categorize the costs and benefits of FU SL LFT&E; (2) determine the relative

significance ofFU SL LFT&E to the V/L assessment plan ofthe weapon system; and (3) compare

competing V/L assessment plans for a system.

Descriptions ofthe activities ofFU SL LFT&E, the costs and benefits (i.e., impacts) ofthose

activities, the complexities encountered in the reporting of the impacts, and approaches for

addressing the complexities are presented. A discussion of the contributions of FU SL LFT&E

to a V/L assessment strategy and suggestions for improving the cost-effectiveness of FU SL

LFT&E are included.

The Taxonomy of the V/L Analysis Process (Deitz and Ozolins 1989) is proposed as a

framework for identifying the data voids to be addressed in a V/L assessment plan. The potential

of adapting the principles ofthe Cost as an Independent Variable methodology to the evaluation

of competing plans of V/L assessment is explored.
Acknowledgments

This work was supported by the Survivability/Lethality Analysis Directorate, U.S.

Army Research Laboratory, Aberdeen Proving Ground, MD, under the auspices of the

U.S. Army Research Office Scientific Services Program administered by Battelle

(Delivery Order 31, Contract No. DAAH04-96-C-0086).

The author wishes to acknowledge the contributions made to this study by

personnel of the following organizations:

U.S. Army Research Laboratory, Survivability/Lethality Analysis Directorate.

Headquarters, U.S. Army Test and Evaluation Command.

Project Manager, Abrams Tank System.

Project Manager, Bradley Fighting Vehicle System.

Product Manager, Improved Army Tactical Missile System Block IA.

U.S. Army Materiel Systems Analysis Activity.

U.S. Army Operational Test and Evaluation Command, Evaluation Analysis

Center.

• U.S. Army Test and Evaluation Command, Aberdeen Test Center.

Special appreciation is expressed to Contracting Officer's Technical Representatives

Mr. Dennis C. Bely, Chief, Engineering Analysis Branch, Survivability/Lethality

Analysis Directorate, and Dr. Paul H. Deitz, Technical Director, U.S. Army Materiel

Systems Analysis Activity, for their considerable assistance in completing this research

effort.

in
INTENTIONALLY LEFT BLANK.

IV
Table of Contents

Page

Abstract u

Acknowledgments m

List of Figures vn

1. Introduction 1

1.1 Research Objectives 2

1.2 Research Approach 2

1.2.1 Systems 2

1.2.2 Impacts of Full-Up System-Level Live-Fire Test

and Evaluation (FUSLLFT&E) 3

1.2.3 Comparison of Competing Strategies 4

1.3 Organization of Report 4

2. Research Methodology 5

2.1 Systems 5

2.1.1 AbramsMlA2 5

2.1.2 ATACMSBlockIA 6

2.1.3 Bradley A3 8

2.2 Information Sources 10

3. Research Results 12

3.1 Activities of FUSLLFT&E 12

3.2 Costs of FUSLLFT&E 14

3.3 Costs of FUSLLFT&E: Costs of Direct Materials 16

3.3.1 Cost of Test Articles 17

3.3.2 Cost of Spare Parts 19

3.3.3 Cost of Munitions and Targets 20

3.4 Costs of FUSLLFT&E: Costs of Activities 21

3.4.1 Cost of SLAD (ARL) Activities 21

3.4.2 Cost of Test Center Activities 25

3.4.3 Cost of HQ,TECOM Activities 28

3.4.4 CostofUSAOC&SActivities 29

3.4.5 Cost of EAC Activities 29

3.5 Benefits of FUSLLFT&E 29

3.5.1 FUSLLFT&E and System V/L 30

3.5.2 FUSLLFT&E andBDAR Activities 31

3.5.3 FUSLLFT&E andM&S 32

3.5.4 FU SL LFT&E and Users'Decisions 32

3.5.5 FU SL LFT&E and Survivability/Lethality Assessment

Programs 32
3.6 Project Manager Offices: Impact of FU SL LFT&E 33

3.6.1 Objectives of FUSL LFT&E 35

3.6.2 Design of FUSL LFT&E f

3.6.3 Alternatives to FUSL LFT&E 36

3.6.4 Lessons Learned from FU SL LFT&E 38

3.6.5 Suggested Actions 40

4. Framework for Vulnerability/Lethality Assessment 42

4.1 Taxonomy of the V/L Analysis Process 43

4.2 V/L Taxonomy and Vulnerability Assessment 47

4.2.1 The Required Data 48

4.2.2 The Taxonomy of Vulnerability Assessment 49

4.2.3 Data Sources 53

4.2.4 Critical Data Voids 55

4.3 Vulnerability Assessment Plans 57

4.3.1 CATV Principles 58

4.3.2 Construction of Vulnerability Assessment Plan 60

4.3.3 Capability of Vulnerability Assessment Plan 61

4.3.4 Cost of Vulnerability Assessment Plan 64

4.3.5 Selection of Optimal Vulnerability Assessment Plan 66

4.4 Complexities of Methodology 70

5. Conclusion 71

5.1 Role of FU SL LFT&E in Cost-Effective Vulnerability Assessment... 71

5.1.1 Cost-Effective Vulnerability Assessment Process 72

5.1.2 Risks of Vulnerability Assessment Process 73

5.1.3 Risks of Vulnerability Assessment and FU SL LFT&E 74

5.2 Cost-Effective FU SL LFT&E 75

5.2.1 Prioritization of Data Voids 76

5.2.2 Consideration of Alternative Activities and Designs

ofFUSLLFT 76

5.2.3 Evaluation of Control Structure 76

5.2.4 Review of FU SL LFT&E Outcomes 77

5.2.5 Contributions of FU SL LFT&E to V/L Assessment 78

5.3 Cost-Effective Vulnerability Assessment and Acquisition Process... 78

6. References 83

Acronyms 89

Distribution List 91

Report Documentation Page 103

VI
List of Figures

Figure Page

1. Graphical Representation of the Taxonomy 45

2. Taxonomy of Vulnerability Assessment 50

3. Priorities of Data Voids Addressed in Vulnerability

Assessment Plan 56

4. Potential Elements of Vulnerability Assessment Plan to Address

Critical Data Voids 62

5. Elements and Estimated Execution Cost of Alternative

Vulnerability Assessment Plans 63

6. Risk Assessment Measures of Alternative Vulnerability

Assessment Plans to Access Identified Critical Data Voids 65

7. Maximum Levels of Risk Acceptable in Vulnerability Assessment

Plan to Address Identified Critical Data Voids 67

8. Comparison of Maximum Acceptable Levels of Risk and Budget

for Vulnerability to Risk Assessment Measures and Estimated

Costs of Alternative Vulnerability Assessment Plans 69

Vll
INTENTIONALLY LEFT BLANK.

Vlll
1. Introduction

U.S. congressional legislation requires weapon systems in certain categories to

undergo Full-Up System-Level Live-Fire Testing and Evaluation (FU SL LFT&E) prior

to the system entering full-scale production, unless a waiver is requested by the Service

acquiring the system and granted by the Secretary of Defense [1]. The waiver request

must show why FU SL LFT&E is unreasonably expensive and impractical and include

an alternative strategy for assessing the survivability/lethality of the system.*

Significant resources are consumed in the planning, execution, and evaluation of a

Full-Up System-Level Live-Fire Test (FU SL LFT). In this era of decreased defense

budgets and limited resources, it is important for decision-makers to determine the

optimal strategy for assessing the vulnerability or lethality (V/L) of a weapon system

and the role of FU SL LFT&E in that strategy.+ The selection of the optimal assessment

strategy requires that a consistent methodology be in place for the identification and

measurement of the costs and benefits of potential assessment plans, the weighing of

the costs against the benefits for plans considered, and the comparison of alternative

competing plans.*

* U.S. Code requires Full-Up System-Level Live-Fire Test and Evaluation to be conducted on a covered

system (any vehicle, weapon platform, or conventional weapon system that includes features designed

to provide some degree of protection to users in combat and that is an Acquisition Category I or II

program), a major munitions program, a product improvement to a covered system or a major

munitions program, or a missile program before it can proceed beyond low-rate initial production [1].

The Secretary of Defense may waive the application of tests, if the Secretary, before the system enters

engineering and manufacturing development (EMD), certifies to Congress that a FU SL LFT of such

system or program would be unreasonably expensive and impractical. A waiver and alternative

LFT&E plan must have been submitted and approved by Milestone II (i.e., approval to enter EMD).

+ FU SL LFT&E, as described in this report, addresses questions of vulnerability and lethality, not the

broader concepts of system survivability, effectiveness, and susceptibility.

* In Live-Fire Test (LFT) legislation, an alternative strategy is a strategy that does NOT include FU SL

LFT&E. In this report, however, the term alternative, as in alternative strategy or alternative plan, is

defined more broadly and is used to indicate more than one option or a choice among options.

Therefore, one or more alternative vulnerability or lethality (V/L) assessment plans or strategies may

include FU SL LFT&E as an element.
The ultimate goal of the Live-Fire Test and Evaluation (LFT&E) process, which

includes analysis, experimental, and testing activities in addition to FU SL LFT, is to

quantify the V/L characteristics of the platform or munition. Each activity of the

LFT&E process may provide an important element(s) to the V/L assessment of the

system, and some data critical to the assessment may be provided by alternative

activities. When designing strategies for V/L assessment, it is therefore important to

consider the whole acquisition cycle and all potential sources for the data needed

by evaluators in the assessment process.

1.1 Research Objectives. This study seeks to establish a foundation for the

development of a methodology to (1) identify, measure, and categorize the costs and

benefits of FU SL LFT&E, (2) determine the relative significance of FU SL LFT&E (one

element of a V/L assessment plan) to the V/L assessment plan taken as a whole, and

(3) compare competing V/L assessment plans for a system. The comparison of

assessment plans should consider the costs, benefits, and uncertainties associated with

each of the alternative plans presented.

It is proposed that the development of the aforementioned methodology will

provide information needed for

• Ranking alternative plans of V/L assessment.

• Budgeting resources for the FU SL LFT&E element of a V/L assessment plan.

• Making specific decisions, including the appropriateness of a FU SL LFT&E

waiver for systems covered by Live-Fire Test (LFT) congressional legislation.

1.2 Research Approach.

1.2.1 Systems. The FU SL LFT&E programs of three U.S. Army weapon systems

included under the LFT congressional mandate are analyzed in this study:
• Abrams MlA2: FU SL LFT&E, an element of the vulnerability assessment plan of

the weapon system, completed in 1993.

• Army Tactical Missile System Block IA (ATACMS Block IA): FU SL LFT&E, an

element of the lethality assessment plan of the weapon system, completed in

1996.

• Bradley Fighting Vehicle System-A3, M2A3/M3A3 (Bradley A3) FU SL LFT&E,

an element of the vulnerability assessment plan of the weapon system, scheduled

to be completed in 1999.

1.2.2 Impacts ofFU SL LFT&E. The specific activities that define the FU SL LFT&E

element of a system's assessment strategy are identified. For each system, the

components of each identified activity and the costs related to those components are

determined. In addition, the following practices and procedures are examined:

• Current procedures used in the measurement of the costs related to identified

activities.

• Current practices used in reporting the computed costs.

• Current practices used in budgeting for the costs of FU SL LFT&E and

computing and reporting the variances between actual and budgeted costs.

Identifying the benefits of FU SL LFT&E, as well as the benefits of other elements of

alternative assessment plans, is an important part of the methodology that compares

competing assessment strategies. Although there is a large element of subjectivity in

describing these benefits, several broad categories of FU SL LFT benefits are identified

through discussions with individuals associated with the V/L assessment plans of the

systems studied. The capabilities of an element of a V/L assessment plan to address the

critical evaluation issues of the system are considered to be the benefits of that plan

element.
Complexities encountered in identifying, measuring, and reporting the costs and

benefits or impacts (i.e., costs and benefits are often described by the term impacts in the

cost-benefit literature) of FU SL LFT&E and its associated activities are recognized, and

approaches for addressing these complexities are suggested.

It is proposed that the methodology used in identifying and measuring the impacts

of FU SL LFT&E is applicable equally to the computing of the costs and benefits of other

elements of the assessment strategies for weapon systems.

1.2.3 Comparison ofCompeting Strategies. The Taxonomy of the V/L Analysis

Process (V/L Taxonomy)* provides the framework for identifying (1) the specific data

set required for an adequate assessment of the critical V/L issues of a weapon system,

(2) the subset of the required data set available from existing reliable sources, and

(3) data voids to be addressed in a V/L assessment plan.

The potential of adapting the principles of the Cost as an Independent Variable

(CATV) methodology to the evaluation of competing strategies of V/L assessment is

explored. The analyses of the identified impacts of alternative assessment strategies,

the uncertainties (i.e., risks) associated with these impacts, and the priorities assigned

by analysts to the critical data voids expected to be produced by these assessment

strategies serve as inputs to the evaluation approach grounded in CATV principles

proposed in this report.

Complexities encountered in predicting the data or information to be obtained from

a V/L assessment plan prior to completing the plan and prioritizing the information

needs of decision-makers are discussed.

1.3 Organization of Report. The remainder of the report is organized as follows.

The research methodology employed in this study is developed in section 2 and

includes a description of the three weapon systems studied and the identification of the

* The Taxonomy of the V/L Analysis Process was first introduced in Deitz and Ozolins [2].
agencies that provided much of the information upon which this report is based. The

results of the research, including descriptions of the activities of FU SL LFT&E, the

impacts of those activities, the complexities encountered in the reporting of the impacts,

and approaches for addressing the complexities are reported in section 3. A discussion

of Project Manager (PM) personnel's perceptions of the role of FU SL LFT&E in the

survivability/lethality assessment process and their suggestions for improving the costeffectiveness of the assessment process complete this section of the report. Section 4

includes a discussion of the application of the V/L Taxonomy framework and CATV

principles to the decision-making process of selecting the optimal V/L assessment

strategy. The report concludes in section 5 with a discussion of the relationship of FU

SL LFT&E, cost-effective assessment strategies, and the acquisition process.

2. Research Methodology

2.1 Systems. A description of the three weapon systems examined in this study

and a brief review of the FU SL LFT&E events related to those systems are provided in

the following paragraphs.

2.1.1 Abrams M1A2. The M1A2 Abrams Main Battle Tank system is described as

follows in the "FY1996 Annual Report of the Director, Operational Test and

Evaluation [3]:"

The mission of the MlA2 Abrams tank is to close with and destroy enemy

forces using firepower, maneuver, and shock effect. The M1A2 is an

upgrade of the Abrams MlAl intended to improve target acquisition and

engagement rates, improve survivability, and maintain operational

suitability at the level of the MlAl. Many of the enhancements were made

by replacing or modifying previous items to take advantage of the

introduction of digital distributed data and power architecture of the

MlA2. The Inter-Vehicular Information System and the Position/

Navigation equipment are designed to improve battlefield command,

control, and communications. The Commander's Independent Thermal

Viewer increases the rate of target acquisition so that the gunner is able to

engage targets more rapidly.
The FU SL LFT, an element of the vulnerability assessment plan of the Abrams

MlA2, was conducted from April to October 1993 at the U.S. Army Aberdeen Test

Center (ATC) at Aberdeen Proving Ground (APG), MD.* A total of 11 shots (i.e.,

10 shots planned, 1 shot repeated) were fired against 2 of the initial 62 M1A2 vehicles

produced in the Low-Rate Initial Production (LRIP) contract. The Office of the

Secretary of Defense (OSD) submitted the Abrams MlA2 Live-Fire Test & Evaluation

Report to the U.S. Congress in May 1994.

The Abrams M1A2 FU SL LFT was designed to investigate the effects of nonperforating and perforating threats likely to be encountered in combat. The M1A2

Abrams Live Fire Vulnerability Test objectives include [4]:

• To determine the vulnerability of the MlA2 vehicle and total system

(crew and vehicle) along with the corresponding loss of mobility and/or

firepower function as a result of nonperforating or perforating threats

likely to be encountered.

• To determine the vulnerability of the crewmen in the modified crew

compartment.

• To investigate the survivability improvements incorporated into the

MlA2 as a result of previous testing.

• To determine if there were unexpected or unacceptable MlA2

vulnerabilities and determine how these vulnerabilities might be

reduced or eliminated if they were significant.

• To assess the repairability of the vehicle.

2.2.2 ATACMS Block IA. The ATACMS Block IA system is described as follows in

the "FY1997 Annual Report of the Director, Operational Test and Evaluation [5]:"

The Army Tactical Missile System (Army TACMS) is a family of longrange, near all-weather guided missiles fired from the Multiple Launch

Rocket System (MLRS) M270 launcher and deployed within the

ammunition loads of corps MLRS battalions. ... The Block IA is an upgrade

intended to double the range of the current Army TACMS Block I missile.

* The Aberdeen Test Center (ATC) was formerly known as the Combat Systems Test Activity (CSTA).
Army TACMS Block IA will dispense M74 Anti-Personnel, Anti-Materiel

(APAM) bomblets, as does the Block I. ...

The primary changes in Army TACMS Block IA are a reduced payload

(from 950 to 300 bomblets) and the addition of a Global Positioning System

(GPS) navigation aid to the inertial guidance system. The reduced payload

is necessary to achieve the extended range. To compensate for the reduced

payload, the accuracy of the missile has been improved with inflight GPS

updates. If GPS is rendered inoperable, the Army TACMS Block IA reverts

to inertial guidance only and maintains Block I accuracy.

The FU SL LFT&E of the ATACMS Block IA did not include vulnerability tests of

the chassis and launcher of the missiles. The evaluation of the FU SL LFT&E of the

ATACMS Block IA includes the results of 3 test events: (1) the full-up live-fire arena

tests, conducted from May to June 1996 and designed to observe effects of 23 M74

bomblets detonated at various ranges against 3 operationally representative threat

targets, (2) the 2 developmental and operational test live-fire launcher-to-target firings

(previously referred to as end-to-end firings), conducted in August and September 1996

against an array of 9 and 15 nonoperating targets and designed to provide a

demonstration of ATACMS potential to inflict damage upon a multiple-vehicle area

target, and (3) the tire fragment tests, conducted from June 1994 to June 1995 and

designed to test the capability of M74 and other munitions with small fragments to

inflict damage on tires (the tire tests were not an official part of the LFT plan, but their

results were included in LFT evaluation). The arena and tire fragment tests were

conducted respectively at ATC and the Survivability/Lethality Analysis Directorate,

U.S. Army Research Laboratory (SLAD/ARL) at APG, and the launcher-to-target flight

tests were performed at White Sands Missile Range (WSMR), NM. Milestone m for

ATACMS Block IA was deferred to allow more time to address issues related to

acquisition, in-flight survivability, and lethality. Follow-on tests were conducted both

at SLAD and ATC, and the ATACMS Block IA LFT&E report was submitted by OSD to

the U.S. Congress in April 1998.

The ATACMS Block IA FU SL LFT was designed to provide information to assess

the ability of the system to eliminate and/or degrade the operational functions of

representative threat-target systems, thus providing insights into the principal damage

mechanisms and target failure modes occurring as a result of munitions and target
interaction. The "Live Fire Independent Evaluation Plan/Test Design Plan (IEP/TDP)

for the Army Tactical Missile System (ATACMS) Block IA" reports the critical test and

evaluation issues of the ATACMS Block IA program as follows [6]:

• Bomblet lethality ... What is the lethality of the M74 bomblet against

personnel and light materiel targets? Subissues include: What are the

M74 fragmentation characteristics? What are the penetration

characteristics of the M74 fragments? Can the M74 fragments routinely

damage components of the desired targets? How is the performance of

the M74 bomblet affected by degraded conditions such as uneven

terrain and various impact media?

• Pattern lethality ... What is the lethality of the ATACMS Block IA missile

against the required target elements, given the element is in the bomblet

pattern? Subissues include: What is the density and distribution of the

M74 bomblets within the bomblet ground pattern? What is the M74

bomblet dud rate and distribution within the ground pattern?

• System Effectiveness ... What is the system effectiveness of the ATACMS

Block IA system against the required target sets to include both point

and area targets? Subissues include: What is the system accuracy

relative to the aimpoint? What are the pattern dimensions? What are

the target dimensions (area targets)? What is the Target Location Error

for the target of interest?

2.1.3 Bradley A3. The Bradley A3 system is described as follows in the "FY1998

Annual Report of the Director, Operational Test and Evaluation [7]:"

The M2A3 and M3A3 Bradley Fighting Vehicle Systems (BFVS) are

improved versions of the M2A2 and M3A2 BFVS. The BFVS-A3 includes

enhancements intended to improve lethality, mobility, survivability, and

sustainability. Additionally, these enhancements are intended to provide

increased situational awareness and digital command and control

capabilities necessary to provide information superiority to the dominant

maneuver force

The mission of the BFVS is to provide mobile protected transport of an

infantry squad to critical points on the battlefield and to perform cavalry

scout missions. The BFVS will also provide overwatching fires to support

dismounted infantry and suppress or defeat enemy tanks and other

fighting vehicles. BFVS-A3 enhancements include:

• Incorporation of Force XXI Battle Command, Brigade and Below

Embedded Battle Command to share command and control and

situation awareness with all components of the combined arms team.
• The improved Bradley acquisition system and commander's

independent viewer,..., to improve target acquisition and target

engagement.

• A position navigation system with a Global Positioning System

receiver to enhance situational awareness.

The following comparison of the Bradley A2 and the Bradley A3 is provided in the

LFT&E Strategy included in the "Bradley Fighting Vehicle System, A3 Test and

Evaluation Master Plan (TEMP) [8]:"

Although much of the ballistic protection is the same as on the Bradley

A2, some changes have been made to both internal and external

configurations: electronic systems have been changed and upgraded,

new subsystems have been added to the vehicle both internally and

externally, armor covering most of the roof and some of the upper

portion of the sides has been improved, a pontoon system is being

developed and a 10th crew member has been added.

The Bradley A3 FU SL LFT, an element of the vulnerability assessment plan of the

system, began in December 1998 at ATC in Aberdeen Proving Ground, MD, and is

scheduled for completion in 1999. Eighteen shots are planned against two production

qualification test (PQT) vehicles, upgraded to production configuration with

3.0Z software end product, and an LRIP vehicle.

The Bradley A3 FU SL LFT is designed to investigate the effects of nonperforating

and perforating threats likely to be encountered in combat. The critical test and

evaluation (T&E) issues are defined as follows in the LFT&E Strategy included in the

"Bradley Fighting Vehicle System, A3 Test and Evaluation Master Plan (TEMP) [8]:"

• What is the vulnerability of the Bradley A3 system to the spectrum of

expected threats and how does this compare to previous Bradley

Fighting Vehicles? Subissues [include:] How are the vulnerabilities of

the Bradley A3 changed ... by the additional roof armor? What impacts

do the A3 modifications have on Bradley system vulnerabilities? ...

What is the impact of the A3 modifications on the Bradley system's

vulnerability to ballistic shock?

• What is the vulnerability of the A3 crew and troops to the spectrum of

expected threats and how does this compare to previous Bradley

Fighting Vehicles? Subissues [include:] How is the vulnerability of the
Bradley A3 crew and troops changed ... by the additional roof armor?

What impacts do the A3 modifications have on Bradley crew and troop

vulnerability? ...

• How effective is battle damage assessment and repair (BDAR) in

restoring the A3 to rriinimum functional combat and full mission

capability following a hit by expected threats (direct and indirect)?

Subissues [include:] What changes in BDAR procedures are required

for the Bradley A3 compared to the Bradley A2? How effective are the

applicable BDAR manuals, procedures, tools, and supplies for the A3?

• What are the vulnerabilities of the BFVS A3 system/crew/troops to new

and/or emerging threats?

• Are there any unexpected BFVS A3 vulnerabilities? Subissues [include:]

What is the operational significance of any unexpected vulnerability?

Can these vulnerabilities be reduced?

2.2 Information Sources. The following agencies/organizations provided

information for this research study:

• Survivability/Lethality Analysis Directorate (SLAD), U.S. Army Research

Laboratory (ARL).

• Evaluation Analysis Center (EAC), U.S. Army Operational Test and Evaluation

Command (OPTEC).

• Headquarters, U.S. Army Test and Evaluation Command (HQ, TECOM)/

• U.S. Army Aberdeen Test Center (ATC).

• U.S. Army Materiel Systems Analysis Activity (AMSAA).

• PM Offices (PMOs):

0 ATACMS PMO, U.S. Army Aviation and Missile Command (AMCOM),

Redstone Arsenal, AL.

* In a recent command reorganization, EAC, OPTEC, and TECOM became elements of the Army Test and

Evaluation Command (ATEC).

10
0 Abrams PMO, U.S. Army Tank-automotive and Armaments Command

(TACOM), Warren, MI.

0 Bradley PMO, TACOM, Warren, MI.

• Institute for Defense Analysis (IDA).

Discussions with the personnel from SLAD, EAC, TECOM, ATC, and AMSAA

focused on general background information of the respective agency, the role of the

agency in FU SL LFT&E and general T&E activities, funding and budgeting for the

activities of FU SL LFT&E by the agency, the impacts of FU SL LFT&E activities of the

agency, and the complexities encountered in identifying, measuring, and reporting

those impacts.

Discussions relative to the application of the V/L Taxonomy and CATV principles

to the analysis of competing V/L assessment strategies were an important part of the

conversations with AMSAA, EAC, and SLAD personnel. Limited discussions with IDA

personnel focused on the role of IDA in the V/L assessment of weapon systems.

Interviews with PMO personnel included discussions of the history of the system

associated with the PMO, congressional funding for the system, contracts associated

with various cycles of the acquisition process for the system, the role of the PMO in the

acquisition process and the FU SL LFT&E of the system, the impacts of the FU SL

LFT&E of the system, and the complexities encountered in the T&E activities of the

system.

In total, more than 55 scheduled discussions or interviews were conducted to gather

the information needed for this research project with approximately 60 people

participating in these conversations. In addition, many informal discussions occurred

at defense-related conferences and workshops attended during the most recent 2- to

3-year period.*

* Conferences and workshops included LFT&E—10 Years and Counting Conference, Lawrence

Livermore National Laboratory, Livermore, CA, January 1997; Fourteenth International Symposium on

Military Operational Research, Royal Military College ofScience, Shrivenham, Wiltshire, United

11
Many documents relative to the three systems, LFT&E and its impacts, the V/L

Taxonomy, the CAIV methodology, T&E, and other related topics were reviewed.

3. Research Results

To determine the impacts of FU SL LFT&E, the activities that constitute the FU SL

LFT&E program must be delineated. In reality, conducting an Army FU SL LFT&E

program results in activities in many agencies, including the Office of the Director,

Operational Test and Evaluation (DOT&E), the Office of the Deputy Director,

Operational Test and Evaluation, Live-Fire Testing (DDOT&E [LFT]), the Office of the

Deputy Under Secretary of the Army for Operations Research (DUSA [OR]), the Office

of the Deputy Chief of Staff for Intelligence (DCSINT), the U.S. Army Medical Research

and Materiel Command (MRMC), the U.S. Army Training and Doctrine Command

(TRADOC) including the U.S. Army Ordnance Center and School (USAOC&S) and the

U.S. Army Transportation Center and School (USATC&S), the U.S. Congress and its

subcommittees, analysis groups utilized by the aforementioned offices (e.g., IDA),

TECOM, SLAD, ATC and other test centers, AMSAA, EAC, and the PMOs associated

with the specific weapon system tested. For the purposes of this research project, the

FU SL LFT&E program is defined more narrowly and the activities examined are

limited to those described in the following section.

3.1 Activities of FU SL LFT&E. The following are identified as the activities

completed in a FU SL LFT&E. Specific divisions and agencies with lead responsibilities

for the activities are noted as appropriate.

Kingdom, September 1997; Conference on the Economics of Test and Evaluation, TEREC, Georgia

Institute of Technology, Atlanta, GA, October 1997; Fourteenth Annual Test & Evaluation Conference

and Exhibition, San Diego, CA, March 1998; Fifteenth International Symposium on Military Operational

Research, Royal Military College of Science, Shrivenham, Wiltshire, United Kingdom, September 1998;

Thirty-seventh Annual Army Operations Research Symposium, Fort Lee, VA, October 1998; Test Plan

Optimization Workshop, TEREC, Georgia Institute of Technology, Atlanta, GA January 1999;

Simulation and Modeling for Acquisition, Requirements, and Training Conference, San Antonio, TX,

January 1999; 10th Annual Army Ground Vehicle Survivability Symposium, Monterey, CA, March 1999;

and 67th Annual Military Operations Research Society Symposium, West Point, NY, June 1999.

12
Formation of Live-Fire Integrated Product Team (LFTPT)

EAC leads the working group under the Test and Evaluation Integrated Product Team;

members ofworking group include representatives ofthe offices ofDOT&E, DUSA(OR),

a representative ofthe military intelligence community (e.g., National Ground

Intelligence Center), in addition to personnelfrom EAC, SLAD, TECOM, the PMO, the

test center, USAOC&S/USATC&S, and TRADOC.

Modeling and Simulation

SLAD performs the extension or expansion, the verification, and the validation of

models required to be exercised in FU SL LFT&E ofweapon system; exercises the

models in pre-shot predictions; and providesfull-view V/L estimates to EACfor

use in the FU SL LFT evaluation.

Development of LFT&E Event Design Plan (EDP) that Replaces Prior

Independent Evaluation Plan/Test Design Plan (IEP/TDP)

EAC serves as lead in preparation ofEDP; receives inputfrom SLAD and other members

ofLFIPT.

Writing of Detailed LFT&E Test Plan (DTP)

Tester prepares DTP.

0 Tester ofground systems: TECOM

0 Tester ofaviation systems: SLAD

TECOM reviews DTP ofLFTs assigned to other agenciesfor execution.

Performance of FU SL LFT

0 Setup and execution

Designated tester sets up and executes.

0 Battlefield damage assessment and repair

USAOC&S conducts the BDAR ofground systems.

USATC&S conducts the BDAR ofaviation systems.

0 Damage assessment and casualty assessment

SLAD coordinates damage assessment and casualty assessment.

13
• Documentation

0 Damage assessment shot records

SLAD prepares damage assessment shot records.

0 Detailed Damage Assessment Report

SLAD prepares Detailed Damage Assessment Report.

0 Detailed Test Report (DTR)

Tester serves as lead in preparation ofDTR, receiving inputfrom

Damage Assessment Team; TECOM reviews DTRfor LFTs assigned to

other agenciesfor execution.

• Independent Evaluation and Preparation of System Evaluation Report (SER),

Including the Analysis of LFT&E Activities

EAC prepares the SER.

3.2 Costs of FU SL LFT&E. A weapon system included in the U.S. congressional

LFT&E mandate must conduct a FU SL LFT&E as part of the assessment of the

survivability/lethality program of the system, unless the waiver requested by the

Service acquiring the system is granted by the Secretary of Defense. The cost of the FU

SL LFT&E is an important consideration in determining the appropriateness of a

waiver, in budgeting dollars for the execution of a FU SL LFT&E, and in comparing

various V/L assessment strategies. Perhaps the most difficult issue faced in

determining the cost of FU SL LFT&E is defining the term cost. The analyst must

determine what components of the FU SL LFT&E activities (listed in the previous

paragraph) should be costed and how the costs of those components should be

measured. In this process, the analyst must address the following questions:

• How are the costs of the tested systems (i.e., test articles) computed, if systems

are damaged but salvageable? How are the costs computed, if the test articles

are completely destroyed?

• How are the costs of spare parts used in repairs computed?

14
• How are the costs of targets used in lethality testing or munitions used in

vulnerability testing computed?

• How are the labor costs of individuals participating in the planning; modeling

and simulation (M&S); testing; or evaluation process of FU SL LFT&E computed?

• How are the costs of the test ranges and instrumentation used in FU SL LFTs

computed or allocated across the many systems tested at the test center or across

the multiple time periods over which the test center is in operation?

To make a valid comparison of assessment strategies (e.g., those that include FU SL

LFT&E and those that do not), decision-makers must be provided with cost data that

are identified and measured according to a consistent methodology employed across

assessment plans. Clear descriptions of FU SL LFT&E activities and the separate

components of each activity and their related costs must be identified. In situations in

which there are alternative ways of measuring the costs of an activity, the measurement

method employed should be identified and described. Allocated costs should be so

identified, and the bases of allocation or the rates used in allocation should be fully

explained. In addition, information that allows the decision-maker to compute the costs

in an alternative manner should be disclosed by the cost analysts.

For purposes of this research study, cost sheets of the completed FU SL LFT of both

the ATACMS Block IA and the Abrams MlA2 were provided by the ATACMS and

Abrams PMOs and TECOM. The cost sheet of the 1993 Abrams MlA2 included costs

aggregated in five very broad categories (i.e., Test Center, Repair Parts, ARL/

TECOM/AMSAA, Ammunition and Nondestructive Tests, and Contractor Test

Support). The time period between the completion of this test and the changes in

personnel across agencies made the costs related to the Abrams M1A2 difficult for

interviewees to re-construct. The cost sheet of the more recent 1996 FU SL LFT&E of the

ATACMS Block IA included a more detailed breakdown of the costs related to work

performed by SLAD and the two test centers (i.e., ATC and WSMR) in addition to some

target-related costs. Costs for the forthcoming Bradley A3 FU SL LFT&E were available

15
only in total budgeted amounts for the tests. All costs referred to in the following

sections were taken from the sheets provided for this study.

Personnel from all three PMOs reported that records are kept of the FU SL LFT&E

costs incurred by government agencies and by contractor(s) and subsequently

reimbursed by their offices. ATC reports that it provides the PMO with both interim

reports of the costs incurred by the test center (i.e., during the execution of the FU SL

LFT) and a final accounting of test-related costs at the conclusion of the FU SL LFT. The

PMO, however, is not required to produce a formal report detailing the total costs of all

FU SL LFT&E-related costs incurred by government agencies and contractors and

reimbursed by the PMO.

This research project did not have the objective of critically reviewing the costs

incurred or reported by the systems studied but of understanding how the costs of FU

SL LFT&E are identified, measured, and reported and what complexities are

encountered in accomplishing these tasks. An examination of the information included

in the available cost sheets and discussions with individuals associated with the

activities of FU SL LFT&E provided much insight into these issues.

In this report, the costs of FU SL LFT&E are placed in one of two categories:

(1) costs of direct materials used and consumed in the FU SL LFT&E or (2) costs of

activities, including the costs of indirect materials, labor, and facilities used in the

planning, execution, and evaluation of the FU SL LFT and its results.

3.3 Costs of FU SL LFT&E: Costs of Direct Materials. The costs of direct

materials used or consumed in the FU SL LFT&E are defined as all material costs that

can be feasibly or economically traced to the cost objective (i.e., FU SL LFT&E). The

direct material costs of FU SL LFT&E include the costs of the following test assets:

(1) the test articles (i.e., weapon systems tested), (2) the spare parts provided for the test

articles, (3) the munitions fired (i.e., applicable to vulnerability tests), and (4) the targets

fired upon (i.e., applicable to lethality tests). It is assumed in this report that the total

cost of all test assets discussed will include the transportation and preparation costs

incurred to ready the asset for FU SL LFT at the designated test center.

16
3.3.1 Cost of Test Articles. The following costs are proposed as alternative

reporting values for test articles used in FU SL LFT&E:

• Cost of replacing the test article with an article identical to the original article

(i.e., replacement cost).

• Cost of returning the test article following FU SL LFT to its condition prior to

FU SL LFT (i.e., restorative cost).

• Cost of acquiring the original test article (i.e., historical cost).

The costs of test articles have been accounted for in various ways in completed FU

SL LFT&E programs. For example, the total program costs for the FU SL LFT&E

provided by the Abrams PMO do not include the costs of the vehicles used in testing.

Both test articles were deemed salvageable at the conclusion of the 1993 LFT program

with an expenditure of $1.5 million estimated to return the two Abrams M1A2 LRIP

vehicles to combat condition. The PMO requests for dollars to return the articles to

combat condition were not funded, and the two tanks are currently in use as training

vehicles (e.g., track tests) at APG. Replacement cost or the current cost to convert an

Abrams Ml to an M1A2 (cost includes some engineering support for potential

production problems) is estimated by the Abrams PMO at $4-5 million per vehicle.*

Historical costs or the actual costs incurred to construct the two LRIP Abrams M1A2

vehicles used in the FU SL LFT&E were not available.

Neither the cost of the missiles used in the ATACMS Block IA tests (i.e., launcherto-target tests) nor the relatively small cost of the M74 bomblets (i.e., approximately 100

bomblets used at $30/bomblet) in the arena tests was included on the LFT&E cost sheet

produced by the ATACMS PMO. PMO personnel stated that the missile cost was

treated by its office as part of the costs of the developmental and operational tests of the

ATACMS Block LA. The PMO quoted a figure of $1.01 million as the cost of producing

* It should be noted that this is the replacement cost estimate in 1998; replacement cost in 1993 is

unavailable.

17
an ATACMS Block IA missile (i.e., replacement cost). This cost includes the costs of

contractor, support of PM office, and production testing.

The projected cost figures for the LFT&E program of the Bradley A3 include

$2.7 million (of the total $6 million FU SL LFT&E budget of the Bradley PMO) budgeted

for the replacement of two PQT vehicles.*

From the aforementioned three examples, it is evident that the total costs of FU SL

LFT&E may be significantly affected by the choice of the alternative reporting value

chosen for the cost of the test article(s). It is proposed in this report that the restorative

cost (i.e., the cost to return the test article to its original condition) is the most

appropriate value to report as the test article cost if (1) the test article is in a salvageable

condition following testing, (2) the restorative cost is able to be estimated, and (3) the

restorative cost is the smallest of the three proposed alternative values. The restorative

cost is a cost oftesting, regardless of whether or not the expenditure is made. If the

restorative cost is higher than one or both of the remaining alternative values (i.e.,

replacement cost and historical cost) or the test article cannot be restored, a choice must

be made as to which of the two alternatives will be reported as the test article cost. Each

value has its own merits, but it is suggested that the lower of the two values be reported

as the cost of the test article.' Regardless of which value is reported, all available

alternative values for costing the test article should be disclosed in the cost report and

made part of the database available to those budgeting for FU SL LFT&E of future

systems. The computations of all reported values, including the amounts attached to

* An LRIP vehicle was added to the test articles planned to be used in the FU SL LFT. The PMO stated

that all Bradley A3 vehicles were expected to be «manufactured from used vehicles, and there were no

current plans to build any new BFVS vehicles for the U.S. Army.

+

In general, an asset is reported at its historical cost under generally accepted accounting principles

because the historical cost is assumed to be objectively determined in an arm's length transaction

between two independent parties and therefore reflects the fair value of the asset at the tramacbon date.

In the case of FU SL LFT&E, however, the historical cost of the prototype test asset used in FU SL LFT

could be relatively high in comparison to the replacement cost of the asset. It is hard to defend the

position that if the test asset is lost in the FU SL LFT, the entire historical cost of the prototype should be

reported as a cost of FU SL LFT&E instead of reporting the replacement cost of the asset. In reality, a

system would not be expected to be restored IF its replacement cost were lower than its restorative cost

(i.e., given enough time for replacement). Therefore, the suggested rule reduces to report the lowest of

the three values available: restorative cost, replacement cost, or historical cost.

18
these costs (e.g., engineering support costs), should be explained in full to facilitate the

processing of this information in comparative analyses of assessment plans and in

future budgeting activities.

The costs of assets other than the test articles include the costs of the spare parts

provided for the repairs of the test articles made during the FU SL LFT&E, the

munitions fired (i.e., vulnerability tests), and the targets fired upon (i.e., lethality tests).

As in the cost of test articles, there are often alternative values (e.g., historical cost,

restorative cost, and replacement cost) available for assets included in this category.

3.3.2 Cost of Spare Parts. The accounting for spare parts for test articles in the FU

SL LFTs of two of the systems studied raises the question, "Are spare parts provided to

the test center as part of a larger contract, and are those costs able to be separated from

the major costs of the contract and reported as a line item in a FU SL LFT&E cost

report?"* In the Abrams M1A2 FU SL LFT&E, the spare parts were provided at a fixed

cost as a test system support package attached to the LRIP Contract for 62 tanks with

General Dynamics Land Systems, and ATC accounted for the spare parts as used. The

Abrams FU SL LFT&E cost sheet reported a cost of $1.6 million for repair parts, but the

Abrams PMO stated this estimate could be as high as $2 million. The attachment of the

spare parts cost package to the LRIP contract made this cost more difficult to estimate.

The Bradley PMO reported that it plans to acquire spare parts needed for the FU SL

LFT&E of the Bradley A3 from Army depots (i.e., Bradley common parts) and the LRIP

contractor, United Defense Limited Partners (i.e., Bradley A3 unique parts). Pre-shot

predictions have allowed the PMO to project parts that will be needed in the FU SL

LFT. The LRIP contractor will provide the parts as needed from vehicles completed in

its contract, and a contract modification will be added following the completion of the

FU SL LFT to account for parts required during the testing phase.

Costs for spare parts of test assets are not applicable to the lethality tests of the ATACMS Block IA (with

the exception of launcher spare parts).

19
The cost incurred at the time of purchase (i.e., historical cost) is generally reported

as the cost of spare parts. In addition, the reporting of this cost should include a

description of the contract agreement (e.g., pay contractor a fixed amount for all parts

needed, pay contractor an amount that varies with the type and number of spare parts

needed, etc.). Disclosure of the spare parts used vs. spare parts purchased (i.e., in fixeddollar contract) and related costs may prove useful in planning of subsequent FU SL

LFT events.

3.3.3 Cost ofMunitions and Targets. The costs of some assets related to FU SL

LFT&E are particularly difficult to determine. The costs of munitions used in

vulnerability tests or targets used in lethality tests may not be readily available or easily

estimated. Munitions and targets may have been obtained without exchanging dollars,

if acquired in prior combat or in nonmonetary trades with other agencies. ATC reports

that munitions are okenfree or obtained at a relatively insignificant cost from a foreign

source.

No cost information relative to the munitions used in the completed Abrams MlA2

FU SL LFT&E (i.e., vulnerability test) is included on the cost sheet provided. Neither

the costs of the targets used in the launcher-to-target and arena tests of the ATACMS

Block IA (i.e., lethality tests) nor the costs of the repairs to the targets used in those tests

were totally accounted for on the FU SL LFT&E cost sheet provided.* Additional targets

in the launcher-to-target tests conducted at WSMR were required and funded by OSD

and were not included in the costs reported on the cost sheet as reimbursable by the

PMO. The cost sheet does include a portion of the cost of transportation for targets

used in the arena tests.

Target repair costs were not included on the ATACMS Block IA cost sheet. The cost

sheet does not include the sum requested by the Project Manager, Instrumentation,

* Of the 3 targets used in the arena tests of the ATACMS Block IA, only 1 was a target included in the

Operational Requirements Document (ORD) list of 11 targets; the other 2 targets had characteristics

similar to the other 10 ORD target types. Targets used in the launcher-to-target tests included some

target surrogates, some foreign targets with elements representative of ORD targets, and some targets

that were less vulnerable than ORD targets.

20
Targets and Threat Simulators (point of contact for obtaining, controlling, operating,

and sustaining threat equipment or surrogates for use in LFT) for restoring the targets

used in the ATACMS Block IA FU SL LFT. The PMO reported the intent to pursue

some "trading" of other target assets with the Target Management Office to compensate

the Office for damage sustained to borrowed targets. It may be difficult to obtain an

historical cost or to estimate a replacement cost for the munitions used in lethality tests

or the targets used in vulnerability tests. A complete accounting of the costs of test

assets, however, should include, if available, an estimated cost of the munitions/targets

with the relevant range of values for that estimated cost or a disclosure of the inability

to estimate the cost for identified assets.

3.4 Costs of FU SL LFT&E: Costs of Activities. In this report, the costs of

activities of FU SL LFT&E are defined as the costs incurred for indirect materials

consumed, labor performed, and facilities used in the planning, execution, and

evaluation of a FU SL LFT and its results. To compute the cost of an activity, all

components of the activity that consume resources and the costs associated with those

components must be identified. The agencies associated with the activities of FU SL

LFT&E described in the beginning of this section account for their costs in different

ways as shown in the following sections.

Labor costs are a large component of the costs of all FU SL LFT&E activities across

all agencies. In accounting for the labor costs of employees engaged in activities in an

agency, the following questions must be addressed:

• Do labor costs include overhead administrative costs of the employee's agency?

• Are labor costs based on the actual hours worked or the hours estimated to be

needed for the required service? In other words, is the cost of labor reported in

fact the budgeted cost of labor, similar to the estimated or budgeted cost of labor in

a contract that is fulfilled with the delivery of the product?

3A.1 Cost ofSLAD (AKL) Activities. There are two major SLAD activities:

(1) M&S and (2) damage and casualty assessment, in addition to LFHT participation.

21
A closer look at the SLAD M&S activity reveals four components: (1) extension

and/or expansion of models to include descriptions and characteristics of the

test article, munitions, target(s) of interest, and damage mechanisms not previously

modeled, in addition to improvements in damage mechanisms already part of the

model (e.g., behind-armor debris); (2) verification and validation (V&V) of existing

model(s); (3) exercise of model in pre-shot predictions and reruns of pre-shot

predictions (e.g., reruns needed if actual geometry of shots is different than planned

shots); and (4) provision of full-view V/L estimates to EAC for use in the FU SL LFT

evaluation. Modeling activities related to pre-shot predictions provide important data

for decisions related to test planning and execution (e.g., selection of shot lines, stock

levels of spare parts, etc.) and a certain measure of quality control for the modeling

function. On the other hand, modeling activities related to the production of full-view

V/L estimates generate data important to the evaluation of the FU SL LFT and the

overall assessment of the V/L of the system.

As stated previously, the cost sheet for the FU SL LFT&E of the ATACMS Block IA

system includes a more detailed accounting of costs than is available for the other two

systems studied and is therefore used as an example of the costing of SLAD M&S

activities. The FU SL LFT of the ATACMS Block IA incurred costs for the development

of the inputs to the model including target descriptions, criticality analyses, and the

probability of component kill assessments for the three targets of the arena tests. Preshot predictions were also accomplished for the 23 planned shots against the 3 arena

targets. Following the post-shot analysis, the vulnerability analysis was updated with a

rerun of the SAFE model including the target description updates (9 targets included in

Operational Requirements Document [ORD]). As part of the follow-on effort,

algorithms were developed to characterize damage mechanisms and reactions not

previously modeled, such as pyrophonic and fragment effects given fuel and propellant

interaction.

Verification, validation, and accreditation (W&A) of models exercised as part of

the FU SL LFT&E must be achieved prior to relying on modeling efforts in FU SL

22
LFT&E.* SAFE and Stochastic Quantitative Analysis of System Hierarchies (SQuASH)

models, as exercised in the FU SL LFT&E of ATACMS Block IA, have recently

undergone model calibration exercises in which the models are compared to existing

test results. The costs of these exercises for the ATACMS Block IA FU SL LFT&E were

absorbed by SLAD mission funds. The FU SL LFT&E of the Abrams M1A2 relied on the

results of model FORTRAN SQuASH, but W&A, as it is now known, was not

performed for the Abrams system in 1993. The SQuASH and SAFE sections of the

Modular UNIX-based Vulnerability Estimation Suite (MUVES) have been consolidated

into a new combination model, MUVES-S2. The V&V of MUVES-SQuASH for Bradley

A2 has been completed by SLAD in preparation for the Bradley A3 LFT, but V&V for

SAFE relative to Bradley A3 is ongoing. The costs of the effort for V&V required for

Bradley A3 have again been absorbed by SLAD mission funds. SLAD personnel expect

that in the future PMOs will be asked to contribute to the V&V effort, including the

reimbursement of material and labor costs incurred in the experiments and analyses

necessary to support the changes in codes and algorithms associated with the models

required for a system's FU SL LFT&E.

The damage assessment team, generally chaired by SLAD personnel, is responsible

for the damage assessment at the conclusion of each FU SL LFT shot. Members of the

team generally include personnel from SLAD and the test center and may include

personnel from the proponent TRADOC school and EAC among others. The damage

assessment shot records, in which the damage assessments for all FU SL LFT shots are

detailed, as well as ancillary-related data, are prepared by SLAD and test center

personnel for inclusion in the DTR produced by the test center. The Damage

Assessment Report, which includes the comparisons of the results of the FU SL LFT

shots with pre-shot predictions, is also prepared by SLAD personnel.

The personnel casualty assessment effort is also led by SLAD personnel with

support from the test center and others as needed (e.g., personnel from MRMC,

* Accreditation of models is not performed by Survivability/Lethality Analysis Directorate (SLAD)

personnel.

23
including Walter Reed Army Institute of Research and Aeromedical Research

Laboratory).*

Examples of the costs incurred by SLAD in damage and personnel casualty

assessments can be found on the cost sheet for the FU SL LFT&E of the ATACMS

Block IA system. Included are the costs for damage assessment at ATC and WSMR,

damage assessment support and model verification, and personnel casualty assessment

at ATC and WSMR. SLAD employee-years budgeted by branch chiefs include the

participation of branch employees in system-related working group meetings. SLAD

also budgets for its labor costs incurred in the support of the tester's plans and reports.

For example, the ATACMS FU SL LFT&E cost sheet included dollar figures for

IEP/TDP support, preparation of DTP, DTR, General Support, and Final Test Report.

A SLAD branch chief budgets labor hours for FU SL LFT&E activities by identifying

the employees and the number of employee-years needed by each identified employee

in the branch to complete the tasks required for the LFT effort. Labor cost per

employee-year is multiplied by the number of employee-years estimated for the LFTrelated task, and this product becomes the total estimated costs for that employee on the

identified task.

A labor cost per employee-year for each branch employee and an average branch

labor cost per employee-year are computed by the SLAD division and directorate

offices and provided to each branch chief for the purposes of budgeting and costing

services rendered by branch employees. The employee-year cost computation includes

the salary and fringe benefits of the employee plus an amount (based on rate per

employee-hours worked) to cover overhead associated with division and directorate

administrative offices.+Projections for SLAD costs related to LFT are made often with a

5-year horizon, but estimates are generally revised on an annual basis.

* Most test centers provide some form of data collector support (e.g., photography and assistance with

post-shot component and subsystem checks). Generally, ATC is enlisted to perform data reduction and

analyses for damage and personnel casualty assessment.

+ The SLAD overhead hourly rate has three components, including the direct and indirect overhead rates

of the division office and the general and administrative rate of the directorate office.

24
In addition to the costs incurred in LFT&E activities reviewed in preceding

paragraphs, SLAD employees also incur travel costs related to LFT&E activities. By

law, the PMO of the tested system is responsible for the reimbursement of all costs

related to the FU SL LFT&E of a system, including travel-related costs, with the

exception of costs incurred by agencies with mission-related responsibilities in the

LFT&E program. As previously mentioned, some of the FU SL LFT costs related to

M&S have been financed with mission funds of SLAD (e.g., V&V of models for Bradley

A3 FU SL LFT&E). SLAD mission funds are delegated to systems at the beginning of

the year, based on the priorities of the systems for which the SLAD Mission Area

Manager (MAM) has responsibility in the period.

SLAD personnel report that the variance or difference between the number of

employee-hours budgeted (basis for request for PMO reimbursement) and the number

of employee-hours actually used to complete a FU SL LFT&E task is not tracked in

detail. If, however, there are significant changes made in the tasks required of SLAD

employees, the budgeted reimbursements are adjusted to reflect the difference in hours

needed for completion of assigned tasks. Labor costs associated with the additional

hours required are funded frequently with a combination of PMO and mission dollars.

Costs incurred for materials (e.g., experimental materials in modeling) required in

SLAD tasks are included in dollar amounts requested by SLAD for PMO

reimbursement. The significant portion of the SLAD costs, however, is for employee

services (including the overhead administrative costs related to SLAD division and

directorate offices).

3.4.2 Cost of Test CenterActivities. There are three major LFT activities performed

by the test center: (1) planning for the FU SL LFT, including preparation of the DTP,

(2) execution of the FU SL LFT, including instrumentation and target repair and

maintenance, and (3) preparation of the DTR. In addition, test center personnel serve as

members of the LFTPT led by EAC and the damage assessment team led by SLAD. The

test center assembles a dedicated test team, headed by the test director and specifically

trained in the operation, maintenance, and repair of the test articles. The test team

25
indudes data coUectors who provide the required test data to the test director for

indusion in the test report.

The test center typically is involved in planning activities for several years prior to

the actual execution of the FU SL LFT. Its planning activities include developing or

procuring unique instrumentation as needed for test execution, investigating and

developing techniques to fire unique and/or foreign ammunition in accordance with

LFT objectives, coordinating training necessary for test execution events, and handling

logistics of obtaining and maintaining test assets, spare parts, etc. As part of its

partidpation in the planning for the LFT program, the test center provides cost and time

estimates for each of the test center activities that make up the FU SL LFT&E.

As an example of the costs of a (TECOM) test center, ATC costs are discussed in the

following paragraphs. The FU SL LFTs of the Abrams M1A2 and the ATACMS Block

IA (arena tests) were conducted at ATC at APG. The Bradley A3 FU SL LFT, begun at

ATC during the month of December 1998, is scheduled to be completed in July 1999. In

addition to test ranges, ATC provides technical shops, performance measurement

laboratories, and instrumentation and analysis laboratories. Employing

instrumentation designed to evaluate crew survivability, ATC personnel measure crew

incapacitation parameters as part of the tests of vehide vulnerability.

The assigned ATC test director has the responsibility to develop a test plan that

meets the technical objectives of the customer at the least possible costs. In preparing its

cost estimates for the PMO customer, ATC reports that it conducts a detailed analysis of

labor and material costs, relying on historical cost data, expert opinion, and relevant

cost estimate methodologies.

ATC is reimbursed by the PMO for costs incurred in FU SL LFT on the bases of the

activities required by the LFT program (e.g., setup, repair, analysis, completing test

report) and the costs of materials and services required to complete those activities. In

computing the expected costs for a specific activity to be performed as part of the FU SL

LFT program, the ATC project manager (1) identifies all tasks and subtasks required to

complete the activity, (2) determines the skill (e.g., data quality, international imaging,

26
welding) and skill level required to complete the identified tasks, and (3) projects the

number of employee-hours needed to complete all identified tasks. Using the project

management software, the cost estimate for the FU SL LFT program is established by

summing across aU required tasks (1) the respective products of estimated hours

projected for each skill and average wage rate established for the identified skill and

(2) the other nonlabor costs incurred in the task.

ATC charges the PMO customer for labor costs of government and contractor

personnel supporting the test program and the costs of materials and supplies;

petroleum, lubricants, and oil (PLO); travel; and other test-specific requirements as

directed by the PMO. ATC accounts for the cost of each employee's service by

multiplying the actual hours worked by the ATC employee engaged in a FU SL LFT

task by the specific wage rate for that employee's services. The wage rate is computed

by adding the employee's hourly wage rate (i.e., basic salary with fringe benefits) to the

hourly test support distributive rate identified for services rendered at ATC. The test

support distributive rate is established by the ATC budget office at the beginning of the

fiscal year by determining the relationship of estimated indirect costs of testing for the

period (e.g., testing utility costs, maintenance contracts for support vehicles, hazardous

waste disposal, ammunition handling, printing, range control, test support supervisors

above the working leader level, costs of range control) to projected direct labor hours to

be performed by ATC employees for the period. As Department of Defense (DoD)

customers, PMOs do not incur any additional charges for the general and

administrative costs of ATC operations.

The PMO pays for interruptions and shutdowns that are necessitated by its system,

and repeated startup and shutdown cycles increase the costs incurred by the PMO over

and above budgeted amounts. In general, ATC attempts to minimize the costs of the

PMO by reassigning those support personnel originally designated for the FU SL LFT of

a system to other jobs when shutdowns occur.

Vehicle and crew instrumentation (e.g., instrumentation to measure ballistic shock,

blast overpressure, crew acceleration, toxic gases, and thermal radiation) costs include

only the costs of labor associated with the instrumentation activities, unless the

27
instrumentation is destroyed or damaged during the test (e.g., fire consumes a TV

camera in a test article following shot). Expendable supplies used in testing (e.g.,

wiring, connectors, tubing, transducers) are charged to the PMO. If instrumentation

needed in test execution is not owned by ATC, ATC investigates the feasibility of

borrowing or renting that instrumentation. If attempts to obtain instrumentation by

other means are not successful, ATC purchases the instrumentation and charges the

cost to the PMO. Training costs that are generic to instrumentation for multiple systems

are budgeted into the test support distributive rates of ATC, but system-specific

training costs must be directly reimbursed by the PMO of the system.

ATC provides the PMO with a formal cost estimate prior to test execution and

requires the PMOs to front-forward the budgeted costs of ATC activities (i.e., pay prior

to performance of FU SL LFT activities) and incrementally fund the program per fiscal

year. If the formal cost estimates funded by the PMO exceed actual costs, the PMOs are

allowed to withdraw the unused funds or apply those funds to a follow-on test

program, an upgrade in ATC's facilities, or the purchase of materials or instruments for

a future test program.

ATC reports ongoing LFT program costs to the PMO upon request. Costs are

prepared with a software package that computes costs incurred to date on the basis of

activities completed and ATC assigned costs for actual employee services rendered and

materials used. The liaison officer of the PMO has direct access to the computer system

(i.e., TRMS) for tracking costs.

3.4.3 Cost ofHQ,TECOMActivities. HQ, TECOM oversees the coordination of

the DTP and the DTR of systems designated to be tested by TECOM centers and is

responsible for the review of the DTP and the DTR for LFTs assigned to non-TECOM

test centers. In addition, HQ, TECOM personnel serve on the LFEPT led by EAC. The

TECOM test manager assists the evaluator and the PMO in the development of the

LFT&E strategy, the resolution of LFT&E issues such as scheduling and test

configuration, and the execution tasking to other test centers. With the exception of

limited travel costs, costs incurred by HQ, TECOM are personnel costs and are part of

28
the HQ,TECOM budget. These costs are not reimbursed by PMO funds. Travel costs

related to LFT program activities are reimbursed by the PMO.

3AA Cost of USAOC&S Activities. The major FU SL LFT&E activity conducted

by USAOC&S is BDAR analysis, in addition to participation on both the LFIPT led by

EAC and the damage assessment team led by SLAD. In an effort separate from the

damage assessment effort led by SLAD, USAOC&S conducts the BDAR analysis to

determine the ability of the system tested to be returned to an operational status after a

hit from threat munitions (i.e., vulnerability test) or the ability of the threat vehicles (i.e.,

targets) to be returned to operational status after a hit from the system tested (i.e.,

lethality test). Reimbursement by the PMO to USAOC&S for BDAR analysis and

participation in damage assessment, activities generally performed by military

personnel, is made only for travel-related costs required to accomplish these activities.

3A.5 Cost ofEACActivities. The three major LFT activities performed by EAC are

(1) serving as lead for the LFIPT, working group under the T&E IPT; (2) preparation of

the EDP, an independent evaluation of the FU SL LFT; and (3) preparation of the SER.

PMOs are responsible for all costs related to FU SL LFT&E except for the costs of those

organizations that have mission-related responsibilities in the LFT&E program,

including EAC as a division of OPTEC. EAC reports that PMOs are "taxed" a certain

amount for evaluation activities performed for certain Acquisition Category (ACAT)

systems. The dollar amount in the PM budget line designated for OPTEC activities is

determined at the Pentagon level and is part of the original allocation of funds to the

PMO. PM personnel report that funds required by EAC and Operational Evaluation

Command (OEC) divisions to perform evaluation activities relevant to their systems are

part of the PMO budget, and funds are distributed to these agencies during the fiscal

year.

3.5 Benefits of FU SL LFT&E. DoD Regulation 5000.2-R reports [9]:

The objective of LFT&E is to provide a timely and reasonable assessment of

the V/L of a system as it progresses through its development and prior to

full-rate production. In particular:

29
• to provide information to decision makers on potential user casualties,

vulnerabilities, and lethality, taking into equal consideration

susceptibility to attack and combat performance of the system;

• to ensure that knowledge of user casualties and system vulnerabilities

or lethality is based on testing of the system under realistic combat

conditions;

• to allow any design deficiency identified by the testing and evaluation

to be corrected in design before proceeding beyond

low-rate initial production; and

• to assess battle damage repair capabilities and issues (... not a statutory

requirement of LFT&E,... assess such capabilities whenever prudent

and affordable).

Supporters of mandated testing suggest there are significant benefits associated with

requiring FU SL LFT&E for major U.S. weapon systems. Discussion of the rationale of

proposed benefits is presented in the paragraphs that follow.

• FU SL LFT&E provides an opportunity to identify unknown vulnerabilities of

the system, thus allowing design or engineering changes to be made prior to the

system entering full-scale production.

• FU SL LFT&E provides insights into BDAR potential.

• FU SL LFT&E provides an opportunity to validate M&S tools developed by

analysis groups.

• FU SL LFT&E provides valuable input to doctrine, tactics, and training decisions

of system users.

• FU SL LFT&E requirement helps to ensure adequate consideration is given to

V/L issues throughout the acquisition cycle.

3.5.1 FU SL LFT&E and System V/L. The General Accounting Office (GAO) reports

that FU SL LFT&E provides the opportunity not only to identify unknown unknowns in

the tested system, but also to gain insight into the system's known unknowns that are

30
suspected but not well understood. The GAO researchers propose that FU SL LFT&E is

the only V/L assessment method that provides direct "visual observation of the

weapon/target interaction under realistic combat conditions [10]."

In general, FU SL LFT&E provides an opportunity to gain insights into (1) the

synergistic effects of multiple damage mechanisms, (2) the phenomena of blast and

shock, (3) the effects of fire, toxic fumes, and the fire suppression system, (4) the effects

of secondary and cascading damage (e.g., secondary debris and ricochet) not revealed

in component or subsystem testing, (5) the effects of noncritical component damage on

critical components, and (6) the effectiveness of redundancy and other vulnerability

reduction measures with respect to continued operation of the system tested [11-14]. In

addition, LFT provides data to support target signature impact research and the

quantification of radiation hazard in target vehicles.

Proponents of FU SL LFT&E argue these insights can lead to the correction of errors

and defects through redesign and engineering changes of the weapon system at the

component, subsystem, and system levels, thus decreasing the potential for personnel

casualties, system and environmental losses and increasing the probability that the

system will be able to complete its mission when operating in a hostile environment.

Mr. James F. O'Bryon, DDOT&E (LFT), believes LFT&E motivates decision-makers to

find feasible solutions for V/L deficiencies [11]. He cites multiplefixes that have been

implemented as a result of FU SL LFT&E, including the shielding of components, the

adjustment of trigger thresholds, and the revision of stowage compartments.

3.5.2 FU SL LFT&E and BDAR Activities. Although not a statutory requirement of

LFT, FU SL LFT&E provides valuable input to decisions relative to BDAR activities. FU

SL LFT&E affords insights into the adequacy of current stock levels of spare parts and

the effectiveness of current BDAR and troubleshooting procedures [10,15,16]. BDAR

activities related to FU SL LFT produce experiences from which new repair techniques

and procedures are able to be developed and provide realistic time estimates for

returning damaged vehicles to combat.

31
3.5.3 FU SL LFT&E and M&S. FU SL LFT data provide input to the calibration/

validation of lethality and vulnerability models. Because these V/L models are used to

develop pre-shot predictions for FU SL LFT&E and may be used to extend the results of

testing, the data provided by the FU SL LFT&E of any one system may be significant to

the FU SL LFT&E of future systems. In the past, critics of FU SL LFT&E conceded that

tests may disclose valuable V/L issues that had been overlooked in existing M&S tools

(e.g., toxic fumes or penetrator break-up in BFVS) but submitted that FU SL LFT&E data

were unable to provide information of significant value in the validation of models [10,

17]. In recent years, however, FU SL LFT&E data have provided the bases for

validation activities. For example, the entire validation of the MUVES-S2 model for the

Bradley A3 LFT was based on the results of the Bradley High Survivability (HS) LFT,

and the validation of the SAFE model for the ATACMS LFT&E was based on the results

of the arena tests that were part of the LFT program. Similar plans exist for validation

of models to be used in the FU SL LFT&E of the M1A2 System Enhancement Package

(SEP). Data collected from the LFT activities of one system may provide useful data for

the V&V of models to be used in the T&E activities of similar systems or later models of

the same system.

3.5.4 FU SL LFT&E and Users' Decisions. FU SL LFT&E provides valuable input to

the development of both the procedures for preparing crew and system for battle and

the doctrine and tactics for a system's operations in hostile environments [13-15]. Data

prepared for FU SL LFT incidents furnish information for training simulators and

operational tests' assessments [14,15].

3.5.5 FU SL LFT&E and Survivability/Lethality Assessment Programs. Because

the results of testing cannot be the sole basis for V/L assessment, it is important that an

appropriate weight is given to V/L issues in the design and engineering phases of the

system's development [17]. Supporters of FU SL LFT&E suggest congressionally

mandated LFT&E helps to ensure adequate attention is given to V/L issues throughout

the acquisition cycle by the PMO and contractor(s) [12,13].

Proponents of FU SL LFT&E question whether the discipline to proceed with a

well-organized program of V/L assessment would continue should FU SL LFT&E

32
requirements be terminated. Mr. Thomas Julian, Action Officer, DOT&E (LFT), reports

the results of a recent survey of PMs, testers, and analysts indicate the majority of those

surveyed believe "the old way of doing business" would return if FU SL LFT&E were

no longer required of U.S weapon systems prior to entering full-scale production [15].

The high visibility of the FU SL LFT&E mandate appears to provide a certain degree of

motivation for PMOs and contractors to consider V/L issues over all phases of

acquisition [17]. Some suggest this high visibility also helps to ensure a higher level of

congressional funding for V/L assessment activities than may be provided for

alternative strategies of assessment that do not include the FU SL LFT element.

3.6 Project Manager Offices: Impacts of FU SL LFT&E. Visits to the ATACMS

PMO, AMCOM, Redstone Arsenal, AL, and the Abrams and Bradley PMOs, TACOM,

Warren, MI, were part of the information-gathering process for this study. In each

location, members of the PMO participated in discussions relevant to the programs of

survivability/lethality in operation for their respective systems and the role of FU SL

LFT&E in assessment of system V/L. Prior to visits with PM personnel, questions

concerning the system and survivability/lethality assessment were sent to the PMOs.

Discussions focused on the responses of the PM personnel to those questions and topics

that arose from their responses/

Discussions with PMO personnel covered a wide range of topics, including:

• objectives of the research study,

• current methodologies requiring the identification and measurement of the

impacts of competing T&E strategies,

• organization of the PMO and the role of PMO in system acquisition,

The views of PM personnel described in this report were not expressed by all persons in all three PMOs.

Although a group of core topics formed the foundation for all PMO discussions, there were some topics

explored in only one or two offices. In general, personnel in any one office did not disagree with the

views expressed by other persons in that same office participating in the discussion. Following each

PMO visit, PMO personnel were sent a summary statement of the questions asked and the responses

recorded during the visit and were asked for comments. PM personnel were also given an opportunity

to review this part of the report (i.e., Project Manager Offices: Impacts of FU SL LFT&E) prior to

submission.

33
phases of acquisition and the activities relevant to each phase,

history of live-fire testing of earlier models of the identified system,

contracts for all phases of system development and production, including use of

CAIV and target costing in contracts,

production history of current/earlier models of the system and related costs,

effects of foreign military sales of system on contractor charges to government,

responsibilities of the contractor and government agencies in developmental and

operational testing,

budgeting for FU SL LFT&E and follow-on tests,

activities of FU SL LFT&E and the role of the PMO in those activities,

costs incurred in FU SL LFT&E, reported and unreported,

design of the FU SL LFT&E,

role of M&S in V/L assessment, and

benefits of FU SL LFT&E or lessons learned from FU SL LFT&E.

Much of the specific system information gathered from the PM discussions has been

incorporated into other sections of this report. This part of the report focuses on the

perceptions of PM personnel relative to the costs and benefits of FU SL LFT&E as one

part of the survivability/lethality assessment process and the suggestions of PM

personnel for improving the cost-effectiveness of the assessment process.

Discussions related to the impacts of FU SL LFT&E emphasized four main topics:

(1) the objectives of FU SL LFT&E, (2) the effect of the system's existing program of

survivability/lethality evaluation on the design of the FU SL LFT, (3) the alternatives to

34
FU SL LFT&E, including M&S, and (4) the lessons learned from a completed FU SL

LFT&E.

3.6.1 Objectives ofFU SL LFT&E. Guidelines for LFT&E cite the importance of

carrying out tests sufficiently early in the development phase to allow for design

deficiencies to be detected and corrected in the weapon system tested. Objectives of

LFT&E include (1) ensuring that knowledge of user casualties and system

vulnerabilities and lethality is based on realistic testing of the system configured for

combat against expected threats and (2) gaining insights into potential design flaws so

that they can be corrected before the system enters full-rate production [9].

In discussions with PM personnel, questions were raised as to whether FU SL

LFT&E, conducted at the conclusion of the Engineering and Manufacturing

Development (EMD) phase, occurred too late in the acquisition process to impact

design or secure (relatively) inexpensive fixes. The ATACMS PMO argued that if a

discovery were made in a FU SL LFT relative to the adequacy of the system's bomblet

lethality, subsequent changes in the bomblet would affect the entire production line.

Changes in design made late in the production cycle would impact all qualitative tests

completed prior to the design change and, in turn, would impact the system evaluation

required prior to the system entering full-scale production. PM personnel emphasized

that problems related to design and engineering requirements need to be forecasted as

early in the acquisition process as possible. FU SL LFT is often too late to afford

effective fixes in a cost-efficient manner.

3.6.2 Design ofFU SL LFT&E. Does the system's existing program of

survivability/lethality affect the plan of the FU SL LFT&E? Is the scope of the FU SL

LFT plan more restricted (i.e., less shots) for a system that has a well-developed agenda

for evaluating system survivability/lethality built into its acquisition program than the

scope of the FU SL LFT plan for systems without such an agenda? Is the FU SL LFT of a

system with a building-block program of survivability redundant (i.e., a series of tests

repeated at an independent location)?

35
Abrams PMOs believe the well-managed survivability test program associated with

Abrams systems addressed most of the vulnerability issues prior to the MlA2 FU SL

LFT&E. The MlA2 FU SL LFT&E in 1993 confirmed the PMO conviction that the M1A2

was a very survivable tank based on the limitations of its 69.9-ton gross weight. The

Abrams' PMO reports that from its initial concept, the Abrams was developed with

survivability as its first priority, and the results of the 1993 FU SL LFT came as no

surprise to the PMO. In its different variants, the Abrams has undergone extensive

survivability testing, including tests of its armor, structures, and ballistic hull and turret

(BH&T), as well as system-level tests not under independent congressional LFT&E

auspices. PM personnel questioned whether congressionally mandated FU SL LFT&E

was not a very expensive final check on crew and tank survivability under live-fire

conditions. Several interviewees voiced the belief that when EMD includes a welldesigned and well-managed survivability/vulnerability program, the results of FU SL

LFT&E are predictable and the likelihood of major surprise outcomes is small.

3.6.3 Alternatives to FU SL LFT&E. Individuals interviewed acknowledge that a

LFT program provides insight into the areas of component, subsystem, and system

vulnerability/lethality; damage and casualty mechanisms; perforating and

nonperforating impacts on the system; and system- or threat-specific subissues such as

compartmentalization, stowed munitions, fuel and hydraulics, fire suppression

systems, and blast, shock, and incendiary effects. The question, however, remains,

"Are there more economical ways than FU SL LFT&E to obtain the information needed

by decision-makers or to discover the unknown vulnerabilities?" Is FU SL LFT&E

designed with the objective of obtaining information needed by decision-makers that is

not available from other sources?

Abrams PM personnel provided an example in which off-line tests were conducted

to verify the design of two very important survivability fixes. The tests consisted of

seven high-caliber shots on an earlier model with fixes incorporated. The total test cost

was reported to be approximately 25% of the cost of one FU SL LFT&E shot. The

question then becomes, "If resources are limited, is adequate consideration given to

alternative methods (i.e., methods other than FU SL LFT&E) to gather information

required for assessment decisions?" Interviewees agreed that the response to this

36
question is different for each FU SL LFT&E. Citing recent technological developments,

Abrams PM and Bradley interviewees noted that in the survivability programs of

earlier models, some system-level effects were only testable in FU SL LFT&E. Currently,

however, some of the same effects can be assessed in off-line tests (e.g., effects of blast

and shock in WSMR air blast tests).

A second issue raised in discussions concerned the balance of engineering-shot lines

(i.e., shots selected to address specific V/L issues) and random-shot lines (i.e., shots

generated from likely hit points) in the FU SL LFT program. Interviewees suggested

that controlled damage tests (CDTs) and the engineered shots of FU SL LFT&E often

produce the data that decision-makers require with minimum test asset repairs.

Personnel expressed frustration that in some situations the PMO is aware of and

concedes damage with planned random shots, but the random shots remain part of the

FU SL LFT&E plan. Frustration was also expressed that attempts by the PMO to

modify configurations of the test items to eliminate expensive line replacement units

were often unsuccessful. Interviewees hastened to add that they understood random

shot lines were a part of the realistic combat setting required in FU SL LFT and that a

balance of random and engineering shots was important to the perception of a fair test

of the system's V/L. Concern was expressed, however, that many more dollars are now

spent on random shots than on engineering shots in which more useful information is

often received.

In particular, PM personnel were asked if they believed M&S could serve as an

effective substitute for live-fire testing in the assessment of the V/L of a weapon system.

Interviewees were in agreement that both M&S and testing are needed in V/L

assessment—that they serve as complements to rather than substitutesfor each other.

They emphasized that testing is required in the development of M&S tools to ensure

M&S robustness and fidelity; in turn, M&S is needed to extend the results of testing to a

broad range of weapon system-target encounters.

Although all interviewees expressed the opinion that significant progress in M&S is

needed before any substantial cuts in testing would be feasible, interviewees expressed

the hope that M&S would eventually be mature enough to allow substantial reduction

37
in the required number of FU SL LFT shots.* They cited the important role that M&S

plays in the design of the system (e.g., layouts of interior or placement of armor) and

the advancements in modeling of damage mechanisms (e.g., penetrating shots). They

strongly encourage testers and modelers to continue to work together to ensure that

testers understand the type of information that is required for building models.

ATACMS PMO cited the valuable input provided by the tire tests (i.e., tests related to

the ATACMS Block IA LFT) to the subsequent modeling of tire effects. In a similar

manner, ATACMS PM personnel believed that ATACMS follow-on tests would

contribute to the modeling of fire effects.

Although all groups supported the effort to devote more dollars to the development

and improvement of M&S tools, all were quick to point out that the funds available for

testing and M&S are limited and that tradeoffs are a fact of life. Several interviewees

suggested more effort should be made to integrate modeling and testing to provide

information needed for decision-makers in a more cost-effective manner. Views were

expressed that even with its current limitations, M&S can be combined with

nondestructive testing to produce valuable data. All realized that this is a complex

situation, as funding of additional tests and experiments to improve the fidelity of

models often leads to more questions that must be resolved by additional testing.

In general, there appeared to be consensus across the groups that two major issues

related to M&S remain to be resolved: (1) how to obtain the financial resources needed

for the development of high accuracy M&S tools and (2) how to ensure the acceptability

of an increased role for M&S in the assessment of survivability/lethality of a weapon

system.

3.6.4 Lessons Learnedfrom FU SL LFT&E. Although it is difficult to measure the

benefits of FU SL LFT ex ante, it may be helpful to look at the lessons learned from a

* Some argue that a substantial reduction in the required number of shots has already taken place for a

variety of reasons. For example, Phase II of the FU SL LFT of the Bradley M2 and M3 in 1986-1987

included over 80 shots (i.e., random, prescribed, and conceded/repeated shots, as well as repeated

Phase I and Development Test shots), as compared to the scheduled 18 shots of the 1998-1999 Bradley

A3 FU SL LFT.

38
completed FU SL LFT&E program as an ex post measure of the effectiveness of the FU

SL LFT activity. Interviewees in the PMOs of the two systems in this study that had

completed a FU SL LFT&E were asked about their perceptions of the benefits of the test

and what actions were taken in response to system deficiencies discovered in the test

series.

ATACMS PM personnel reported that, while the FU SL LFT for the ATACMS Block

IA did not answer all questions about the effectiveness of all M74 bomblet damage

mechanisms, it did point out the potential for product improvement. It also pointed out

the need for more modeling and analysis in the characterization of the bomblet earlier

in the acquisition process. At the time of the interview, the ATACMS PM office was

planning follow-on tests (required before full-scale production could begin) to further

examine the effectiveness of the M74 bomblet against potential target elements.

Abrams PM personnel reported that the MlA2 Abrams FU SL LFT verified some

fixes on problem areas discovered on prior FU SL LFTs and indicated some new

vulnerabilities that could be eliminated or minimized with modifications. Problem

areas that surfaced were reported to be small. Personnel pointed out that two prior

fixes had been verified on off-line tests related to LFT&E and that FU SL LFT basically

confirmed what the PMO already knew about the M1A2. They believe the results of the

FU SL LFT support their belief that the Abrams survivability test program previously

addressed most of the vulnerability issues of the weapon system. Personnel estimated

that 80-90% of data uncovered in 1993 FU SL LFT could now be discovered in a test

program that did not include FU SL LFT. Personnel acknowledge, however, that some

of the technology that is available at this date was not available at the time of the MlA2

Abrams FU SL LFT. For instance, the use of shock simulators currently affords a

method of testing off-line that was unavailable in 1993.*

* The ballistic shock simulator developed by ATC is not a replacement for shock testing on a full-up

vehicle. Although useful in identifying weak components prior to FU SL LFT, the simulator offers no

guarantee that a component that does not fail on the ballistic shock simulator will survive the FU SL

LFT.

39
Abrams personnel reported that potential fixes are briefed as part of MS m

deliberations but that modifications or fixes are determined not solely by the PMO but

by an oversight committee with input from system users and trainers, system

developers, and logistics personnel. Decisions relative to which fixes are implemented

are made on the bases of the priorities assigned fixes uncovered in testing and the

availability of funds. Often, corrective actions are incorporated into later designs of the

system.

The Abrams MlA2 FU SL LFT&E resulted in 12 recommendations, including 9 fixes

that were initiated for production retrofit or incorporation into future tank design,

2 that were reviewed but not implemented, and 1 that was not addressed. In addition,

some LFT insights were incorporated into the tactics, training, and planning of the

system user to improve the survivability of system personnel in the combat

environment.

3.6.5 Suggested Actions. In discussing the effectiveness of FU SL LFT&E and the

lessons learned from tests completed, PMO personnel offered the following suggestions

to improve the cost-effectiveness of survivability/lethality assessment and, in

particular, the cost-effectiveness of FU SL LFT&E as part of the assessment process:

• Enlist multiple supporters for the development and V&V of M&S tools (OSD,

PMO, etc.). Encourage modelers and testers to integrate efforts.

• Provide additional funds in the early phases of acquisition to allow up-front

discussions of M&S with SLAD personnel; currently, PMOs are unable to fund

this activity prior to the establishment of user requirements.

• Organize a round-table discussion in the early stages of the acquisition process

for ARL personnel and all parties of T&E activities to consider carefully

alternative methods of obtaining information needed by decision-makers in the

assessment of system V/L.

40
• Enlist involvement of the evaluator very early in the T&E process, using a team

approach with the evaluator as a member of the team. PM personnel suggest the

oversight of government evaluators in contractor testing allows the positive

results of those tests to be included in the government evaluation. PM personnel

noted that this cooperation has been more common in recent LFT programs than

in prior test programs.

• Consider limiting the scope of the FU SL LFT&E of those systems managed by

PMOs with well-established building-block programs of survivability/lethality

assessment in place. Scope limitation could serve as an incentive for

incorporating V/L assessment into all phases of the acquisition cycle.

• Consider carefully the balance of engineered and random shots in FU SL LFT,

weighing the cost incurred for information obtained. Consider alternative tests

or forms of analysis and the costs associated with those tests and analyses in

designing the number and types of shots in the FU SL LFT&E strategy.

• Consider granting concessions in FU SL LFT&E. In years prior to the

congressional LFT&E mandate, critical components of a system often were

removed before testing in FU SL LFT, if the PM office conceded the system

would fail with hit. Some PM personnel questioned the cost-efficiency of FU SL

LFT congressional test requirements that mandate that critical components

remain in the test asset.

PMO personnel explained that some of their problems with FU SL LFT&E originate

with the fact that PMOs focus on demonstrating that a system is in compliance with the

ORD for the system, but FU SL LFT&E often is structured to evaluate a system beyond

its requirements. Therefore, PMOs frequently find it difficult to fully support LFT&E

and the incorporation of design fixes that the FU SL LFT indicates are needed for the

system that has been evaluated beyond its requirements.

In conclusion, it is important to emphasize that the PMOs visited did not discount

the benefits of an LFT&E program. Interviewees stated that changes in materials and

41
technology in recently produced weapon systems require a program of LFT&E and that

alternative tests may not catch everything found in a FU SL LFT. Most of the above

suggestions offered by PM personnel attempt to address the question of how limited

resources may be employed to design the most effective plan for assessing the

survivability/lethality of a weapon system.

4. Framework for Vulnerability/Lethality Assessment

In the current era of constrained resources, the effectiveness and efficiency of a V/L

assessment strategy may be increased by the identification of the specific set of data

required for assessment, the determination of the subset of the required data currently

available from reliable sources, and the selection of a cost-effective assessment plan (i.e.,

M&S, experimentation, FU SL LFT&E, etc.) to fill the data voids. Although the

approach described in the following paragraphs is applicable equally to lethality

assessment, the discussion offered in this part of the report is limited to vulnerability

assessment.

In assessing the vulnerability of a weapon system, analysts evaluate the system's

likelihood to sustain personnel casualties, suffer catastrophic losses, or fail to complete

its mission when faced with those threats deemed likely to be encountered in combat.

Faced with decreased defense budgets, it is important for analysts to employ an

assessment strategy that allows vulnerabilities to be detected as early in the acquisition

process as possible, thus affording the maximum time to develop fixes prior to the

system entering full-scale production. An effective vulnerability assessment program

begun early in the acquisition process has the potential to detect system vulnerabilities

and reduce or minimize not only the costs associated with combat losses but also the

costs associated with retrofit of systems in the late stages of production.

* Fixes associated with vulnerabilities detected late in the acquisition cycle are less likely to be able to be

achieved through redesign or re-engineering of the system and more likely to be associated with the

higher costs of retrofitting of completed systems. Serious vulnerabilities detected late in the acquisition

cycle could lead to system abandonment. Although it is acknowledged that not all vulnerabilities can

be eliminated, knowledge of a system's vulnerabilities may affect users' tactics and strategies in combat.

42
Limited resources have led to an increased emphasis on the costs as weU as the

benefits of conducting vulnerability assessments of weapon systems. It is proposed in

this section of the report that an effective vulnerability assessment strategy identifies the

specific set of data required for vulnerability assessment, determines the subset of the

required data currently available from reliable sources, and then designs a vulnerability

assessment plan (i.e., M&S, experimentation, FU SL LFT&E, etc.) to fill the data voids.

Alternative vulnerability assessment plans, designed to provide the missing critical

data, must be compared on the bases of (1) the capacity of each plan to provide the

critical data (i.e., benefits of plan), (2) the cost to accomplish each plan, and (3) the

uncertainty (risk) associated with each plan's capacity to provide the critical data at the

identified cost within the time constraints of the assessment process/

An effective vulnerability assessment strategy includes both an analysis of the likely

combat scenarios and a determination of the links between the measures of battlefield

effectiveness and the measures of capability required of the system to complete its

mission. In turn, the vulnerability assessment plan is designed to fill the data voids that

exist in the analyses of (1) the relationship between the threat-system interaction and

the collection of components associated with system and personnel vulnerability that

have been damaged and (2) the relationship between this collection of damaged

components and the measures of capability required by the system to complete its

mission.

The V/L Taxonomy is suggested as the appropriate framework for identifying the

information required in the vulnerability assessment process, as well as the data voids

to be addressed in the vulnerability assessment plan [2]. A description of the

methodology for the design of cost-effective vulnerability assessment plans and the

selection of the optimal plan from those alternatives follows a review of the Taxonomy.

4.1 Taxonomy of the V/L Analysis Process. The V/L Taxonomy provides a

framework for examining the elements of the complex vulnerability assessment process.

Time constraints must consider the availability of hardware required to complete test plan components.

43
The V/L Taxonomy was first introduced by Deitz and Ozolins in 1989 as a conceptual

framework for armored fighting vehicle vulnerability assessment in association with the

computer simulation of the Abrams Live-Fire Field Testing [2]. The general description

of the V/L Taxonomy that follows is based on the cited publication plus several

subsequent publications that have discussed the analytical properties of the V/L

Taxonomy [2,18-24].

The V/L Taxonomy reflects the vulnerability analysis process as a sequence of

several levels, each containing specific information about the analysis process.* The

information in one level is mapped to the information in the subsequent level by a

transformation or a mapping process. A description of the taxonomy includes four

levels with associated spaces+as shown in the graphical representation found in

Figure 1.* Mapping from higher to lower levels (i.e., Level 1 to Level 2; Level 2 to

Level 3; and Level 3 to Level 4), the taxonomy can be described as follows:

• Level 1 defines the initial configuration of the specific threat and weapon

system (i.e., target for which vulnerability is assessed) encounter. Space 1

includes the set (i.e., all combinations) of the initial configurations of the threat

and system just prior to threat-system interaction.

• Level 2 defines the physical state or the damage state of the weapon system that

results from the interaction of the threat and system. Space 2 includes the set

(i.e., all combinations) of damage states. Each state, represented by a point in

Space 2, is generally expressed as an n-tuple with elements describing the status

of certain specific n number of components of interest. In the V/L Taxonomy,

* The V/L Taxonomy also applies to lethality analysis. In this report, however, only vulnerability

analysis is addressed.

+ A level includes all the information necessary to define the state of the system at the associated stage of

the vulnerability analysis. A space of points may be defined at each level to represent the state of the

system at that level. Each point in a defined space is a vector with a specific number of elements, each

element referring to the status of an entity related to the system (e.g., system components in Level 2).

There may, however, be multiple spaces at each level, each space defined by a different set of elements

in the vectors that make up that space at that particular level [23].

* Figure 1 is adapted from Figure 1: V/L Taxonomy Illustrated via a Mapping Abstraction, as presented

in Deitz and Starks [19].

44
O•i-i

5H

u

H

cn

QJ O

Ö0-J3

*H .T5

°^ £ ns *J O

QJ QJ w

^ H -H

o

s s

T3

QJ

bC

en

ß £

m 22

Q o

Fi DH

5H

3

fci

o

U

i—H

PH

QJ ü

(8

6eö

Q

QJ

bO

PH»H

»H

o

«s

(-1

QJ

C

C ^

O "^

6 I

S.P O

1 « r j +3

QJ

• 1—1 i_ *j cn JJ *J >~,

o « SJ ^ « .-ä

O, 15 ^32®5

U

o

*H ffl Cß «S S •—1

o Sus

ÄS

llilp

»

ir

OH->

•i—i

5H

U

«3

a>

Qi

C/j

"öS

>

1 _|

1

<»

c

T—1

QJ

bC ?SHo

o

T3

QJ H »4-1

o

U

13

>

QJ

QJ

-2

"DH

T *H d O

^-v CO

•ä» bo* 2 o

CN

'•3i—i

-

Q

3 P, u QJ

1

QJ

b0

)Hm y—"N

H s T" ' 1 *H <* o o >,

M-l H-"

QJ CO

QJ

5H

r plat

Utili

CC o

QJ

o

O

X

«8

H

0)

4H

o

c

c

I»

5H

DM

tu

f-H(«U

2

(8

5H

ü

2

5*

45
each Level 2 state is expressed as a damage state vector with each element of the

vector representing the status of a component following the threat-system

interaction.

The transformation of the undamaged system of Level 1 to the damaged system

in Level 2 (Ou) is a physical process and is characterized by physical damage

mechanisms, such as threat penetrators, fragments, shock, or fire. The process of

mapping a point in Space 1 to a point in Space 2 is generally stochastic, and a

single point in Space 1, representing a threat-system configuration, may map to

multiple points in Space 2 with each point in Space 2 represented by a different

damage state vector. Thus, identical shots by a specific threat against a specific

system, when repeated, often result in different damage state vectors for the

targeted system. It is important to note that multiple points in Space 1 may map

to the same single damage state vector in Space 2 [19,20].

Level 3 defines the capability state of the weapon system that results from the

threat-system interaction. Space 3 includes the set (i.e., all combinations) of

capability states. Similar to the damage state of Level 2, the capability state of

Level 3 is expressed often as an m-tuple with elements describing the status of

certain specific m number of capability or performance measures (e.g.,

capabilities related to firepower, mobility, communications, etc.) relevant to the

system's mission.* Each state represents a point in Space 3 and is expressed as a

capability state vector with elements representing specific levels of identified

capabilities of the system. The capability state vector represents the aggregate

capability of the system to function in areas relevant to the system's mission

following threat-system interaction.

The transformation of a damage state vector in Space 2 to a capability state vector

in Space 3 (O^) is a mapping of damaged components to the resulting reduced

capability state. It is important to note that although a single damage state vector

* In this report, capability (not performance) measures are used to describe the metrics of Level 3 of the

V/L Taxonomy.

46
in Space 2 maps to a single capability state vector in Space 3, multiple damage

state vectors in Space 2 may map to the same single capability state vector in

Space 3.

• Level 4 defines the operational utility state of the weapon system that results

from the aggregate system capabilities measured in Level 3. Data from Levels 2

and 3, combined with information relative to the combat scenario (i.e., threat

environment, physical environment, and mission objectives) and users' tactics

and doctrine, are used to determine the mission status points or the measures of

effectiveness (MOEs) included in Space 4.

The transformation of the capability vector in Space 3 to the operational utility

state in Space 4 (C\4) is often perceived as outside of the process of vulnerability

analysis and requiring the input of system users and military strategists. In

reality, this mapping is often included as part of the vulnerability assessment

with input provided by the system user or strategist.

4.2 V/L Taxonomy and Vulnerability Assessment. The V/L Taxonomy presented

in the previous section provides a framework for an analysis of the vulnerability

assessment process. In the vulnerability assessment of a weapon system, analysts ask

the question, "To what extent will the interactions of the weapon system and the threats

that the system is likely to encounter in combat result in personnel casualties (i.e.,

personnel vulnerability) or the inability to complete the system's missions (i.e., system

vulnerability)?"

Before analysts are able to design cost-effective vulnerability assessment plans to

address the aforementioned question, however, they must ascertain what information is

needed to adequately answer the assessment question(s) and determine the subset of

that information that is available from reliable sources. A comparison of the required

information to the subset of available, relevant, and reliable information leads to the

identification of the data voids (i.e., subset of the required information that is

unavailable or unable to be relied upon). The data voids define the information that

must be gathered in the vulnerability assessment plan.

47
4.2.1 The Required Data. To determine what data are needed to evaluate the crew

and system vulnerability to threats likely to be encountered in combat, the analysts

must have a clear definition of personnel and system vulnerability. U.S. Defense

acquisition policies define vulnerability as [9]:

The characteristic of a system that causes it to suffer a definite degradation

(loss or reduction of capability to perform its designated mission) as a result of

having been subjected to a certain (defined) level of effects in an unnatural

(man-made) hostile environment.

In vulnerability assessment, analysts are interested not only in threat-system

interactions that render the crew unable to complete its mission but also in interactions

that result in injuries to the system's crew members and passengers even though the

mission is able to be completed. System vulnerability includes any damage to the

weapon system, including the catastrophic loss of the system, that does not allow for

mission completion. To provide decision-makers with data characterizing the

vulnerability of the system and its personnel, analysts must be able to collect

information relevant to the following questions:

1. What are the likely combat scenarios in which this system will operate?

2. Considering the characteristics of the combat scenario (i.e., threat, physical

environment, and mission objectives) and users' tactics and doctrine, what

activities will the system be required to perform?

3. What enables mission success or completion in the identified combat scenarios?

How is battlefield effectiveness of the system or mission success measured?*

Traditionally, question nos. 1,2, and 3 have been considered to be beyond the purview of the

vulnerability analyst. The methodology described in this report emphasizes the importance of

understanding the relationship between the tasks that must be completed to ensure a successful mission

and the minimum levels of system capabilities needed to complete those tasks in a cost-effective

vulnerability assessment.

48
4. What specific capabilities (i.e., capabilities in areas of mobility, firepower,

communications, etc.) must be evaluated in the assessment of the effectiveness of

the system?

5. What levels of the specific capabilities identified in #4 must exist to ensure

mission completion?

6. What subsystems of the weapon system must be in operation to produce the

levels of capabilities identified in #5 as necessary for mission completion?

7. What are the critical components of the subsystems identified in #6?*

8. What type of damage (i.e., personnel casualties, catastrophic loss of the system,

damage to critical components of the system) to the weapon system is expected

from the interaction of the system and threats likely to be encountered in

combat?

9. Given the damage states identified in #8 and the critical components of the

system identified in #7, to what extent are the system and its personnel

vulnerable?

4.2.2 The Taxonomy of Vulnerability Assessment. A few changes to the

V/L Taxonomy shown in Figure 1 produce the Taxonomy of Vulnerability Assessment

(Taxonomy VA) in Figure 2, an appropriate framework for addressing the preceding

questions/ The Taxonomy VA provides the bases for describing the activities to be

accomplished in the identification of the information required and available for the

vulnerability assessment of a weapon system, as well as the information required but

* A system is composed of subsystems, and each subsystem is a set of components. Critical components

are those components that if lost will result in a degradation of one or more subsystem functions and,

consequently, a reduction in system capability [25].

+ Figure 2 is an adaptation of the Mission-Based Acquisition Strategy, chart by Deitz [26].

49
u

ONOO

CO

HJou

PQ

LO "*

CO

cc

-4-> in

^£s

CO

Q CD

CUc

CO

COu

cu

VC0

OQ

Ü CO COou CO D

ro tc

CO

H

• i—t

^0,52 > u «

«->

CO

CD

>S JS <" a! en ^ CO

CD

>X

CD

cu

en

CCoH->

CO

coB

cc

CO

en

!fi

w

Ü

s_i CC W en

CO

0) ^-0 u 'S G as OH O

OH

en £~ §

en &\S

en

en

<U

finefine

u CD CD 3 DO

cö rl u cti

a;u

O* CO ° £

00&

CO H->

OH eo O CD

CU>

CD O

00

CO

<D Q

CO

CO H->

.a CO

-M x> en ;=! O •43

H->

3

0) fi > <1>

CO

pU COÄ

cc 4J r>u

° OH£ ol-H

CO

CO

0) U

CC

OH

en

II 0

CO

CD

XJ

•l-H <

CO

CO

CD^

CD Q

fit

PH CO Q. O CD ,0 " CD

..>

CD <D

pp

<2> «5 CD

CN CO rt<

CO

CO 4J

a»

*e

£8

CC "-!

LO **£> t>s 00Ä^

CO B

CC

.02

CDC

CO

CC^o

OH

CO

CO

CDu

cc

OHOU

C3<

rooXH

CD

»HP

bo

50
unavaüable or unreliable (i.e., data voids). The activities listed in Figure 2 begin at

Level 4 and are described as follows:

1. Define system utility status (Level 4): Define the likely combat scenarios and the

role played by the weapon system in successful completion of identified

mission(s). Define the measures of battlefield effectiveness available to describe

the system utility or mission status (i.e., mission complete or not complete). The

characteristics of likely combat scenarios (i.e., physical environment, threat

environment, and mission objectives), as well as users' tactics and doctrine

relevant to the system, serve as input to this activity. In Figure 2, the subspace

shown at Level 4 includes those points representing outcomes in Space 4 that are

associated with mission completion/success.

2. Develop mission operators (03J: Develop the operators that map the missionrelevant system capability measures in Level 3 to the MOEs in Level 4. The

mapping operators establish the link(s) between the m-tuple capability state

vectors representing the specific m number of capability measures to be

evaluated, and the MOEs, representing the mission status measures of battlefield

success (i.e.,mission complete or not complete). A single capability state vector

in Level 3 may map to one or more points or to no points in Level 4, depending

on the tasks required in the combat scenarios represented in Level 4. In other

words, a set of aggregate capabilities may lead to mission success in one or more

given scenarios, but that same set of capabilities may not lead to mission

completion in other scenarios that require a different combination of capabilities.

3. Define (mission-relevant) system capability status (Level 3): Define the minimum

level of each of the m specific mission-relevant system capability measures (i.e.,

measures identified in activity #2) that are required for mission completion. This

activity identifies the specific m-tuple capability state vectors in Level 3 (shown as

subspace of Level 3 in Figure 2) that map to mission completion in Level 4 for a

specific combat scenario. Each of the m elements in the capability state vector

reflects the minimum level of the m number of measures of specific capabilities

that must be achieved for mission completion.

51
4. Develop capability operator® (02,3): Develop the operators that map the damage

states in Level 2 to the measures of capability in Level 3. The mapping operators

or degraded state operators establish the link(s) between the damage state vectors,

representing the damage states of the system following threat-system interaction,

and the capability state vectors, representing the capabilities required to

complete the mission(s). This mapping defines the components and subsystems

that must be in operation to deliver the levels of capability identified in

activity #3.

5. Define system physical status (Level 2): Define the specific critical components that

must remain in operation (i.e., not killed) after threat-system interaction to result

in the identified capability levels required for mission completion. Redundancies

among components must be considered. This activity identifies the specific ntuple damage state vectors of Level 2 (shown in subspace at Level 2 in Figure 2)

that map to the required m-tuple capability state vectors of Level 3 (as identified

in#3), which in turn map to mission completion.*

6. Define initial conditions (Level 1): Define the set of initial configurations of the

threat (likely to be encountered in combat) and weapon system just prior to

interaction. This set of configurations selected considers the mission objectives,

the physical environment of combat, and the users' tactics and doctrine.

7. Develop damage operators (Ou): Develop the damage operators expected in

interactions of the system and threat (e.g., damage mechanisms, such as threat

penetration, fragment penetration, fire, or shock). The damage operators

establish the link(s) between the initial conditions (Level 1) of the threat and

weapon system prior to interaction and the damage state vectors of Level 2,

representing the system's physical status following threat-system interaction.

* This mapping will also identify the specific n-tuple damage state vectors in Level 2 that will not map to

the identified required m-tuple capability state vectors in Level 3.

52
8. Define system physical status (Level 2): Define the specific n-tuple damage state

vectors (Level 2) that are mapped from the specific points (i.e., initial conditions)

of threat-system interaction defined in Level 1. The n elements of the damage

state vector describe the status (i.e., kill, no kill) of the n number of components

of the system following a threat-system interaction.

Analysts need to be able to predict not only the interactions that are likely to

result in the loss of critical components, but also the interactions that are likely to

result in personnel casualties (i.e., personnel vulnerability). Personnel casualties

may also affect the probability of mission completion (i.e., specific number of

fully functioning crew members may represent a critical component of system).

This activity identifies system vulnerabilities associated with personnel

casualties, as well as the potential for catastrophic loss of the system.

9. Compare system physical status in activity #5 to status in #8: Compare the

physical status of the system required for mission completion (activity #5) and

the identified physical status of the system following threat-system interaction

(activity #8). This activity compares (1) the specific critical components that

must remain in operation (i.e., not be killed) after the threat-system interaction to

deliver the identified capability levels required for mission completion to (2) the

specific critical components that are expected to remain in operation following

the threat-system interaction.

This activity identifies system vulnerabilities associated with a system's failure to

complete its mission.

4.2.3 Data Sources. The Taxonomy VA clearly illustrates that the data required

for the assessment of system vulnerability fall in one of three categories: (1) Data Set A

(see Figure 2), data that link the mission-relevant capability measures of the system to

the mission-status measures of battlefield success (03/4) in combat scenarios likely tobe

encountered by the system, given users' tactics and doctrine associated with those

scenarios; (2) Data Set B (see Figure 2), data that link the damage state vectors resulting

from threat-system interaction to the capability state vectors associated with mission

53
success (Ow); or (3) Data Set C (see Figure 2), data that link the initial conditions of the

threat-system interaction to the resulting damage state vectors (Ou). Sources for

obtaining the required data are explored in this section.

To begin the vulnerability assessment process, the analyst must have a thorough

understanding of the combat scenarios in which the system is likely to operate (i.e., the

physical environment, the likely threats, and the mission[s] to be completed) and the

relevant users' tactics and doctrine. A thorough analysis of likely scenarios forms the

basis for determining the minimum levels of capabilities (i.e., Level 3 output) needed in

the set of combat scenarios selected (i.e., activities #1-3 in preceding analysis). Sources

for this information include the designated users of the weapon system, military

strategists, and analysts and operations researchers associated with the evaluation

agencies of the military service.

It is proposed that a major objective of vulnerability assessment is to evaluate the

extent to which a weapon system retains the capabilities that were determined at the time

ofacquisition to be neededfor mission success when the system interacts with threats it is

likely to encounter in combat. It is assumed, therefore, that the data defining the

relationship between the mission-relevant capability measures and the mission-status

measures of battlefield success serve as input to decisions associated with the design

and engineering of the system. The operational requirements for the system,

incorporating the levels of relevant capabilities needed by the system for mission

success, define the minimum levels of those capabilities that must be retained by the

system in combat and provide the basis for determining the data required for the

vulnerability assessment. These data (i.e., Data Set A in Figure 2) should be readily

available to personnel designing the vulnerability assessment plan.

To complete the assessment, the analyst must obtain data relevant to (1) the

relationship between the damage vectors and the measures of capability required by the

system to complete its mission (02/3) and (2) the relationship between the threat-system

interaction and the damage vectors associated with system and personnel vulnerability

(Ou). Potential data sources include (1) results of prior tests of materials, components,

subsystems of earlier and current models of the system or similar systems, (2) results of

54
prior system-level tests of earlier models of the system or similar systems, (3) combat

data relevant to damage mechanisms, system damage, and residual capabilities of

system as associated with the identified threats, (4) design analyses of the system with

consideration given to the new materials and technologies incorporated into the system,

(5) prior engineering analyses or CDTs, and ( 6) M&S runs that incorporate the system

description, threat characteristics, and damage mechanisms expected in threat-system

interactions. These data (i.e., Data Sets B and C in Figure 2) would be sought from

defense databases, the system contractor, and defense analysis, testing, and evaluation

agencies. Information relevant to the 01/2 or the 02/3 mappings that is either unavailable

from the identified sources or is not considered reliable by analysts defines the subset of

the required information (i.e., data voids) to be addressed in the vulnerability

assessment plan.*

4.2A Critical Data Voids. Many resources would be required to design and

conduct a vulnerability assessment plan that explores all data voids associated with all

possible combat scenarios. To ensure an efficient and effective use of limited resources,

analysts must prioritize the data voids to be addressed in the assessment plan (i.e.,

establish critical data voids). The first cut in restricting data voids to be addressed is

achieved by limiting the preceding analysis of activities to the combat scenarios that are

more likely to unfold. Analysts must also consider the degree of confidence placed by

decision-makers in the subset of required information that is available. Figure 3

provides one representation of an approach in which priorities (one [1] representing

high priority and four [4] representing low priority) are assigned to several groups of

data voids to be addressed in a vulnerability assessment plan.

A data void in the information set required for vulnerability analysis is assumed to

exist if (1) data are not available or (2) the confidence in available data is not high

(i.e., low to moderate confidence only in available data). In designing an effective

It is assumed that the vulnerability assessment program is begun early in the acquisition cycle. Some of

the identified information sources may have relevant data in place at the beginning of the planning

process; other sources may produce data as part of a vulnerability assessment plan executed to obtain

data relevant to evaluators' decisions.

55
A

II

II

II

II

(ft

.2

u

C/3

*->

<«

1

o

U

>^

II

II

II

II

II

V

«3

o

U

o> C

o

*-(

«s

O A

p. 0>

'S «

s-s 9(ft 6o

Su

».2

IS c

(ft C/3

«

0

«*- CH «

o I

0> w

3

O «

1M

ro CN

CO

^

w

'S

JS >

00 ««

(N

Oi ... **

S C «

o >

CN

.S 2

o> 4->

<Q (0

ffi

an .

a> «s w

£3,2

S 3'3

° I *

c

0)

en

en

2

0)

C

"3

>

c

•ü

a»

en

a» ^

(ft o

'S'0

o

fa

«I

o bo

£ £

a>

S

60

tin

56
vulnerability assessment plan with limited resources, higher priorities would be

assigned to addressing Ow or C\3 data voids associated with reasonably possible or

probable combat scenarios (i.e., likely combat scenarios) in which confidence in

available data is not high (i.e., cells labeled [1] and [2]). Gathering data relevant to Ou

and 02/3 mappings in likely scenarios in which there is high confidence in available data

(i.e., cells labeled [3]) may be part of an assessment plan if funds are obtainable for

assurance testing. Generally, a lower priority would be assigned to the analysis of

activities and data voids associated with combat scenarios considered only remotely

possible (i.e., cells labeled [4]).* Although classifying data voids into four categories

may facilitate a systematic approach to addressing data voids, priorities may well need

to be defined with finer resolution as resources become more limited.

In this report, it is assumed the data voids to be addressed in an environment of

limited resources are restricted to that subset of data voids associated with likely

combat scenarios in which information relative to 01/2 and 02/3 mappings is either

unreliable or unavailable (i.e., priorities [1] and [2] in Figure 3). Prioritization of specific

data voids within this restricted subset may be accomplished by either engaging in

subjective forms of analyses or employing quantitative-qualitative analytical tools, such

as the Analytic Hierarchy Process (AHP) or Quality Function Deployment (QFD). A

description and discussion of the applicability of these tools to group decision-making

may be found in other sources [27-32].

4.3. Vulnerability Assessment Plans. A cost-effective strategy to obtain data

needed to assess the vulnerability of a weapon system prior to fielding requires (1) a

clear identification of the data required for the decisions to be made, the data available

from reliable sources, and the data remaining to be gathered (i.e., data voids) and (2) a

vulnerability assessment plan designed to address the data voids with consideration

given to the limited resources available. The methodology proposed in this report for

the selection of the optimal vulnerability assessment plan from a group of alternative

* It is acknowledged that analysts may choose to assess the vulnerabilities of combat scenarios that are

only remotely possible if the scenarios are associated with very costly losses and the resources for

assessment are available.

57
assessment plans incorporates many of the principles of a Cost as an Independent

Variable (CAIV) strategy.

4.3.1 CAW Principles. The Defense Acquisition Deskbook describes CAIV as

follows [33]: *

CAIV is a strategy that entails setting aggressive, yet realistic cost

objectives when defining operational requirements and acquiring defense

systems and managing achievement of these objectives. Cost objectives

must balance mission needs with projected out-year resources, taking into

account existing technology, maturation of new technologies and

anticipated process improvements in both DoD and industry.

The CATV methodology supports the setting of realistic but aggressive cost

objectives early in the acquisition program and the implementation of cost-reduction

strategies throughout the program, including cost-performance tradeoffs, competition

among suppliers/contractors, integration of program activities through early planning,

and the development of efficiencies in core activities [34].

Characteristics of the CAIV methodology follow [34,35]:

• Cooperation of PMO, system user, contractor/manufacturer of system, Service

leaders, and OSD is required for successful implementation in the production

decision. System users play an active role in the tradeoff process throughout the

life cycle of the system.

• Analyses of measures of mission effectiveness vs. performance requirements and

performance requirements vs. costs of systems are conducted to support

production decisions.

• The program budget defines available funds and drives the acquisition strategy

for the program. Cost goals must meet objectives of affordability and ability to

fund threshold requirements of system.

* Implementation of CAIV, an initiative to reduce life-cycle costs of defense systems, is required of all

new ACAT I and IA programs [33].

58
• System performance parameters (included in ORD) are the designated key

performance parameters. Potential for tradeoffs of cost against performance is

increased if the number of key performance parameters is rriinimized. Key

performance parameters are not tradable below a threshold value, and

parameters must meet users' needs.

• Tradeoff process begins early in the design process with updates to costs and

performance requirements incorporated throughout implementation. Metrics to

track performance are established. Long-range planning is implemented.

• Risks (uncertainties) in CATV implementation over program life cycle include

the stability of the program budget and the priorities for the system, (2) the

stability of the mapping of required measures of effectiveness to minimum

performance requirements, (3) the availability of databases and models

applicable to cost estimations for systems, (4) the potential for tradeoffs of

performance requirements, costs, and schedules, (5) the analysts' capacity to

understand the interrelationships of performance requirements, (6) the

incorporation of performance requirements in production contract with

appropriate incentives, and (7) the developments in technology to enable

achievement of program objectives.

The core of the CAW methodology is the tradeoff process, in which mission needs

are balanced against projected resources with consideration given to scheduling issues

and identified risks. In production decisions employing CATV principles, risks are

managed to achieve cost, schedule, and performance objectives.

In this report, it is proposed that the selection of the optimal vulnerability

assessment plan be made by trading off the plan's performance (i.e., the plan's capacity

to assess the critical issues of personnel/system vulnerability) against the budget for

plan implementation with consideration given to scheduling issues and risks identified

in achieving the plan. Key parameters are identified as the critical prioritized data

voids to be addressed in the assessment plan. Major analyses would include

performance and cost estimates for the vulnerability assessment plans and the elements

59
of the assessment plans. The methodology proposed employs many of the CATV

principles described in the context of production decisions in the prior paragraphs of

this section and emphasizes the exploration of alternative assessment strategies to

achieve vulnerability assessment objectives.

4.3.2 Construction ofVulnerability Assessment Plan. The Taxonomy VA (Figure 2)

presents a framework for identifying the data voids, a subset of the information

required to conduct a vulnerability assessment of a weapon system. To complete the

vulnerability assessment, analysts must design a blueprint, a vulnerability assessment

plan, to address those data voids. The design of an assessment plan requires the

identification of potential sources or activities (i.e., elements of the plan) for obtaining

data to address critical data voids and the evaluation of the capacity of each element to

produce the critical data within the time constraints of the assessment process.

Assuming alternative vulnerability assessment plans to provide the critical data are

available and the budgets for obtaining the data are limited, the analyst must select the

optimal (i.e., cost-effective) vulnerability assessment plan from the group of alternative

plans by weighing (1) the capability or capacity of each plan to provide the critical data

(i.e., benefits of plan) and (2) the cost to execute all elements of the plan with

consideration given to the uncertainty (risk) associated with each plan's capacity to

provide the critical data at the identified cost within the time constraints of the

assessment process.*

Constructing a vulnerability assessment plan requires the identification of potential

sources or activities that could provide data to address the critical data voids. These

activities serve as elements of the assessment plan and may include modeling and

simulation; controlled damage testing; experimental testing; developmental testing;

design and engineering analyses; component-, subsystem-, and system-level live-fire

testing; and FU SL LFT. Factors that provide input to the design of the vulnerability

assessment plan include the nature of the identified prioritized critical data voids, the

* Uncertainty (risk) associated with a plan's capacity to provide the critical data within the time

constraints at the identified cost includes the uncertainty associated with the availability of system

hardware needed for execution of the plan.

60
budget for assessment, the availability of hardware, the timetable for the completion of

the assessment process, and other constraints unique to the identified data sources.

For example, Figure 4 shows alternative elements that may have the potential to

address identified data voids. Each element would be described in detail and the cost

of executing the element estimated. Figure 5 illustrates three vulnerability assessment

plans, each including several alternative elements to address the identified critical data

voids. Each plan includes a different combination of elements and is associated with a

different total cost.*

4.3.3 Capability ofVulnerability Assessment Plan. In designing a vulnerability

assessment plan to address critical data voids, an important question arises: "How does

one measure the capability or the performance of a plan that is designed to investigate

the unknown?" This is a difficult question to answer ex ante or ex post execution of the

plan. An objective measure of capability is not available for this type of analysis, and an

appropriate subjective measure is sought to define the capability or performance of the

plan. As previously noted, in selecting the optimal vulnerability assessment plan, the

analyst must consider not only the capacity of a plan to produce the critical data, but

also the uncertainty associated with the plan's ability to supply the required data within

the time and financial constraints of the assessment environment.

A subjective metric of risk assessment is proposed for evaluating the capability of a

vulnerability assessment plan. Thus, the capability of a plan is rated by the degree of

uncertainty a designated panel of experts associates with the plan's capacity to

adequately address the prioritized critical data voids within the available time for

assessment at the identified cost. A four-point scale of risk assessment is proposed with

ratings of high, medium-high, medium, and low risk. An example is provided in the

following paragraphs.

* If the elements of an assessment plan (e.g., modeling, experimental testing, controlled damage testing)

differ in detail (i.e., different models are exercised within modeling component, different tests are

included in experimental testing), the vulnerability assessment plans also differ.

61
H iß iß ««• iß iß iß

0<

•

••

••

•

o

u es

1—H

\Z< iß iß iß iß &t iß 4-> • • • • • •

h-i

QQ

«5

Exp

Tests 1

iß iß iß iß &t iß t»

"" *•; **"";

<

•

•

••

•

•

3^sS

^i2 iß iß iß iß iß iß

3? i-i-l a> >> H

20 CD iß iß iß iß iß iß c ^ ^ pa • • • • • •

a> «a

S

Ele

Crit

*« 55 t« iß iß iß- iß iß iß •1-4 SO

«8<fi

•

•

•

s p*

<

^

3s

*

*

CO*

^*

in

••••

*H

>D

>D

>0

>0

D

g

t>0

ft

62
mmm&m mmmi

ipÄ

,1m...,,..

c/>

<

gs&

>

'wMw&M&,

%S>0%k$$M'

wmmm

mmtmmumtmtm »mmmmmnmm t

s;

L

;

::.'/;yV;

X

>

X

tt0w&yffi$y%f&k#&

Wmmm

pff

iywliiÄ

vWW&

WM2$MsM

V0W0SW'

Mtt*&tf$mMt*ttif*

ys&ffivs.?y.±

wßmm-Wt

III

x

M 2>

>

Cß

C

d

cy

Cß

0>

Cß

cß

<

o>

d

d

>

a»

>•

c

01

Cß

o

u

d

o

d

u

0>

X

W

•o

01

cß

w

d

(ß

d ^.

O) cß

o» <

w ^.

o>

S

00

63
Assume an expert panel has assessed the risk associated with the three vulnerability

assessment plans illustrated in Figure 5. The described four-point scale is employed to

assess the risks associated with each plan's capability to address the specific identified

data voids. As illustrated in Figure 6, the expert panel assesses a low risk (i.e., little

uncertainty) to be associated with the capacity of Vulnerability Assessment Plan #1 to

adequately address Data Voids #3 and #5, but a medium risk to be associated with

addressing Data Voids #1, #2, and #4. The low risk assessment assigned to the

Vulnerability Assessment Plan #1 and Data Void #3 intersecting cell is interpreted as

little uncertainty associated with the capacity of Vulnerability Assessment Plan #1 to

adequately address Data Void #3 within the given time frame at the specified cost.

4.3.4 Cost ofVulnerability Assessment Plan. An important consideration in the

selection of the optimal assessment plan is the total estimated cost of accomplishing the

identified elements of the alternative plans. As discussed in section 3 of this report, a

valid comparison of alternative assessment strategies requires the establishment of

accounting procedures to ensure consistency and comparability in estimating costs

across plans, as well as across periods and systems. Decision-makers must be provided

with cost data relevant to all elements of each alternative plan prepared according to a

cost methodology applied consistently across all plans. This methodology includes

• Identification of all tasks included in the elements of each plan and the cost

components of those tasks (e.g., direct materials and labor, indirect

materials and labor, facilities costs, etc.).

• Identification of the behavior of identified costs (e.g., cost varies with volume,

cost fixed over relevant range of volume, etc.).

• Identification and description of method(s) employed to measure costs, if

alternative methods are available; disclosure of alternative values, if available.

• Identification of all allocated costs and bases for cost allocation.

• Identification and explanation of all incomplete cost data.

64
co

T5

•rHo>

*->

«8 0

AS

CJ

•rH

H-»

•auv

ID>0

o

fi3•rH

0)

fid•i-i

0)

2*>D

•rH

a;

fid•rH

a; s

fid•rH

a;

>D

oo

fi

•iH

8>D

fid

0)

oo

>0

fid

• 1-1

o

o

rJ

Alternative

Vulnerability

Assessment Plans

rH

PLi <>

M

P* <>

PH2

COd

«8

fi3

•OH

0) ssd

fi

CO

CO

CU

CO

CO

rd

00

•iHssd•iH

0)

S*

d>

0) >

•rH

fi

cu

CO >g

CO

•rHo>

3 «J

.. . *8

oD

CO *-H

0) (8

-HU

rH «rH 3 HW

CO «rH

CD

bod

dI

CO

CO

cu

CO

CO <

CO

PÜ

r-H

CO

d

<u .„

I—I *"*

CU©

rH hJ < <8

v©&d

bo

65
There are benefits in identifying the costs of the individual elements of an

assessment plan and the costs of the tasks that are included in each element. ActivityBased Costing (ABC), a costing methodology that focuses on the activity or task as the

basic cost objective, has been widely used in recent years to determine a more accurate

cost of producing a product or delivering a service. Employing ABC to cost the

elements of an assessment plan requires the identification of all tasks in each plan

element that consume resources and the total costs associated with each task. Tasks

must be carefully examined to identify the tasks' cost drivers, the factors that most

closely drive or cause the costs to increase or decrease.

The advantage of employing ABC to cost individual elements of an assessment plan

is that ABC provides a means to weigh the value added against the costs incurred for

each plan element. ABC affords the framework to identify tasks of an element that are

non-value-added and are able to be eliminated as well as tasks that are value-added but

are able to be made more efficient [36-38].

4.3.5 Selection of Optimal Vulnerability Assessment Plan. To select the optimal

(i.e., cost-effective) vulnerability assessment plan from a group of alternative plans, the

decision-makers must define (1) the maximum levels of risk in addressing each of the

critical data voids they are willing to accept in a vulnerability assessment plan and

(2) the maximum cost they are willing to incur in plan execution.

The priorities assigned to the identified data voids serve as input to decisions

relevant to the risk assessment levels evaluators require in an acceptable assessment

plan. For example, assume analysts (using the four-point scale of risk assessment

previously described) have determined certain maximum levels of risk they are willing

to accept in a vulnerability assessment plan (i.e., plan addressing five prioritized data

voids). Those maximum levels of acceptable risk are shown in Figure 7. Higher

priorities assigned to critical data voids would be associated with lower maximum

levels of risk acceptable to the analysts. In the example described in Figure 7, a plan is

termed acceptable (i.e., with respect to performance), if analysts associate a low risk

with the plan's capacity to adequately address Data Voids #1 and #2 and medium

66
(ß

•pHo>

H-> Du•pH

H-» uv

>0

cr> sa

'S

3>0

5

£5>

sa

>Q

<*> So

p-J

>0

***** o

Maximum

Acceptable

Risk Level

Cft bo

Cß

d VI •PH 2

*dK

T3<

P*

Cß

•pH

O P-H d £

«ö

o

&H pj M -ö d d

0) fÖ SER

cn

£

cu

Cß

Cß

s

• PH

<^

cu

•pH >—( •pH

pd"

00

pQ

• pH

ffi

d PH

i—*

?

d •pH > CU d• pH £

CU

p—•) J*

PQ 00 m • pH + »

OH ffi

CUUU

•V

Cß

<

•PHo

P*

>

Cß ««

Cst <8 o 0pa^

Cß «8

^H

u

cu •PH > • pH cu PH J U 6d

cu

•PH

•PHX

•pHd

«8 cu £

T3

p—i

IN

CU

PHd

bo

l-H

67
(or less) risk with the plan's capacity to adequately address Data Voids #3, #4, and #5

within the time constraints of the vulnerability assessment process and at the specified

cost of plan execution.

The measure of a successful vulnerability assessment depends on which data voids

are addressed by the vulnerability assessment plan, the capacity of the assessment plan

to address those data voids as measured by the risk assessment metric, and the

priorities attached to those data voids by the evaluators of the vulnerability of the

system. The capabilities required of an acceptable vulnerability assessment plan as

measured by the maximum risk levels allowed must be compared to the capability

measures of the alternative vulnerability assessment plans proposed for consideration.

Figure 8 provides a comparison of the risk assessment ratings of Vulnerability

Assessment Plans #1, #2, and #3 (see Figure 6) to the maximum levels of risk

determined to be acceptable (see Figure 7). The comparison reveals both Vulnerability

Assessment Plans #2 and #3 are acceptable in terms of expected plan performance. A

comparison of the expected costs of completing either of the two plans to the maximum

amount allowed in the budget for plan execution also finds both plans to be acceptable.

The comparison of alternative assessment plans affords decision-makers an

opportunity to look at the tradeoffs in alternative plans relative to cost and performance

(i.e., capacity of the plan to adequately assess vulnerabilities associated with the data

voids). Although Vulnerability Assessment Plan #3 initially appears to be the optimal

plan (i.e., Vulnerability Assessment Plan #3 meets maximum levels of risk allowed of

plan and has a lower cost than Vulnerability Assessment Plan #2), possible questions

for the decision-makers may be

• Realizing that one remains within the budgetary maximum of $20, is one willing

to pay $2 more and adopt VAP #2, thereby decreasing the level of risk in

addressing Data Void #3 from moderate to low?

• Is one willing to adopt VAP #1 and accept more risk in addressing Data

Voids #1 and #2 (and less risk in addressing Data Voids #3 and #5) to save $3?

68
A

II

II

II

II

II

II

II

II

II

II

II

II

II

II

II

II

II

II

II

II

II

II

eno>Duu

II

II

II

II

II

II

II

II

II

II

II

II

II

II

II

II

II

II

II

II

II v

Cost of Plan

Execution

MaximumAcceptable

Budget

Is

$20

T—1

*ß" ÄB

J3

en W

ON

rH

SATS

CU

R5 6

en W

IN

rH

COa

•X3

en W

>0

5u <u £v- ^s ^ 1

o

CU

•i-i

<**>0

£ "S -^^S

B

•i-i

6

CU

TJ

CU

CO*>Q

ss^ -52

Oo

•rH

T3

0) s

0

.£1 -5S^«

B

• rH

o

1

*>0

5«°-* £S

6

• r-l

o

o

69

*

p* >

*

PH>P* <>

*J (A C CM

0)g

<>

CA

CO

CU

CA

CA

CA «s <

l-H

^ ^->

CA

c

PH

cus

o CA *-» CA *» V

cy CA

on CA

T3

<

03 'W*

T3

•v4

C JO

«8 <8 ^

«H

CÜ

CA

C

&

S

O

>

CA at > ÜCU Ö

h-l

s

«y ai

^* 1—1 <

^-< M-l & O

CA •**

u CA < o BsS U

nj g «J ••M

MH

CA

o *o o

CA CA a

CU

CA

S «8 u S

00

eu

)HsM

)

HH
• If the costs of VAP #1 are allowed to increase, are the risk levels of VAP #1

lowered to acceptable levels?*

It is important to realize that the estimated costs of all alternative vulnerability

assessment plans may be over and above the amounts allocated or budgeted for the

vulnerability assessment program. It is at this point that program managers must

carefully weigh the pros and cons of reallocating dollars from other programs to V/L

assessment or of requesting additional funds for conducting an assessment plan that

meets the performance standards (i.e., allowable levels of risk in addressing critical data

voids) determined to be acceptable to decision-makers.

4.4 Complexities of Methodology. There are several complexities inherent in the

proposed approach to vulnerability assessment that must be addressed. First, the

capability or the capacity of the vulnerability assessment plan to adequately address

critical data voids is rated by a subjective measure, the assessment of risk in the plan's

capacity. The lack of an objective measure to rate the plan's ability to provide all ofthe

data relative to identified data voids or address the unknown unknowns is not a

problem unique to weapon system analysis/ In many disciplines (e.g., finance, law,

and medicine), opinions of expert panels or groups of individuals with experience in

similar decision-making tasks are sought for evaluations in which objective measures

are not available.*- Group assessment or evaluation (i.e., panel of experts) is proposed

* A complete analysis that includes ALL possible combinations of data source components that have

associated costs within the budget range would include the suggested alternative vulnerability

assessment plan (i.e., increased costs and decreased risk levels). However, given the limitations in

information processing of individuals, this type of comparison affords a check on the completeness of

alternatives generated by analysts/testers/evaluators.

T To employ an objective measure to assess the capability of a vulnerability assessment plan to provide all

ofthe data relative to critical data voids would require knowledge or definition of all ofthe data.

Knowledge of all unknowns would be necessary to answer the question, "Were all the unknown

unknowns identified?" Although components of vulnerability assessments plans may be compared

(e.g., vulnerabilities detected in component testing versus vulnerabilities detected in modeling), a

comparison of even several vulnerability assessment plans would be economically unfeasible.

Therefore, the decision-maker must search for the next best solution, the opinions of experts as

surrogate for the objective measure of capability.

* Recent studies in other disciplines have shown that experts with domain-specific experience often make

decisions more compatible with those of statistical models than do experts with more global experience.

To avoid unneeded bias, it is imperative that care be given to the choice of experts invited to join the

decision process [39,40].

70
because numerous studies have established that group information-processing often

results in judgments that are superior to the judgments of individuals [41].

A second complexity arises in the determination and comparison of the costs of

vulnerability assessment plans. At the present time, the costs of testing and evaluation

of weapon systems are not reported consistently across services, weapon systems, or

time periods. The computation and recording of costs of components of vulnerability

assessment plans according to a single methodology that is consistent across periods

and comparable across weapon systems would provide a useful database for analysts

designing vulnerability assessment plans in the future. Suggestions for addressing this

issue have been discussed in this report.

Finally, it is recognized that implementation of the proposed methodology requires

the commitment of time and personnel to the design and evaluation of alternative

assessment strategies. A cost-effective vulnerability assessment strategy is an important

part of a cost-effective acquisition process. Failure to commit funds to the prevention and

detection of vulnerabilities early in the design and production phases of the system may

well result in the expenditure of much larger dollar amounts in system design and

engineering modifications or retrofits following production. This issue is discussed in

more depth in the Conclusion.

5. Conclusion

5.1 Role of FU SL LFT&E in Cost-Effective Vulnerability Assessment. A

methodology to identify, measure, and categorize the costs and benefits of FU SL

LFT&E, as well as other elements of a vulnerability assessment plan, is presented in

prior sections of this report. As an element of an assessment plan, however, FU SL

LFT&E is unique. For many systems, FU SL LFT&E is a congressionally mandated

activity conducted by independent agencies* prior to the system entering full-scale

production [1]. Evaluating the role of FU SL LFT&E in a cost-effective vulnerability

* Activities conducted by independent agencies are defined as activities not performed under the

direction of the manufacturer/contractors) of the system or the PMO associated with the system.

71
assessment strategy requires an understanding of the vulnerability assessment process

and the risks inherent in the process.*

5.1.2 Cost-Effective Vulnerability Assessment Process. A cost-effective vulnerability

process emphasizes the achievement of assessment objectives within the constraints of

resources available. This report proposes the effectiveness and efficiency of an

assessment strategy may be increased by the identification of the specific set of data

required for vulnerability assessment, the determination of the subset of the required

data currently available from reliable sources, and the selection of a cost-effective

vulnerability assessment plan to fill the data voids. Because the design of vulnerability

assessment plans is based not only on the capacity of the plan's elements to address the

critical data voids but also on (1) the time constraints of the assessment schedule, (2) the

production schedule and availability of system hardware, and (3) the costs attached to

the execution of the plan, there are often tradeoffs among plan performance (i.e.,

capacity), cost, and schedule that must be explored before the optimal plan is

determined.

A cost-effective strategy of vulnerability assessment begins at the Concept

Exploration phase and continues through Program Definition and Risk Reduction and

Engineering and Manufacturing Development phases of the acquisition process. It may

include experimental testing, M&S, design and engineering analyses, controlled

damage testing, component and subsystem LFT, FU SL LFT, as well as other activities

that address the identified critical data voids relevant to assessment decisions.

Planning early for vulnerability assessment may increase the number of options

available to address the critical data voids and may allow assessment activities to be

completed early in the acquisition process at a time when design or engineering

changes are more feasible and/or more economical. For example, technical tests

conducted early give insight into design and engineering issues and often employ test

articles (e.g., mock-ups, replicas, etc.) that are less expensive than the realistic test

* To facilitate discussion of the methodology presented in section 4 of the report, the Conclusion is

restricted to a discussion of cost-effective vulnerability assessment issues.

72
articles required to be used in an FU SL LFT. There are, however, some critical data

voids (e.g., cascading and synergistic damage mechanisms) that are unable to be

addressed by alternative activities and unable to be explored until late in the production

process when the required hardware for testing is available.

5.1.2 Risks of Vulnerability Assessment Process. There is a risk in the vulnerability

assessment process that no matter what actions are taken during the acquisition process

by the contractor, the PMO, the IPT committee, or the independent testing and

evaluator groups, a serious or significant vulnerability will remain undetected in the

fielded system. One objective of a vulnerability assessment strategy is to minimize the

risk, a vulnerability assessment risk (VA risk), that a material vulnerability (i.e., one that

could result in catastrophic loss of system, failure to complete mission, and/or

personnel casualties) will exist undetected in a fielded weapon system. The VA risk

could be described as a three-fold risk: inherent risk, control risk, and detection risk*

Inherent risk is the susceptibility of the weapon system to material or significant

system and personnel vulnerabilities. Assessing the level of inherent risk associated

with a system requires an understanding of the environment in which the system must

function, the expected mission(s) of the system and the tasks required to ensure mission

success, the threats the system is likely to encounter, and the potential mitigating effects

provided by the tactics and doctrine of the system users. Information relevant to the

vulnerabilities of prior models of the system or similar models, the complexity of the

design of the system, and the impact of technological developments on the system serve

as input to the assessment of inherent risk.

Control risk is the risk that a material or significant vulnerability will not be

prevented or detected during the design and production phases of the system under the

control structure of the PMO program of survivability or vulnerability assessment. An

accurate assessment of the level of control risk associated with a weapon system

Inherent, control, and detection risks are terms borrowed from a financial auditing context. Although

there are parallels between the auditing of financial statements and the assessment of vulnerability in

weapon systems, the terminology as used in this report is not strictly analogous to the terminology in

auditing contexts.

73
requires an analysis of the survivability or vulnerability assessment program (e.g.,

design and engineering analysis, experimental testing, etc.) conducted by the PMO and

the critical data voids addressed in this program prior to full-scale production.

Detection risk is the risk that a material vulnerability will not be discovered by

analyses, T&E, or other activities conducted prior to fielding of the system by sources

independent of the manufacturer/contractor(s) and PMO. Detection risk is a function

of the nature, timing, and extent of tests (or other activities) that are performed with the

objective of assessing system vulnerability. The level of detection risk is managed to a

great extent by the independent testers and evaluators of the system.

It is proposed that in a cost-effective vulnerability assessment process, independent

testers and evaluators would (1) determine the level of detectable risk they are willing

to accept on the bases of their assessment of the levels of inherent and control risks

associated with the system and (2) design the nature and extent of their analyses/tests

on the bases of those assessments. In other words, higher levels of detection risk may

be acceptable to a tester/evaluator planning the T&E of a system associated with low

levels of inherent and control risks. Only very low levels of detection risk, however,

may be acceptable to testers/evaluators assessing the vulnerabilities of a system

associated with very high levels of control and inherent risk.

5.13 Risks of Vulnerability Assessment and FU SL LFT&E. As discussed in the

previous section, one objective of a vulnerability assessment strategy is to minimize the

risk that a material vulnerability will exist undetected in a fielded system. The level of

detection risk that is acceptable in independent tests is contingent upon the assessed

levels of inherent and control risks; the level of detection risk associated with a testing

activity is determined in a large part by the extent and nature of the testing activities

conducted by independent agencies.

As discussed previously, an understanding of the missions of the system, the tasks

required to complete those missions, the combat environment, the threats likely to be

encountered, and the tactics and doctrine of the system users is important in

determining the relationship between the system's capabilities and the measures of

74
mission success (03/4 mapping). The data required for establishing the minimum level

of capabilities needed for mission completion are also important to assessing the level of

inherent risk associated with the system. The susceptibility of the weapon system to

material vulnerabilities (i.e., inherent risk) should be an important consideration in the

design of the control structure that is established under the direction of the PMO with

the objective of preventing and detecting vulnerabilities in the system produced.

In a cost-effective vulnerability assessment plan that includes FU SL LFT&E,

independent testers and evaluators might be expected to restrict the scope of the FU SL

LFT to a greater extent for a system that has a well-established PMO-directed

survivability or vulnerability assessment program (i.e., low control risk assessed) than

for a system with similar inherent risks but a less mature or less developed control

structure in place.

5.2 Cost-Effective FU SL LFT&E. The manufacturer/contractor(s) of the system

and the PMO, as well as independent agencies, participate in the execution of an

effective assessment strategy. FU SL LFT&E is one of several possible activities

conducted by an independent agency that may be selected as an element of a

vulnerability assessment plan. The methodology presented in this report (i.e., section 4)

would support the inclusion of FU SL LFT in a vulnerability assessment plan, if it

compared favorably with other elements on the basis of identified costs and benefits

(i.e., capacities to address critical data voids) and uncertainties associated with the

identified impacts. For certain covered weapon systems, however, FU SL LFT&E is a

congressionally legislated element of the assessment plan, not a discretionary activity of

the assessment process [1]. For that reason, a closer look at increasing the costeffectiveness ofFU SL LFT&E is important.

Many of the suggestions to increase the cost-effectiveness of FU SL LFT&E that

were offered by interviewees can be better understood in the context of the objectives of

cost-effective assessment, the risks of assessment, and the activities of FU SL LFT&E.

Generally, suggestions fell into four broad categories: the prioritization of data voids,

the consideration of alternative activities for FU SL LFT&E and alternative designs of

the FU SL LFT, the evaluation of the control structure under the PMO direction, and the

75
review of the outcomes of prior and current FU SL LFT&E activities. The suggestions of

interviewees are summarized in the following paragraphs.

5.2.1 Prioritization ofData Voids. Prioritize data voids to ensure limited funds are

spent effectively in addressing the most critical data voids. Consider the benefits (i.e.,

data to address critical data voids) expected to be received from engineered vs. random

shots, as weU as benefits from each additional shot planned with attention given to the

priorities established for the identified data voids. Consider alternative activities in

conjunction with a restriction of the scope of FU SL LFT&E to effectively and efficiently

address critical data voids.

5.2.2 Consideration ofAlternative Activities and Designs ofFU SL LFT&E.

Explore alternative activities to investigate the critical data voids of the system that are

planned to be addressed by FU SL LFT&E. Consider substitution of alternative

activities if less costly to conduct than FU SL LFT&E and/or occur earlier in the

acquisition process than FU SL LFT&E, at a time when fixes may be more feasible

and/or more economical.

Consider the cost-effectiveness of the design of FU SL LFT (e.g., random vs.

engineered shots, number of shots, etc.). Balance the objectives of sampling efficiency

and avoidance of bias perceptions by employing smart testing strategies (e.g., random

sampling from combat distribution).

Encourage modelers and testers to integrate efforts in the development and W&A

of M&S tools. Facilitate discussions early in the acquisition process among analysts,

manufacturer/contractors), PMO, and independent testers and evaluators to increase

time available to consider, plan, and conduct alternative activities for addressing critical

data voids.

5.2.3 Evaluation ofControl Structure. Evaluate the control structure established by

the PMO to monitor system survivability/vulnerability. Encourage early involvement

of the evaluator in assessment process to ensure minimal funds are spent on redundant

testing and analyses. In planning the scope of the FU SL LFT&E, give careful

76
consideration to the activities managed by the control structure to prevent and detect

vulnerabilities throughout the acquisition process. An evaluation of the control

structure in place that leads to an assessment of low control risk and the subsequent

restriction of FU SL LFT scope could serve as an incentive to PMO/manufacturer/

contractor(s) to develop and maintain an active agenda for assessment through all

phases of acquisition.

5.2.4 Review ofFU SL LFT&E Outcomes. Compile a database relevant to FU SL

LFT&E outcomes and actions (i.e., system retrofit, redesign, reengineering, etc.) taken

on the bases of those outcomes for all FU SL LFT&E conducted (i.e., ex post analyses).

Include in the database data relevant to fixes proposed and fixes funded for the current

model, as well as alterations implemented in later models. Identify unsuspected

vulnerabilities detected in testing. Require the costs of FU SL LFT&E components to be

recorded and reported according to a consistent methodology.

Outcome data obtained ex post T&E could assist in identifying the types of data

voids that are addressed most effectively in FU SL LFT&E and the areas in which

unexpected vulnerabilities are detected most often. This information would provide

valuable input to (1) the design of PMO control structures for the prevention and

detection of vulnerabilities in system design and production and (2) the planning of

future FU SL LFTs by IPT committees, testers, and evaluators.

For example, FU SL LFT&E is conducted with the expectation of increasing

evaluators' understanding of the relationship between the system interaction with

threats likely to be encountered in combat and the damage vectors associated with

system and personnel vulnerability following the interaction (i.e., Ot2 mapping).

Assume a review of the outcomes of prior FU SL LFTs finds that a large number of the

significant unexpected vulnerabilities or unknown unknowns uncovered in FU SL LFT&E

can be categorized as data relevant to the 02/3 mapping— data that explain the

relationship between damage vectors and the measures of capability. Because 02/3 data

voids can be addressed more efficiently and effectively by other types of assessment

activities (e.g., engineering analyses, CDT), this finding may indicate additional

resources are needed to direct assessment activities that address 02/3 relationships.

77
Similar to outcome data, cost data gathered ex post from completed FU SL LFT&E

activities are relevant to the planning of future FU SL LFT&E. Cost data identified by

the components of FU SL LFT&E facilitate the identification of value-added and nonvalue-added components, as well as efficiencies and inefficiencies in the T&E process.

5.2.5 Contributions ofFU SL LFT&E to V/L Assessment. Although many offered

suggestions for improving the cost-effectiveness of survivability/lethality assessment as

a whole and FU SL LFT&E in particular, only a few of those interviewed for this report

discounted the benefits or contributions of FU SL LFT&E. Contributions of FU SL

LFT&E cited in interviews, as well as published reports and articles, included the

capacity of FU SL LFT&E to identify both suspected and unknown vulnerabilities and

lethalities; provide insights into potentially significant damage mechanisms and the

quantification of their contribution to system V/L; supply data for the calibration and

validation of lethality and vulnerability models; provide insights into BDAR potential;

and contribute to the development of battle preparation procedures, and operational

doctrine and tactics. The importance of the high visibility of FU SL LFT&E to ensuring

Congressional funding for V/L assessment and to providing incentive for PMOs and

contractors to consider V/L issues over all phases of acquisition cannot be minimized.

In addition, the intangible benefits received from T&E conducted by an independent

agency that serves as validation for the survivability/vulnerability programs of

PMO/contractor(s) must be considered in a weighing of the impacts of FU SL LFT&E.

5.3 Cost-Effective Vulnerability Assessment and Acquisition Process. In this

study, the FU SL LFT&E of three separate weapon systems were reviewed with the

purpose of identifying the activities of FU SL LFT&E, the impacts of those activities, and

the complexities in the reporting of the impacts. Various approaches to address the

complexities uncovered were considered. This report describes the current FU SL

LFT&E process, including its role in a program of vulnerability assessment, and offers

suggestions for improving the cost-effectiveness of vulnerability assessment strategies.

In planning a LFT&E program, the LFTPT brings together the expertise of analysts,

testers, evaluators, system PMO personnel, system users, and others to identify and

prioritize data voids of the system and to consider alternative data sources in the design

78
of an effective LFT&E strategy. In the review of the current planning process for FU SL

LFT&E activities, evidence was found that the V/L community has consistently

attempted to implement smarter testing strategies, develop ways to select shots that

yield more useful information, and build on the lessons of prior FU SL LFT&E

conducted.

It is suggested, however, that more effort is needed in developing cost-effective

strategies for improving the assessment programs of new systems vs. systems in the

middle of the acquisition process with a majority of budget resources already

committed to programs. Developing flexible alternative assessment programs early in

the acquisition process affords maximum opportunity to increase the efficiency and the

effectiveness of the assessment of the system. Linking vulnerability assessment

programs to system design and training programs promotes cost effectiveness in both

the gathering and the use of data required by multiple decision-makers.

In conclusion, it is important to consider the relationship of a cost-effective

vulnerability assessment strategy to a cost-effective acquisition process. An effective

vulnerability assessment strategy includes both an analysis of the likely combat

scenarios and a determination of the links between the measures of battlefield

effectiveness and the measures of capability required of the system to complete its

mission. Understanding the mission of the system and associated tactical utility of the

system user in likely combat scenarios is the basis for understanding how users rely on

the weapon system—the foundation for vulnerability assessment.

A mission-based or utility-based strategy ofsystem acquisition advocates the

establishment of weapon system performance requirements after a careful analysis of

the relationship between system capability measures and battlefield utility measures

(i.e., 03/4 mapping) [26]. The relevant set(s) of system performance requirements then

becomes the basis for weapon system design and engineering.

The information relative to combat scenarios and users' tactics and doctrine

gathered for acquisition decisions also serves as valuable input to the supporting

vulnerability assessment process. In the assessment process, an evaluation is made of

79
the extent to which a system, interacting with threats likely to be encountered in

combat, retains the capabilities that were determined at the time of acquisition to be

needed for mission success. The operational requirements for the system, incorporating

the levels of relevant capabilities needed by the system for mission success, define the

minimum levels of the system's capabilities that must be retained. Therefore, measures

of the system's capabilities serve not only as the basis of the system design but also as

the core of vulnerability assessment strategies.

The importance of understanding the relationship between the tasks that must be

completed by the system to ensure a successful mission and the minimum levels of

capabilities needed by the system to complete those tasks is emphasized in the

assessment methodology proposed in this report. It makes sense that the operational

requirements of new systems incorporate the minimum levels of capabilities needed to

complete mission tasks AND that the objective of vulnerability assessment be to determine

the extent to which the system retains those minimum levels of capabilities it needs to

complete its mission(s) when it interacts with those threats it is likely to encounter in

combat.

Traditionally, the link between V/L Taxonomy Level 3 (i.e., measures of capability)

and Level 4 (i.e., measures of effectiveness) metrics has not been considered to be in the

purview of the V/L analyst. The methodology proposed in this report, however,

suggests that effectiveness mapping (i.e., 03A mapping) must be the starting point not

only for the design of the weapon system and the development of operational

requirements but also for the assessment of the system's vulnerability in likely combat

environments.*

The demands of the acquisition process grow with the increase in complex systems

with advanced technology attempting to operate in changing combat environments.

This report emphasizes the efficiencies that are gained by the mapping of the tasks

required for mission success to mission-relevant capabilities and the sharing of these

' There have been some attempts by researchers to identify the tasks that must be completed for mission

success (i.e., mission-to-task decomposition) and to model the time-sequencing of task activities [42-44].

80
data between decisions-makers responsible for the design of the weapon system, the

assessment of system vulnerability, the training of system users, and the repair and

maintenance of the system. In a similar fashion, the data gathered in vulnerability

assessments to evaluate the effect of system damage on the mission-relevant measures

of capability (Ö2ß mapping) provide insights into battlefield damage and repair

activities, feed the operational test evaluation, and serve as inputs to programs of

training simulators [14,15].

In recent years, it has become increasingly more evident that the system contractor,

user, tester, evaluator, and related analyses groups must join in a cooperative

relationship to increase the effectiveness and efficiency of the entire acquisition process.

Evidence suggests this cooperative relationship of principal players may lead not only

to increased cost-effectiveness in the acquisition process but also to improved readiness

for systems in combat.

81
INTENTIONALLY LEFT BLANK.

82
6. References

1. Live Fire Testing, National Defense Authorization Act for FY1987, contained in

Chapter 139, Section 2366 of Title 10, U.S. Code.

2. Deitz, Paul H., and Aivars Ozolins. "Computer Simulations of the Abrams Live-Fire

Field Testing." BRL-MR-3755, U.S. Army Ballistic Research Laboratory, Aberdeen

Proving Ground, MD, May 1989.

3. Office of the Director, Operational Test and Evaluation. "FY 1996 Annual Report of

the Director, Operational Test and Evaluation." http://www.dote. osd.mil/

reports/FY96/96MlA2.html.

4. Blankenbiller, Douglas C. "Volume I Final Report Live Fire Vulnerability Test

(LFVT) of M1A2 Abrams Tank." CSTA-7539 (Volume I), U.S. Army Combat

Systems Test Activity, Aberdeen Proving Ground, MD, December 1993.

5. Office of the Director, Operational Test and Evaluation. "FY 1997 Annual Report of

the Director, Operational Test and Evaluation." http://www.dote. osd.mil/

reports/ FY97/army/97atacms.html.

6. Perry, Martin, Matthew Cross, and David Washington. "Live Fire Independent

Evaluation Plan/Test Design Plan (IEP/TDP) for the Army Tactical Missile System

(ATACMS) Block IA." IEP/TDP-1-96, U.S. Army Materiel Systems Analysis

Activity, Aberdeen Proving Ground, MD, January 1996.

7. Office of the Director, Operational Test and Evaluation. "FY 1998 Annual Report of

the Director, Operational Test and Evaluation." http://www.dote. osd.mil/

reports/army/98bradley.html.

8. Evaluation Analysis Center, U.S. Army Operational Test and Evaluation Command.

"Bradley Fighting Vehicle System, A3 Test and Evaluation Master Plan, Revision 3."

Aberdeen Proving Ground, MD, July 1996, approved 25 February 1997.

83
9. U.S. Department of Defense. Mandated Proceduresfor Major Defense Acquisition

Programs and Major Automated Information System Acquisition Programs. DoD

Regulation 5000.2-R, Appendix IV: IV-1, Washington, DC, 15 March 1996.

10. U.S. General Accounting Office. Live Fire Testing, Evaluating DOD's Program.

GAO/PEMD-87-17, pp. 4, Washington, DC, August 1987.

11. O'Bryon, James F. "Live Fire Testing 'Of... Cabbages—and Kings...' and Live Fire

Things." Presented at ITEA Workshop: Live Fire Testing, August 1995, in Live Fire

Test & Evaluation Reference Materials, LFT&E—10 Years and Counting Conference,

Livermore, CA, January 1997.

12. O'Bryon, James F. "Sewing Buttons on Jell-O, Quantifying T&E Benefits and Other

Special Skills." Presented at the Conference on the Economics of Test and

Evaluation, TEREC, Georgia Institute of Technology, Atlanta, GA, October 1997.

13. Ockerman, Donald L. "An Overview of Aircraft LFT&E." Presented at ITEA

Workshop: Pacific Northwest Chapter, August 1996, in Live Fire Test & Evaluation

Reference Materials, LFT&E &E—10 Years and Counting Conference, Livermore, CA,

January 1997.

14. Wojciechowski, Robert. "LFT&E Tutorial, Full-Up, System-Level Testing."

Presented at the LFT&E—10 Years and Counting Conference, Lawrence Livermore

National Laboratory, Livermore, CA, January 1997.

15. Julian, Thomas. "LFT&E Tutorial, Reporting LFT&E Results." Presented at the

LFT&E—10 Years and Counting Conference, Lawrence Livermore National

Laboratory, Livermore, CA, January 1997.

16. National Research Council. Combat Vehicle Vulnerability to Anti-Armor Weapons.

National Academy Press, Washington, DC, 1989.

84
17. National Research Council. Vulnerability Assessment ofAircraft. Washington DC:

National Academy Press, 1993.

18. Deitz, Paul H. "A V/L Taxonomy for Analyzing Ballistic Live-Fire Events/'

Modeling Ballistic Live-Fire Trilogy. ARL-TR-1274, U.S. Army Research Laboratory,

Aberdeen Proving Ground, MD, December 1996.

19. Deitz, Paul H., and Michael W. Starks. "The Generation, Use, and Misuse of TKs' in

Vulnerability/Lethality Analyses." ARL-TR-1640, U.S. Army Research Laboratory,

Aberdeen Proving Ground, MD, March 1998.

20. Deitz, Paul H., and Michael W. Starks. "The Generation, Use, and Misuse of TKs' in

Vulnerability/Lethality Analyses." The Journal ofMilitary Operations Research, vol. 4,

no. 1, pp. 19-33,1999.

21. Deitz, Paul H., Michael W. Starks, Jill H. Smith, and Aivars Ozolins. "Current

Simulation Methods in Military Systems Vulnerability Assessment." BRL-MR-3880,

U.S. Army Ballistic Research Laboratory, Aberdeen Proving Ground, MD,

November 1990.

22. Klopcic, J. Terrence. "The Vulnerability/Lethality Taxonomy as a General Analytic

Procedure." ARL-TR-1944, U.S. Army Research Laboratory, Aberdeen Proving

Ground, MD, May 1999.

23. Klopcic, J. Terrence, Michael W. Starks, and James N. Walbert. "A Taxonomy for the

Vulnerability/Lethality Analysis Process." BRL-MR-3972, U.S. Army Ballistic

Research Laboratory, Aberdeen Proving Ground, MD, May 1992.

24. Reed, Harry L., and J. Terrence Klopcic (eds.), Fundamentals of Vulnerability/Lethality

Assessment, Book 1: Overview. U.S. Army Research Laboratory, Aberdeen Proving

Ground, MD, in press.

85
25. Roach, Lisa K. "Fault Tree Analysis and Extensions of the V/L Process Structure."

ARL-TR-149, U.S. Army Research Laboratory, Aberdeen Proving Ground, MD, June

1993.

26. Deitz, Paul H. "Parsing SMART: What are the Pieces and How do They Fit

Together?" Presented at the Simulation and Modeling for Acquisition,

Requirements, and Training Conference, San Antonio, TX, January 1999.

27. Ansari, Shahid, Jan Bell, and the CAM-I Target Cost Core Group. Target Costing: The

Next Frontier in Strategic Cost Management. Chicago: Irwin Professional Publishing,

1996.

28. Brinker, Barry J. Handbook ofCost Management. Boston, MA: Warren, Gorham, and

Lamont, 1992.

29. Häuser, John R., and Don Clausing. "The House of Quality." Harvard Business

Review, vol. 88, no. 3, pp. 68-73, May-June 1988.

30. Nelson, Martha K. "Prioritizing Alternatives: Quantitative and Qualitative

Analyses." Presented at the 14th International Symposium of Military Operational

Research, The Royal Military College of Science, Shrivenham, Wiltshire, United

Kingdom, September 1997.

31. Saaty, Thomas L. Decision Makingfor Leaders. Pittsburgh, PA: RWS Publications,

1995.

32. Saaty, Thomas L. Fundamentals ofDecision Making and Priority Theory with the

Analytic Hierarchy Process. Pittsburgh, PA: RWS Publications, 1994.

33. U.S. Department of Defense. The Defense Acquisition Deskbook. Section 3.3, Joint

Program Office, Wright Patterson Air Force Base, OH, 15 December 1997.

86
34. Rush, Benjamin C. "Cost as an Independent Variable: Concepts and Risks."

Acquisition Review Quarterly, pp. 161-172, spring 1997.

35. Winter, Angela D. "Simulation and Modeling for Acquisition, Requirements, and

Training, and Cost as an Independent Variable." Army RD&A, pp. 19-21,

November-December 1998.

36. Brmker, Barry J. (ed.). "Activity-Based Cost Systems." Handbook ofCost

Management, Boston, MA: Warren, Gorham, and Lamont, Boston: Part B, 1992.

37. Cooper, Robin. "The Rise of Activity-Based Costing-Part One: What is an ActivityBased Cost System." Journal ofCost Management I, pp. 45-54,1988.

38. Cooper, Robin. "The Rise of Activity-Based Costing-Part Two: When Do I

Need an Activity-Based Cost System." Journal ofCost Management II,

pp. 41-48,1988.

39. Bonner, Sarah. " The Experience Effects in Auditing: The Role of TaskSpecific Knowledge." The Accounting Review, vol. 65, no. 1, pp. 79-92,1990.

40. Nelson, Martha K. "Strategies of Auditors: Evaluation of Sample Results."

Auditing: A Journal ofPractice & Theory, vol. 14, no. 1, pp. 34-49,1995.

41. Sniezek, J. "Groups Under Uncertainty: An Examination of Confidence in

Group Decision Making." Organizational Behavior and Human Decision Processes, vol.

50, no. 3, pp. 124-155,1992.

42. Beers, Suzanne M. "An Intelligent Hierarchial Decision Architecture for

for Operation Test and Evaluation." PhD. Dissertation, Georgia Institute of

Technology, Atlanta, GA, May 1996.

87
43. Frew, Kellye C, and Ellen M. Killion. "User's Manual for Operational

Requirements-based Casualty Assessments (ORCA) Software System-Alpha+

Version." Applied Research Associates, Inc. Report, Albuquerque, NM, July 1996.

44. Glasgow, Jack, and John Paron. "CINC and Service Requirements Analysis

Missions/Operations/Tasks." Dynamics Research Corporation Report, Andover,

MA, July 1999.

88
Acronyms

ABC = Activity-Based Costing

ACAT = Acquisition Category

AHP = Analytic Hierarchy Process

AMCOM = U.S. Army Aviation and Missile Command

AMSAA = U. S. Army Materiel Systems Analysis Activity

APG = Aberdeen Proving Ground, MD

ARL = U.S. Army Research Laboratory

ATACMS = Army Tactical Missile System

ATC = U.S. Army Aberdeen Test Center

ATEC = Army Test and Evaluation Command

BDAR = Battle Damage Assessment and Repair

BFVS = Bradley Fighting Vehicle System

BH&T = Ballistic Hull and Turret

CATV = Cost as an Independent Variable

CSTA = Combat Systems Test Activity

DCSINT = Deputy Chief of Staff for Intelligence

DoD = Department of Defense

DOT&E = Director, Operational Test and Evaluation

DDOT&E (LFT) = Deputy Director, Operational Test and Evaluation, Live-Fire

Testing

DTP = Detailed Test Plan

DTR = Detailed Test Report

DUSA (OR) = Deputy Under Secretary of the Army for Operations Research

DV = Data Void

EAC = Evaluation Analysis Center

EDP = Event Development Plan

EMD = Engineering and Manufacturing Development

FU SL LFT = Full-Up System-Level Live-Fire Test

FU SL LFT&E = Full-Up System-Level Live-Fire Test and Evaluation

GAO = General Accounting Office

GPS = Global Positioning System

HQ, TECOM = Headquarters, U.S. Army Test and Evaluation Command

HS = High Survivability

IDA = Institute for Defense Analysis

IEP/TDP = Independent Evaluation Plan/Test Design Plan

LFIFT = Live-Fire Integrated Product Team

LFT = Live-Fire Test

LFT&E = Live-Fire Test and Evaluation

LRIP = Low-Rate Initial Production

M&S = Modeling and Simulation

MAM = Mission Area Manager

MOE = Measure of Effectiveness

MRMC = U.S. Army Medical Research and Materiel Command

MUVES = Modular UNIX-based Vulnerability Estimation Suite

ÖEC = Operational Evaluation Command

89
OPTEC = U.S. Army Operational Test and Evaluation Command

ORD = Operational Requirements Document

OSD = Office of the Secretary of Defense

PLO = Petroleum, Lubricants, and Oil

PM = Project Manager

PMO = Project Manager Office

PQT = Production Qualification Test

QFD = Quality Function Deployment

SAFE = Stochastic Analysis of Fragment Effects

SEP = System Enhancement Package

SER = System Evaluation Report

SLAD = Survivability/Lethality Analysis Directorate

SQuASH = Stochastic Quantitative Analysis of System Hierarchies

T&E = Test and Evaluation

TACOM = U.S. Army Tank-automotive and Armaments Command

Taxonomy VA = Taxonomy of Vulnerability Assessment

TECOM = U.S. Army Test and Evaluation Command

TEMP = Test and Evaluation Master Plan

TRADOC = U.S. Army Training and Doctrine Command

USAOC&S = U.S. Army Ordnance Center and School

USATC&S = U.S. Army Transportation Center and School

V&V = Verification and Validation

VA Risk = Vulnerability Assessment Risk

VAP = Vulnerability Assessment Plan

V/L = Vulnerability or Lethality

V/L Taxonomy = Taxonomy of the V/L Analysis Process

W&A = Verification, Validation, and Accreditation

WSMR = White Sands Missile Range

90
NO. OF

COPIES ORGANIZATION

NO. OF

COPIES ORGANIZATION

DEFENSE TECHNICAL

INFORMATION CENTER

DTIC DDA

8725 JOHN J KINGMAN RD

STE 0944

FT BELVOIR VA 22060-6218

1 DIRECTOR

US ARMY RESEARCH LAB

AMSRLD

D R SMITH

2800 POWDER MILL RD

ADELPHI MD 20783-1197

HQDA

DAMOFDQ

D SCHMIDT

400 ARMY PENTAGON

WASHINGTON DC 20310-0460

DIRECTOR

US ARMY RESEARCH LAB

AMSRLDD

2800 POWDER MILL RD

ADELPHI MD 20783-1197

OSD

OUSD(A&T)/ODDDR&E(R)

RJTREW

THE PENTAGON

WASHINGTON DC 20301-7100

DIRECTOR

US ARMY RESEARCH LAB

AMSRL CS AS (RECORDS MGMT)

2800 POWDER MILL RD

ADELPHIMD 20783-1145

DPTY CG FOR RDA

US ARMY MATERIEL CMD

AMCRDA

5001 EISENHOWER AVE

ALEXANDRIA VA 22333-0001

DIRECTOR

US ARMY RESEARCH LAB

AMSRL CILL

2800 POWDER MILL RD

ADELPHI MD 20783-1145

INST FOR ADVNCD TCHNLGY

THE UNIV OF TEXAS AT AUSTIN

PO BOX 202797

AUSTIN TX 78720-2797

DARPA

B KASPAR

3701 N FAIRFAX DR

ARLINGTON VA 22203-1714

NAVAL SURFACE WARFARE CTR

CODE B07 J PENNELLA

17320 DAHLGREN RD

BLDG 1470 RM 1101

DAHLGREN VA 22448-5100

US MILITARY ACADEMY

MATH SCI CTR OF EXCELLENCE

DEPT OF MATHEMATICAL SCI

MADN MATH

THAYER HALL

WEST POINT NY 10996-1786

ABERDEEN PROVING GROUND

DIRUSARL

AMSRL CI LP (BLDG 305)

91
NO. OF

COPIES ORGANIZATION

NO. OF

COPIES ORGANIZATION

OASD C3I

MR BUCHHEISTER

RM3D174

6000 DEFENSE PENTAGON

WASHINGTON DC 20310-6000

1 OADCSOPS FORCE DEV DIR

DAMOFDZ

ROOM 3A522

460 ARMY PENTAGON

WASHINGTON DC 20310-0460

OUSD AT STRT TAC SYS

DR SCHNEITER

RM 3E130

3090 DEFENSE PENTAGON

WASHINGTON DC 20310-3090

HQDA

ODCSPER

DAPE MR (RM 2C733)

300 ARMY PENTAGON

WASHINGTON DC 20310-0300

OUSD AT S&T AIR WARFARE

RM 3E139

R MUTZELBUG

3090 DEFENSE PENTAGON

WASHINGTON DC 20301-3090

OUSD AT S&T LAND WARFARE

RMEB1060

AVIILU

3090 DEFENSE PENTAGON

WASHINGTON DC 20310-3090

UNDER SEC OF THE ARMY

DUSA OR

ROOM 2E660

102 ARMY PENTAGON

WASHINGTON DC 20310-0102

ASST SECY ARMY

ACQUISITION LOGISTICS TCHNLGY

SARD ZD ROOM 2E673

103 ARMY PENTAGON

WASHINGTON DC 20310-0103

ASST SECY ARMY

ACQUISITION LOGISTICS TCHNLGY

SARD ZP ROOM 2E661

103 ARMY PENTAGON

WASHINGTON DC 20310-0103

ASST SECY ARMY

ACQUISITION LOGISTICS TCHNLGY

SAAL ZS ROOM 3E448

103 ARMY PENTAGON

WASHINGTON DC 20310-0103

US ARMY MATERIEL CMD

DEP CHF OF STAFF FOR RDA

SCIENCE TECH ENG

AMCRDA

5001 EISENHOWER AVE

ALEXANDRIA VA 22333-0001

US ARMY MATERIEL CMD

DEP CHF OF STAFF FOR RDA

SCIENCE TECH ENG

AMCRDA T

5001 EISENHOWER AVE

ALEXANDRIA VA 22333-0001

US ARMY ARMAMENT RDEC

AMSTA AR TD

MFISETTEBLDG1

PICATINNY ARSENAL NJ

07806-5000

SBCCOM RDEC

AMSSB RTD

JZARZYCKI

5183BLACKHAWKRD

APG MD 21010-5424

US ARMY MISSILE RDEC

AMSMIRD

W MCCORKLE

RSA AL 35898-5240

NATICK SOLDIER CENTER

SBCNT

P BRANDLER

KANSAS STREET

NATICK MA 01760-5056

92
NO. OF

COPIES ORGANIZATION

NO. OF

COPIES ORGANIZATION

US ARMY TANK AUTOMTV RDEC

AMSTATR

JCHAPIN

WARREN MI 48397-5000

US ARMY INFO SYS ENGRG CMD

AMSEL IE TD

FJENIA

FT HUACHUCA AZ 85613-5300

US ARMY SIM TRNGINST CMD

AMSTI CG

M MACEDONIA

12350 RESEARCH PKWY

ORLANDO FL 32826-3726

US ARMY TRADOC

BATTLELAB INTEGRATION

TECH B CONCEPTS

DIR ATCD B

FT MONROE VA 23561-5000

ARMY TRADOC ANL CTR

ATRCW

MRKEINTZ

WSMRNM 88002-5502

US ARMY RESEARCH LAB

AMSRLSL

PLANS AND PGMS MGR

WSMRNM 88002-5513

ABERDEEN PROVING GROUND

US ARMY DEV TEST COM

CSTE DTC TT T

APGMD 21005-5055

US ARMY EVALUATION CENTER

CSTE AEC MR HUGHES

4120 SUSQUEHANNA AVE

APG MD 21005-3013

US ARMY EVALUATION CENTER

CSTE AEC SV

D DELATTRE

LFILLINGER

S GRILL

R LAUGHMAN

4120 SUSQUEHANNA AVE

APG MD 21005-3013

US ARMY RESEARCH LAB

AMSRLSL

DRWADE

J BEILFUSS

US ARMY RESEARCH LAB

AMSRLSL B

MS SMITH

JFRANZ

M VOGEL

W WINNER

US ARMY RESEARCH LAB

AMSRLSL E

WSMRNM 88002-5513

US ARMY RESEARCH LAB

AMSRL SL BA

M RITONDO

US ARMY RESEARCH LAB

AMSRL SLEA

RFLORES

WSMRNM 88002-5513

US ARMY RESEARCH LAB

AMSRLSL EM

J PALOMO

WSMRNM 88002-5513

US ARMY RESEARCH LAB

AMSRLSL El

J NOWAK

FT MONMOUTH NJ 07703-5602

5 US ARMY RESEARCH LAB

AMSRL SLBD

J MORRISSEY

21 US ARMY RESEARCH LAB

AMSRL SL BE

D BELY (20 CPS)

R SANDMEYER

5 US ARMY RESEARCH LAB

AMSRL SL BG

AYOUNG

5 US ARMY RESEARCH LAB

AMSRL SLBN

D FARENWALD

APG MD 21010-5423

93
NO. OF

COPIES ORGANIZATION

ABERDEEN PROVING GROUND CONTINUED

1 US ARMY RESEARCH LAB

AMSRL SL E

MSTARKS

1 US ARMY RESEARCH LAB

AMSRL SLEA

D BAYLOR

1 US ARMY RESEARCH LAB

AMSRL SL EM

JFEENEY

1 US ARMY RESEARCH LAB

AMSRL SL EC

E PANUSKA

94
NO. OF

COPIES ORGANIZATION

NO. OF

COPIES ORGANIZATION

1 SAIC

6200 UPTOWN BLVD

SUITE 300

C E ADOLPH

ALBUQUERQUE NM 87110

1 DIRECTOR

GEORGIA INSTITUTE

OF TECHNOLOGY

GTRI/ELSYSTEST & EVAL RSCH

& EDUCATION CTR

DR S M BLANKENSHIP

ATLANTA GA 30332-0840

1 CHIEF

CREW STATION R&D BRANCH

MANAGER ADVANCED

M&S COMPLEX

N BÜCHER

AMES RESEARCH CTR MS 243 3

MOFFETT FIELD, CA 94035-1000

1 DIR

US ARMY MODEL &

SIMULATION OFFICE

DAMO ZS W LUNCEFORD

1111 JEFFERSON DAVIS HIGHWAY

CRYSTAL GATEWAY III

SUITE 503 E

ARLINGTON VA 22202

1 US TACOM ARDEC

OR ANALYST

AMSTA AR FSS J C CHU

PICATINNY ARSENAL NJ

07806-5000

1 DIRECTOR

CTR FOR PROFESSIONAL

DEVELOPMENT & TNG

UNIV OF TX AT AUSTIN

DRJG DAVIS

4030 2 W BRAKER LN

AUSTIN TX 78769-5329

1 DIRECTOR

SYSTEMS ANALYSIS

LOCKHEED MARTIN

ELECTRONICS & MISSILES

RE DOUGLAS

5600 SAND LAKE RD MP 211

ORLANDO FL 32819-8907

1 DIRECTOR

GEORGIA TECH RSCH INSTITUTE

ARLINGTON RESEARCH LAB

WEEAGAR

1700 N MOORE ST SUTE 1910

ARLINGTON VA 22209

1 AAAV TECHNOLOGY CTR

DRPM AAAV

DWERDLEY

991 ANNAPOLIS WAY

WOODBRIDGE VA 22191-1215

1 OPERATIONS RSCH ANALYST USAF

CBT WPNS SYS ANLYS DIV

HQ ACC XRYS

NL EVANS

204 ODD VLVD, STE 226

LANGLEY AFB VA 23665-2777

1 GEORGIA TECH RSCH INSTITUTE

GA INSTITUTE OF TECHNOLOGY

ELECTRONIC SYSTEMS LAB

CONCEPTS ANALYSIS DIV

J R GIBBONS

ATLANTA GA 30332-0800

1 THE MITRE CORP

SEN M&S ENGINEER

P A GLASOW

1820 DOLLEY MADISON BLVD W626

MCLEAN VA 22102

1 NEWPORT NEWS SHIPBUILDING

JR GRIFFEN

4101 WASHINGTON AVE

NEWPORT NEWS VA 23607-2770

95
NO. OF

COPIES ORGANIZATION

NO. OF

COPIES ORGANIZATION

1 APPLIED RSCH ASSOCIATES INC

W HACKER

4300 SAN MATEO NE

SUITE A 220

ALBUQUERQUE NM 87110

1 CF HAWKINS

CONSULTANT

13140 LAKEHDLLDR

NOKESVILLE VA 20181-3328

1 ADUSA(OR)

UNDER SECY OF THE ARMY

DRRGHINKLE

SAUS OR

RM 1E643

102 ARMY PENTAGON

WASHINGTON DC 20310-0102

1 SR SYSTEMS ANALYST

RKANE

1250 ACADEMY PARK LOOP

SUITE 119

COLORADO SPRINGS CO

80910-3707

1 CDR

USA TACOM

AMSTA UB M J KING

WARREN MI 48397-5000

1 JOINT WARFARE ANALYSIS DEPT

APPLIED PHYSICS LAB

JOHNS HOPKINS UNIVERSITY

W M KROSHL

11100 JOHNS HOPKINS RD

LAUREL MD 20723-6099

1 LOCKHEED MARTIN

AERONAUTICAL SYSTEMS

WE KRUPP

86 S COBB DR

MARIETTA GA 30063-0484

1 PROGRAM AREA MNGR

JOHNS HOPKINS UNIVERSITY

APPLIED LABORATORY

DR R R LUMAN

11100 JOHNS HOPKINS RD

LAUREL MD 20723

1 OPERATIONS RSCH ANALYST

PROGRAM ANALYSIS AND EVAL

DIRECTORATE

MARCORSYSCOM PAE

D G MAYLUM

2033 BARNETT AVE SUITE 315

QUANTICO VA 22134-5010

1 US ARMY TACOM

SFAEGCSSWBVQ

M S NEIDELMAN

WARREN MI 48397-5000

1 DRRNEW

10959 DEBORAH DR

POTOMAC MD 20854

1 DEPUTY DIRECTOR

OSD/DOT&E/LFT&E

RM 1C730A

JFOBRYON

1700 DEFENSE PENTAGON

WASHINGTON DC 20301-1700

1 PARICIO SPECIALTY ENGRG CO

R PARICIO

16775 COUNTY RD 362

BUENA VISTA CO 81211

1 ANALYST OPERATIONS RSCH

KH PILGRIM

45 C WILSON RD

WEST POINT NY 10996

1 US ARMY TACOM

PM GSIHTIDIV

MRYZYI

PM SURVIVABILITY SYSTEMS

SFAEGCSSWGSIH

WARREN MI 48397-5000

1 DDR INC

DIRECTOR OF BUS DEVEL & SYS

W R SCHILLING

SUITE 515

10560 MAIN ST

FAIRFAX VA 22030

96
NO. OF

COPIES ORGANIZATION

NO. OF

COPIES ORGANIZATION

1 SCIENCE ADVISOR TO THE DIR

OSD/DOT&E

DREASEGLIE

1700 DEFENSE PENTAGON

WASHINGTON DC 20301-1700

1 APPLIED RSCH LABORATORIES

THE UNTV OF TX AT AUSTIN

J SHEEHAN

AUSTIN TX 78713-8029

1 LOGICON

DR C STANIEC

2100 WASHINGTON BLVD

ARLINGTON VA 22204-5706

1 LOGICON SYSCON

E SWARTZ

2341 JEFFERSON DAVIS HWY

SUITE 1138

ARLINGTON VA 22202

3 INSTITUTE FOR

DEFENSE ANALYSES

DR TONNESSEN

DR TURNER

EUSANIO

1801 N BEAUREGARD ST

ALEXANDRIA VA 22311-1733

3 SURIVCE ENGINEERING CO

D VAN DUSEN

ALAGRANGE

E EDWARDS

1003 OLD PHILADELPHIA RD

SUITE 103

ABERDEEN MD 21001-4011

1 USARMYTACOM

SFAEASMBVCC

J P VELLIKY

WARREN MI 38497-5000

1 HEAD WARFARE PRGMS T&E

OFC OF CNO OPNAV N912C

PRESIDENTIAL TOWER 3D FLR

CPTRLVIRGELIOUSN

2000 NAVY PENTAGON

WASHINGTON DC 20350-2000

1 CHIEF SCIENTIST TECH DIR

AFOTEC CN

DR ML WILLIAMS

KIRTLAND AFB NM 87117-5558

1 EPVISCO

3752 CAPULET TERRACE

SILVER SPRING MD 20906

1 BOOZ ALLEN & HAMILTON INC

LH WOODS

SUITE 131

4141 COL GLENN HWY

BEAVERCREEK OH 45431

1 HQ SWC AEA

MAJ S BEERS

730 IRWIN AVE SUITE 83

SCHRJJEVER AFB CI 80912-7383

1 HQDA

ODCSOPS

DAMO ZD V BETTENCOURT

400 ARMY PENTAGON RM 3A538

WASHINGTON DC 20310-0400

1 US ARMY TEST & EVALUATION

MANAGEMENT AGENCY

DACSTE

DRFOULKES

PENTAGON RM3C567

WASHINGTON DC 20310

1 RHELMUTH

12 KIMBERLY OR

BRUNSWICK ME 04011

1 THE MITRE CORP

MS 625 W150 M LAVESfE

1820 DOLLEY MADISON BLVD

MCLEAN VA 22102-3481

1 US ARMY COMMUNICATIONS

ELECTRONICS CMD

AMSEL RD DB MAJ LOCKHART

BLDG 2700

FT MONMOUTH NJ 07703

97
NO. OF

COPIES ORGANIZATION

NO. OF

COPIES ORGANIZATION

1 JOINT ADV DIST SIM

JOINT TEST AND EVAL

TECHNICAL ADVISOR E KECK

11104 MENAUL BLVD NE

ALBUQUERQUE NM 87112

4 UNIVERSITY OF PITTSBURGH

KATZ SCHOOL OF BUSINESS

J BIRNBERG

J H EVANS

V HOFFMAN

D MOSER

PITTSBURGH PA 15260

20 CHAIR

FRANKLIN AND MARSHALL

COLLEGE

DEPARTMENT OF BUS ADMIN

M K NELSON

LANCASTER PA 17604-3003

1 APPLIED RSCH ASSOCIATES INC

R SCUNGIO

219 W BELAIR AVE SUITE 5

ABERDEEN MD 21001

1 ASI TEST CONSULTANT

G BURGNER

825 N DOWNS SUITE C

RIDGECREST CA 93555

1 BOOZ ALLEN & HAMILTON INC

K CROSSWAITE

WPAFB OH 45433-7542

3 OSD/DOT&FVLFT&E

T JULIAN

FMELLO

RWOJCIECHOWSKI

RM 1C730

1700 DEFENSE PENTAGON

WASHINGTON DC 20301-1700

1 PM ATACMS BAT

SFAE MSL AB C Q T R CHELEKIS

RSA AL 35898-5650

1 QUANTUM RSCH INTERNATNL

D BRISTOL

BEL AIR MD

1 QUESTECH INC

M EGLESTON

901 N STUART ST

SUITE 605

ARLINGTON VA 22203

1 DIR OPERATIONAL TEST

& EVALUATION

PCOYLE

RM3E318

1700 DEFENSE PENTAGON

WASHINGTON DC 20301-1700

1 DIR OPERATIONAL TEST

& EVALUATION

HMAIR

1700 DEFENSE PENTAGON

WASHINGTON DC 20301-1700

1 US ARMY DEVELOPMENTAL

TEST COMMAND

CSTE DTC TT T N HARRINGTON

314 LONGS CORNER RD

APG MD 21005-5055

1 US ARMY TEST AND EVALUATION

COMMAND

BBARR

PARK CENTER IV

4501 FORT AVE

ALEXANDRIA VA 22303

1 DIRECTOR

US ARMY RESEARCH LAB

AMSRL PBP J SCZEPANSKI

2800 POWDER MILL RD

ADELPHI MD 20783-1197

1 NAVAL SURFACE WARFARE CTR

CARDEROCK DIVISION

F FISCH

BETHESDA MD 20084-5000

98
NO. OF

COPIES

1

ORGANIZATION

NAVAL SURFACE WARFARE CTR

DAHLGREN DIVISION

G24TWASMUND

17320 DAHLGREN RD

DAHLGREN VA 22448-5100

OFFICE OF NAVAL RESEARCH

TECH ÄRA MGR SURF AERO

WEAPONRY CODE 35

D SIEGEL

800 N QUESfCY ST

ARLINGTON VA 22217-5660

USAF PHILLIPS LABORATORY

PL WSW BAKER

3550 ABERDEEN AVE SE

KIRKLAND AFB NM 87117-5576

USAF WRIGHT LABORATORY

WLFIVS

MLENTZ

WRIGHT PAT AFB OH 45433-6553

USAF WRIGHT LABORATORY

1901 TENTH ST

RLAUZZE

WRIGHT PAT AFB OH 45433-7605

USAFASCXRA

BLDG16

2275D ST SUITE 10

HGRIFFIS

WRIGHT PAT AFB OH 45433-7227

USAF OPER TEST & EVAL CMD

AFOCTE CN

M WILLIAMS

8500 GIBSON BVLD SE

KIRKLAND AFB NM 87117-5558

PEO AIR & MSL DEFENSE

SFAEAMD

PO BOX 1500

HUNTSVILLE AL 35807-3801

PEO ABE & MSL DEFENSE

SFAEAMD ME

PO BOX 1500

HUNTSVILLE AL 35807-3801

NO. OF

COPIES ORGANIZATION

1 PEO AIR & MSL DEFENSE

SFAE AMD THA

PO BOX 1500

HUNTSVILLE AL 35807-3801

1 PEO AIR & MSL DEFENSE

SFAE AMD PA

PO BOX 1500

HUNTSVILLE AL 35807-3801

1 PEO AVIATION

SFAEAV

BLDG5681

RSA AL 35898

1 PEO AVIATION

SFAEAVAAH

BLDG5681

RSA AL 35898

1 PEO AVIATION

SFAEAVAAHATH

BLDG5681

RSA AL 35898

1 PEO AVIATION

SFAEAVAAHLBA

BLDG5681

RSA AL 35898

1 PEO AVIATION

SFAEAVIR

BLDG 5681

RSA AL 35898

1 PEO AVIATION

SFAEAVCH

BLDG 5678

RSA AL 35898

1 PEO AVIATION

SFAEAVCH ICC

BLDG 5681

RSA AL 35898

1 PEO AVIATION

SFAE AV RAH

BLDG 5681

99

RSA AL 35898
NO. OF NO. OF

COPIES ORGANIZATION COPIES ORGANIZATION

1 PEO GRD CMBT & SPT SYSTEMS 1 PEO GRD CMBT & SPT SYSTEMS

SFAE GCSS SFAE GCSS W GSI '

BLDG 171 BLDG 229

PICATINNY ARSENAL NJ WARREN MI 48397-5000

07806-5000

1 PRO GRD CMBT & SPT SYSTEMS

*

1 PEO GRD CMBT & SPT SYSTEMS

SFAE GCSS W AB

WARREN MI 48397-5000

SFAE GCSS W MR

MAIL STOP 508

WARREN MI 48397-5000

*

2 PEO GRD CMBT & SPT SYSTEMS

SFAE GCSS W AB

6501 11 MILE RD

WARREN MI 48397-5000

1 PEO GRD CMBT & SPT SYSTEMS

AMSTATRD/FSCS

MAIL STOP 275

WARREN MI 48397-5000

4 PEO GRD CMBT & SPT SYSTEMS

SFAECGSSWBV

WARREN MI 48397-5000

2 PEO GRD CMBT & SPT SYSTEMS

SFAE GCSS ARMS

PICATINNY ARSENAL NJ

07806-5000

1 PEO GRD CMBT & SPT SYSTEMS

SFAE GCSS CR 1 PEO GRD CMBT & SPT SYSTEMS

BLDG 171 SFAE GCSS TMA

PICATINNY ARSENAL NJ BLDG 354

07806-5000 PICATINNY ARSENAL NJ

07806-5000

1 PEO GRD CMBT & SPT SYSTEMS

SFAE GCSS CR A 1 PEO GRD CMBT & SPT SYSTEMS

BLDG 171 SFAE GCSS TMA SM

PICATINNY ARSENAL NJ PICATINNY ARSENAL NJ

07806-5000 07806-5000

1 PEO GRD CMBT & SPT SYSTEMS

SFAE GCSS CR M

BLDG 171

PICATINNY ARSENAL NJ

07806-5000

1

1

PEO TACTICAL MISSILES

SFAE MSL

BLDG 5250

RSA AL 35898-8000

PM AIR TO GRD MSL SYSTEMS OFC

1 PEO GRD CMBT & SPT SYSTEMS

SFAE GCSS JLW

BLDG 151

SFAE MSL HD

RSA AL 35898-5610

PICATINNY ARSENAL NJ 1 PEO TACTICAL MISSILES r

07806-5000 SFAE MSL AB

BLDG 5250

1 PEO GRD CMBT & SPT SYSTEMS

SFAE GCSS W MTV

RSA AL 35898-5650 *

WARREN MI 48397-5000 1 PM ARMY TACMS BAT PROJ OFC

SFAE MSL AB PB

RSA AL 35898

100
NO. OF

COPIES ORGANIZATION

NO. OF

COPIES ORGANIZATION

1 PM ARMY TACMS BAT PROJ OFC

SFAE MSL AB PI

LTC K BROWINING

RSA AL 35898

1 PM ARMY TACMS BAT PROJ OFC

SFAE MSL AB PP

RSA AL 35898

1 PM CLOSE CMBT ANT-ARMOR

WPNS SYS PROJ OFC

SFAE MSL CC

RSA AL 35898-5710

1 PRODUCT MANAGER

ADVANCED TOW ACQUISITION

SYSTEM

SFAE MSL CC AT

RSA AL 35898-8000

1 PMO

SFAE MSL AM

BLDG3651

RSA AL 35898-5720

1 PEO TACTICAL MISSILES

MLRS

SFAE MSL ML

BLDG 5250 RM B200

RSA AL 35898-5700

1 PEO TACTICAL MISSILES

HIGH MOBILITY ARTILLERY

ROCKET SYSTEM

SFAE MSL ML HM

RSA AL 35898-5710

1 US ARMY SPECIAL OPS CMD

TECHNLGY APPLICATION PROG OFC

AMSAMRDAAZT

BLDG 401 LEE BLVD

FT EUSTIS VA 23604-5577

1 DSAUSARMYAVN&MSLCMD

AMSAM DSA

RSA AL 35898-5000

1 US ARMY AVN & MSL CMD

AMSAM DSA MP

MULTI PURPOSE INDIVIDUAL

MUNITION/SHORT RANGE

ASSAULT WPN

RSA AL 35898

1 US ARMY AMCOM

AMSAM DSA AS ASH

SCOUT/ATTACK

RSA AL 35898

1 US ARMY AMCOM

SHORT RANGE AIR DEFENSE

AMSAM DSA SH

RSA AL 35898

1 US ARMY AMCOM

UTILITY HELICOPTER

AMSAM DSA UH

RSA AL 35898

1 US ARMY AMCOM

UNMANNED GROUND VEH

AMSAM DSA UG

RSA AL 35898

PEO TACTICAL MISSILES

MLRS IMPROVED LAUNCHER

MECHANICAL SYSTEM

SFAE MSL ML IL

RSA AL 35898-5700

US ARMY SOLDIER & BIOLOGICAL

CHEMICAL COMMAND

LAND WARRIOR

10401 TOTTEN RD SUITE 121

FT BELVOIR VA 22060-5852

PEO TACTICA MISSILES

MLRS PRECISION GUIDED

MUNITIONS

SFAE MSL ML PGM

RSA AL 35898-5700

US ARMY TACOM

AMSTA LC

WARREN MI 48397-5000

US ARMY TACOM

AMSTA LC AH

WARREN MI 48397-5000

101
NO. OF

COPIES ORGANIZATION

1 US ARMY TACOM

AMSTA LC AL

WARREN MI 48397-5000

1 US ARMY TACOM

AMSTA LC AM

PICATINNY ARSENAL NJ

07806-5000

1 MARINE CORPS

INTELLIGENCE ACTIVITY

DRCPM AAAV T M LEAF

991 ANNAPOLIS WAY

WOODBRIDGE VA 22191-1215

1 PM ABRAMS TANK SYSTEM

SFAE GCSS W AB QT P LEITHEISER

WARREN MI 48397-5000

ABERDEEN PROVING GROUND

22 DIR USAMSAA

E ATZINGER

P BEAVERS

W BROOKS JR

COLBUCHA

PHDEITZ(IOCPS)

CFOX

WHEAPES

W NIEMEYER

B PARKS

TRUTH

D J SHATTER

R THOMPSON

A WONG

APG MD 21005

102
REPORT DOCUMENTATION PAGE Form Approvtd

OMB No. 0704-0188

Public reporting burden (or thla collection ol Uotmatlon la eetlmaud to average 1 hour per reaponee, Including the dme tor reviewing Inelrucdone. oearcMng existing data eoureeeT

gathering and maintaining the data needed, and completing and reviewing the coeeedon of Information. Sand eommente regarding «la burden eodmale or any other aapeet of thl»

collection of Information, Including suggestions for reducing thle burden, to Waahlngton Headquarters Service», Directorate tor Information Operadone and Reporte, 1215 Jeffereon

Davl» Highway, Suite 1204, Arlington, V* 22202-4302, and lo the Office of Management and Budget Paperwork Roducdon Pro|ect<07f>4-II18B), Waehlnqton, DC 20503,

1 AGENCY USE ONLY (Letve blank) 12. REPORT DATE rII REPORT TVP'E AND DAtES COVERED

April 2000 Final, Apr 97-Jun 99

5. FUNDING NUMBERS

1L622618AH80

4. TITLE AND SUBTITLE

Vulnerability and Lethality Assessment:

Live-Fire Testing and Evaluation

The Role of Full-Up System-Level

6. AUTHOR(S)

Martha K. Nelson*

4. PERFORMING ORGANIZATION1 REPORT NUMBER

7. PERFORMING ORGANIZATION NAMri^ AN5 AbÖRESSfESi

Franklin and Marshall College

Department of Business Administration

P.O. Box 3003

Lancaster, PA 17604-3003

IÖ-SPöNSORING/MONITöRING

AGENCY REPORT NUMBER

ARL-CR450

9. SPONSORING/MONITORING AGENCY NAMES(S) AND ADDRESS(ES)

U.S. Army Research Laboratory

ATTN: AMSRL-SL-BE

Aberdeen Proving Ground, MD 21005-5068

n.süppLäMfcNfARVN6TES

Franklin and Marshall College, Department of Business Administration, P.O. Box 3003, Lancaster, PA 17604-3003

12a. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.

12b. DISTRIBUTION CODE

13. ABSTRACT (Maximum 200 words)

In this era of decreased defense budgets and limited resources, it is important for decision-makers to determine the

optimal strategy for assessing the vulnerability or lethality (V/L) of a weapon system and the role of Full-Up

System-Level Live-Fire Test and Evaluation (FU SL LFT&E) in that strategy. This report presents a foundation for a

methodology to (1) identify, measure, and categorize the costs and benefits of FU SL LFT&E; (2) determine the

relative significance of FU SL LFT&E to the V/L assessment plan of the weapon system; and (3) compare competing

V/L assessment plans for a system.

Descriptions of the activities of FU SL LFT&E, the costs and benefits (i.e., impacts) of those activities, the

complexities encountered in the reporting of the impacts, and approaches for addressing the complexities are

presented. A discussion of the contributions of FU SL LFT&E to a V/L assessment strategy and suggestions for

improving the cost-effectiveness of FU SL LFT&E are included.

The Taxonomy of the V/L Analysis Process (Deitz and Ozolins 1989) is proposed as a framework for identifying

the data voids to be addressed in a V/L assessment plan. The potential of adapting the principles of the Cost as an

Independent Variable methodology to the evaluation of competing plans of V/L assessment is explored.

14. SUBJECT TERMS

live-fire test and evaluation, cost-benefit analysis, vulnerability/assessment, lethality

assessment, full-up system-level test, vulnerability/lethality assessment strategy cost as an

independent variable, vulnerability/lethality analysis taxonomy

17. SECURITY CLASSIFICATION

OF REPORT

UNCLASSIFIED

NSN 7540-01-280-5500

18. SECURITY CLASSIFICATION

OF THIS PAGE

UNCLASSIFIED

19. SECURITY CLASSIFICATION

OF ABSTRACT

UNCLASSIFIED

is. NUMBER OF PAGES

116

16. PRICE CODE

20. LIMITATION OF ABSTRACT

UL

103

Standard Form 298 (Rev. 2-89)

Prescribed by ANSI Std. 239-18 298-102
INTENTIONALLY LEFT BLANK.

104
USER EVALUATION SHEET/CHANGE OF ADDRESS

This Laboratory undertakes a continuing effort to improve the quality ofthe reports it publishes. Your comments/answers to

the items/questions below will aid us in our efforts.

1. ARL Report Number/Author Nelson (TOC: D. Belv) Date of Report April 2000

2. Date Report Received

3. Does this report satisfy a need? (Comment on purpose, related project, or other area ofinterest for which the report will be

used.) —

4. Specifically, how is the report being used? (Information source, design data, procedure, source ofideas, etc.).

5. Has the information in this report led to any quantitative savings as far as man-hours or dollars saved, operating costs

avoided, or efficiencies achieved, etc? If so, please elaborate.

6. General Comments. What do you think should be changed to improve future reports? (Indicate changes to organization,

technical content, format, etc.) . —

Organization

CURRENT Name E-mail Name

ADDRESS .

Street or P.O. Box No.

City, State, Zip Code

, 7. Ifindicating a Change of Address or Address Correction, please provide the Current or Correct address above and the Old

or Incorrect address below.

Organization

OLD Name

ADDRESS

Street or P.O. Box No.

City, State, Zip Code

(Remove this sheet, fold as indicated, tape closed, and mail.)

(DO NOT STAPLE)