GAO 

United States Government Accountability Office

Report to Congressional Requesters

WARFIGHTER 

SUPPORT 

Independent Expert 

Assessment of Army 

Body Armor Test 

Results and 

Procedures Needed 

Before Fielding 

October 2009 

GAO-10-119 
What GAO Found

United States Government Accountability Office

Why GAO Did This Study

Highlights

Accountability Integrity Reliability

October 2009

WARFIGHTER SUPPORT

Independent Expert Assessment of Army Body Armor 

Test Results and Procedures Needed Before Fielding

Highlights of GAO-10-119, a report to 

congressional requesters 

The Army has issued soldiers in Iraq 

and Afghanistan personal body 

armor, comprising an outer 

protective vest and ceramic plate 

inserts. GAO observed Preliminary 

Design Model testing of new plate 

designs, which resulted in the 

Army’s awarding contracts in 

September 2008 valued at a total of 

over $8 billion to vendors of the 

designs that passed that testing. 

Between November and December 

2008, the Army conducted further 

testing, called First Article Testing, 

on these designs. GAO is reporting 

on the degree to which the Army 

followed its established testing 

protocols during these two tests.

GAO did not provide an expert 

ballistics evaluation of the results of 

testing. GAO, using a structured, 

GAO-developed data collection 

instrument, observed both tests at 

the Army’s Aberdeen Test Center, 

analyzed data, and interviewed 

agency and industry officials to 

evaluate observed deviations from 

testing protocols. However, 

independent ballistics testing 

expertise is needed to determine 

the full effect of these deviations. 

What GAO Recommends

GAO makes several 

recommendations, which are 

discussed on the next page, 

including to provide for an 

independent assessment of First 

Article Testing data, to assess the 

need to change Army’s procedures 

based on that assessment, 

documenting this and all other key 

decisions made, and to provide for 

an external peer review of 

Aberdeen Test Center’s protocols, 

facilities, and instrumentation. 

During Preliminary Design Model testing the Army took significant steps to 

run a controlled test and maintain consistency throughout the process, but the 

Army did not always follow established testing protocols and, as a result, did 

not achieve its intended test objective of determining as a basis for awarding 

contracts which designs met performance requirements. In the most 

consequential of the Army’s deviations from testing protocols, the Army 

testers incorrectly measured the amount of force absorbed by the plate 

designs by measuring back-face deformation in the clay backing at the point 

of aim rather than at the deepest point of depression. The graphic below 

depicts the difference between the point of aim and the deepest point. 

Source: GAO analysis.

Guide plane

Point of aim 

depression

Deepest point 

of depression

Clay backing

Armor plate

Back-face 

deformation

Point of aim Aim laser

Shooting barrel

Line of fire

Army testers recognized the error after completing about a third of the test 

and then changed the test plan to call for measuring at the point of aim and 

likewise issued a modification to the contract solicitation. At least two of the 

eight designs that passed Preliminary Design Model testing and were awarded 

contracts would have failed if measurements had been made to the deepest 

point of depression. The deviations from the testing protocols were the result 

of Aberdeen Test Center’s incorrectly interpreting the testing protocols. In all 

these cases of deviations from the testing protocols, the Aberdeen Test 

Center’s implemented procedures were not reviewed or approved by the Army 

and Department of Defense officials responsible for approving the testing 

protocols. After concerns were raised regarding the Preliminary Design Model 

testing, the decision was made not to field any of the plate designs awarded 

contracts until after First Article Testing was conducted. View GAO-10-119 or key components.

For more information, contact William M. Solis 

at (202) 512-8365 or solisw@gao.gov. 
United States Government Accountability Office

Highlights of GAO-10-119 (continued) 

During First Article Testing, the Army addressed some of 

the problems identified during Preliminary Design Model 

testing, but GAO observed instances in which Army 

testers did not follow the established testing protocols 

and did not maintain internal controls over the integrity 

and reliability of data, raising questions as to whether 

the Army met its First Article Test objective of 

determining whether each of the contracted designs met 

performance requirements. The following are examples 

of deviations from testing protocols and other issues 

that GAO observed: 

• The clay backing placed behind the plates during 

ballistics testing was not always calibrated in 

accordance with testing protocols and was 

exposed to rain on one day, potentially 

impacting test results. 

• Testers improperly rounded down back-face 

deformation measurements, which is not 

authorized in the established testing protocols 

and which resulted in two designs passing First 

Article Testing that otherwise would have failed. 

Army officials said rounding is a common 

practice; however, one private test facility that 

rounds told GAO that they round up, not down. 

• Testers used a new instrument to measure backface deformation without adequately certifying 

that the instrument could function correctly and 

in conformance with established testing 

protocols. The impact of this issue on test results 

is uncertain, but it could call into question the 

reliability and accuracy of the measurements. 

• Testers deviated from the established testing 

protocols in one instance by improperly scoring 

a complete penetration as a partial penetration. 

As a result, one design passed First Article 

Testing that would have otherwise failed. 

With respect to internal control issues, the Army did not 

consistently maintain adequate internal controls to 

ensure the integrity and reliability of test data. In one 

example, during ballistic testing, data were lost, and 

testing had to be repeated because an official 

accidentally pressed the delete button and software 

controls were not in place to protect the integrity of test 

data. Army officials acknowledged that before GAO’s 

review they were unaware of the specific internal 

control problems we identified. 

As a result of the deviations from testing protocols that 

GAO observed, four of the five designs that passed First 

Article Testing and were certified by the Army as ready 

for full production would have instead failed testing at 

some point during the process, either during the 

Preliminary Design Model testing or the subsequent First 

Article Test. Thus, the overall reliability and repeatability 

of the test results are uncertain. Although designs passed 

testing that would not have if the testing protocols were 

followed, independent ballistics experts have not 

assessed the impact of the deviations from the testing 

protocols to determine if the effect of the deviations is 

sufficient to call into question the ability of those designs 

to meet requirements. Vendors whose designs passed 

First Article Testing have begun production of plates. 

The Army has ordered 2,500 sets of plates (at two plates 

per set) from these vendors to be used for additional 

ballistics testing and 120,000 sets of plates to be put into 

inventory to address future requirements. However, to 

date, none of these designs have been fielded because, 

according to Army officials, there are adequate numbers 

of armor plates produced under prior contracts already 

in the inventory to meet current requirements. 

GAO’s Recommendations 

To determine what effect, if any, the problems GAO 

observed had on the test data and on the outcomes of 

First Article Testing, the Army should provide for an 

independent ballistics evaluation of the First Article 

Testing results by ballistics and statistical experts 

external to the Department of Defense before any armor 

is fielded to soldiers under this contract solicitation. 

Because DOD did not concur with this recommendation, 

GAO added a matter for congressional consideration to 

this report suggesting that Congress direct DOD to either 

conduct such an independent external review of these 

test results or repeat First Article Testing. 

To better align actual test practices with established 

testing protocols during future body armor testing, the 

Army should assess the need to change its test 

procedures based on the outcome of the independent 

experts’ review and document these and all other key 

decisions made to clarify or change the testing protocols 

during future body armor testing. Although DOD did not 

agree that an independent expert review of test results 

was needed, DOD stated it will address protocol 

discrepancies identified by GAO as it develops 

standardized testing protocols. DOD also agreed to 

document all decisions made to clarify or change testing 

protocols. 

To improve internal controls over the integrity and 

reliability of test data for future testing as well as 

provide for consistent test conditions and comparable 

data among tests, the Army should provide for an 

independent external peer review of Aberdeen Test 

Center’s body armor testing protocols, facilities, and 

instrumentation to ensure that proper internal controls 

and sound management practices are in place. DOD 

generally concurred with this recommendation, but 

stated that it will also include DOD members on the 

review team. 
Page i GAO-10-119 

Contents 

Letter 1

Results in Brief 4 

Background 9 

Army Took Significant Steps during Preliminary Design Model 

Testing to Run a Controlled Test and Maintain Consistency but 

Did Not Consistently Follow Established Testing Protocols and, 

as a Result, Did Not Achieve the Intended Test Objective 16 

During First Article Testing the Army Addressed Some of the 

Problems Identified in Preliminary Design Model Testing, but 

Army Testers Did Not Always Follow Established Testing 

Protocols and Did Not Maintain Some Internal Controls 26 

Conclusions 36 

Recommendations for Executive Action 37 

Matter for Congressional Consideration 38 

Agency Comments and Our Evaluation 38 

Appendix I Scope and Methodology 46 

Appendix II Comments from the Department of Defense 51 

Appendix III GAO Contact and Staff Acknowledgments 104 

Table 

Table 1: Organizations Contacted for Information about Body 

Armor Testing 46 

Figures 

Figure 1: ESAPI Plates as Worn inside Outer Tactical Vest 11 

Figure 2: Timeline of Key Preliminary Design Model Testing and 

First Article Testing Events 13 

Figure 3: Clay Being Calibrated with Pre-Test Drops 21 

Figure 4: Graphic Representation of the Difference between the 

Point of Aim and the Deepest Point 23 

Figure 5: Photographic Representation of the Difference between 

the Point of Aim and the Deepest Point 24 

Warfighter Support 
Figure 6: Tears in Kevlar Backing Material after a Penetration of 

the Plate 33 

Figure 7: Briefing Slide from DOD’s Test Overview (Nov. 14, 2007) 81 

Figure 8: Briefing Slide from DOD’s Test Strategy and Schedule 

(Nov. 14, 2007) 82 

Abbreviations 

ASTM American Society for Testing and Materials 

DOD Department of Defense 

DODIG Department of Defense Inspector General 

ESAPI Enhanced Small Arms Protective Insert 

FSAPV-E Flexible Small Arms Protective Vest-Enhanced 

FSAPV-X Flexible Small Arms Protective Vest-X level 

MDA Milestone Decision Authority 

NIJ National Institute of Justice 

PEO Program Executive Office 

XSAPI Small Arms Protective Insert-X level 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately. 

Page ii GAO-10-119 Warfighter Support
Page 1 GAO-10-119 

 

United States Government Accountability Office

Washington, DC 20548 

 

October 16, 2009 

Congressional Requesters 

Since combat operations began in Afghanistan after September 11, 2001, 

and in Iraq in 2003, U.S. forces have been subjected to frequent and deadly 

attacks from insurgents using improvised explosive devices, mortars, 

rocket launchers, and increasingly lethal ballistic threats. To protect the 

military and civilian personnel of the Department of Defense (DOD) 

against these ballistic threats, since 2003 the U.S. Central Command has 

required that DOD personnel in its area of operations be issued the 

Interceptor Body Armor system, comprising ceramic plates that are 

inserted into pockets of an outer protective vest. Over the past several 

years, the media and Congress have raised concerns about whether the 

Army has adequately evaluated and tested this body armor solution and 

about the transparency of the Army’s body armor testing. Additionally, 

several audits have found problems with the Army’s body armor testing 

programs. For example, in 2009, both the DOD Inspector General and 

Army Audit Agency reported that the Army had not followed established 

test procedures during prior tests of body armor plates.1

In 2007, we reported to the House and Senate Armed Services Committees 

and testified to the House Armed Services Committee about the Army’s 

and Marine Corps’s individual body armor systems.2

 In that report we 

found that the Army relied on several controls to ensure that body armor 

met performance requirements, including testing at National Institute of 

Justice (NIJ)-certified testing facilities. Later, under the Comptroller 

General’s authority, we observed the testing of body armor solutions 

submitted under a May 2007 Army contract solicitation for four categories 

of body armor—specifically, the Enhanced Small Arms Protective Insert 

(ESAPI), the Small Arms Protective Insert-X level (XSAPI), the Flexible 

Small Arms Protective Vest-Enhanced (FSAPV-E), and the Flexible Small 

Arms Protective Vest-X level (FSAPV-X). While present, we observed the 

1

 DOD Inspector General, DOD Testing Requirements for Body Armor, D-2009-047 

(Arlington, Va.: Jan. 29, 2009); and U.S. Army Audit Agency, Body Armor Testing: Program 

Executive Office, Soldier, A-2009-0086-ALA (Alexandria, Va.: Mar. 30, 2009). 

2

 GAO, Defense Logistics: Army and Marine Corps’ Individual Body Armor System 

Issues, GAO-07-662R (Washington, D.C.: Apr. 26, 2007); and Defense Logistics: Army and 

Marine Corps’ Body Armor Requirements, Controls, and Other Issues, GAO-07-911T

(Washington, D.C.: June 6, 2007). 

Warfighter Support 
 

test procedures utilized by Army testers, spoke with Army officials, and 

compared our observations with established testing protocols. The 

purchase descriptions accompanying the contract solicitation 

announcement identified the test procedures to be followed during the 

first round of testing—called Preliminary Design Model testing. 

Traditionally, Army body armor testing had been performed at an NIJcertified facility. However, one manufacturer of flexible small arms 

protective vests, which had failed previous testing conducted for the 

Program Executive Office (PEO) Soldier at an NIJ-certified facility, made 

allegations that the PEO Soldier and the facility had wrongly failed its 

designs. As a result of these allegations, the Army decided instead to 

conduct testing for this current solicitation at the Army’s Aberdeen Test 

Center, which had not performed testing of Interceptor Body Armor for 

PEO Soldier since the 1990s. Additionally, PEO Soldier decided not to 

provide any on-site testing oversight to avoid any appearance of bias 

against that manufacturer.3

Preliminary Design Model testing was conducted by the Army’s Aberdeen 

Test Center from February 2008 though June 2008. The objective of the 

Preliminary Design Model testing was to determine whether candidate 

designs submitted under the solicitation met required ballistics 

performance specifications and would be awarded a production contract.4

In October 2008, on the basis of the Preliminary Design Model testing 

results, the Army awarded four 5-year indefinite delivery/indefinite 

quantity5

 contracts at a total of over $8 billion for the production of the 

ESAPI and the XSAPI—two categories of ceramic plates. No FSAPV-E or 

FSAPV-X solutions passed the testing. The Army decided to repeat testing, 

through First Article Testing, of all of the ESAPI and XSAPI plates that 

 

3

 The designs submitted by that manufacturer also failed Preliminary Design Model testing 

at Aberdeen Test Center. 

4

 The armor plate contracts require First Article Testing, in accordance with the Federal 

Acquisition Regulation, Subpart 9.3, to ensure the contractor can furnish a product that 

conforms to all contract requirements for acceptance. However, the standard Federal 

Acquisition Regulation First Article Testing clause allows the government to waive First 

Article Testing if a design has already been demonstrated to meet the required 

specifications. 

5

 Indefinite delivery/indefinite quantity contracts provide for an indefinite quantity of 

supplies or services during a fixed period of time. These types of contracts are generally 

used when agencies are unable to predetermine, above a specified minimum, the precise 

quantities of supplies or services that the government will require during the contract 

period. 

Page 2 GAO-10-119 Warfighter Support
 

were awarded production contracts to determine whether these plate 

designs indeed met the required ballistics performance specifications 

before fielding the plates. The Aberdeen Test Center conducted First 

Article Testing between November 2008 and December 2008. 

In connection with the Army’s decision to conduct First Article Testing on 

each of the designs that passed Preliminary Design Model testing and that 

were awarded contracts, the House Armed Services Committee and its 

Subcommittee on Air and Land Forces requested that we observe this 

follow-on First Article Testing to assess the degree to which testing was 

conducted according to the established testing protocols. After 

completing our analysis of both the Preliminary Design Model testing and 

First Article Testing of body armor solutions, we are reporting on the 

degree to which the Army followed its established testing protocols during 

(1) Preliminary Design Model testing of the ESAPI, XSAPI, FSAPV-E and 

FSAPV-X and (2) First Article Testing of the ESAPI and XSAPI models that 

were awarded contracts after the Preliminary Design Model testing.6

 We 

did not provide an expert ballistics evaluation of the results of testing. 

To conduct our review, we observed Preliminary Design Model testing and 

First Article Testing at the Army’s Aberdeen Test Center in Aberdeen, 

Maryland. We observed testing from inside the video viewing room and 

firing lanes and also from the conditioning, X-ray, and physical 

characterization rooms. We interviewed and collected information from 

officials from the Aberdeen Test Center, the U.S. Army Evaluation Center, 

PEO Soldier, DOD’s office of the Director of Operational Test and 

Evaluation, and other Army components, as well as from body armor 

manufacturers and private body armor testing facilities. We recorded 

selected test data in a systematic and structured manner using a data 

collection instrument we developed, analyzed selected test data, and 

compared our observations of the way the Aberdeen Test Center 

conducted Preliminary Design Model testing and First Article Testing with 

the testing protocols that Army officials told us served as the testing 

standards at the Aberdeen Test Center. Specifically, these testing 

protocols were: (1) test procedures described in the contract solicitation 

announcement’s purchase descriptions and (2) the Army’s detailed test 

plans and test operations procedures that were to serve as guidance to 

 

6

 We also issued two decisions on bid protests concerning testing under the solicitation. 

Armorworks Enters., LLC.,B-400394, B-400394.2, Sept. 23, 2008, 2008 CPD para. 176 

(protest denied in part and dismissed in part) and Armorworks Enterprises, LLC, B400394.3, Mar. 31, 2009, 2009 CPD para. 79 (protest dismissed). 

Page 3 GAO-10-119 Warfighter Support
 

Aberdeen Test Center testers and that were developed by the Army Test 

and Evaluation Command and approved by PEO Soldier, the office of the 

Director of Operational Test and Evaluation, Army Research Labs, and 

cognizant Army components. In this report, we refer to these testing 

standards that were to be used at Aberdeen Test Center as testing 

protocols. We also reviewed NIJ testing standards because Aberdeen Test 

Center officials told us that, although Aberdeen Test Center is not an NIJcertified testing facility, they have made adjustments to their procedures 

based on those standards and consider them when evaluating Aberdeen 

Test Center testing practices. Complete details on our scope and 

methodology appear in appendix I. 

We conducted this performance audit from July 2007 through October 

2009 in accordance with generally accepted government auditing 

standards. Those standards require that we plan and perform the audit to 

obtain sufficient, appropriate evidence to provide a reasonable basis for 

our findings and conclusions based on our audit objectives. We believe 

that the evidence obtained provides a reasonable basis for our findings 

and conclusions based on our audit objectives. 

During Preliminary Design Model testing the Army took significant steps 

to run a controlled test and maintain consistency throughout the process 

but did not always follow established testing protocols and, as a result, did 

not achieve the intended test objective of determining which designs met 

performance requirements as a basis for awarding contracts. The Army’s 

significant steps to run a controlled test included, for example, the 

consistent documentation of testing procedures using audio, video, and 

other electronic means and extensive efforts to maintain proper 

temperature and humidity in the test lanes. However, we identified several 

instances in which the Aberdeen Test Center deviated from testing 

protocols, including failing to test the ease of insertion of the plates into 

both pockets of the outer protective vest as required by the testing 

protocols; shooting several plates at the wrong velocity or location on the 

plate; and repeating failed clay calibration tests on the same block of 

clay—the latter having the potential to significantly affect test results. In 

the most consequential of the deviations from testing protocols, the Army 

testers incorrectly measured the amount of force absorbed by the designs 

tested by measuring back-face deformation at the point of aim rather than 

Results in Brief 

Page 4 GAO-10-119 Warfighter Support
 

at the deepest point of depression.7 Army testers recognized the error after 

completing about a third of the test and then changed the test plan to call 

for measuring at the point of aim and likewise issued a modification to the 

contract solicitation. At least two8

 of the eight designs that passed 

Preliminary Design Model testing and were awarded contracts would have 

failed if measurements had been made to the deepest point of depression. 

The deviations from the testing protocols were the result of Aberdeen Test 

Center’s incorrectly interpreting the testing protocols. Although Aberdeen 

Test Center officials told us that any deviations from the testing protocols 

required approval from PEO Soldier, the office of the Director of 

Operational Test and Evaluation, and other activities, in all these cases the 

Aberdeen Test Center procedures implemented were not reviewed or 

approved by officials from PEO Soldier, the Director of Operational Test 

and Evaluation, and other activities responsible for approving the testing 

protocols. Furthermore, PEO Soldier representatives were not present at 

Aberdeen Test Center during most of the testing, an absence that may 

have contributed to the fact that these deviations were not identified 

earlier during testing. PEO Soldier officials told us that they were not 

present at testing in order to ensure the independence of the testing 

facility, but they later acknowledged that they should have been more 

involved in that testing and would be more involved in future testing.9

After concerns were raised regarding the testing conducted at Aberdeen 

Test Center under the May 2007 solicitation, the decision was made to not 

field any of the of the ESAPI and XSAPI plates awarded contracts as a 

 

7

 In addition to stopping bullets, body armor absorbs and dissipates the force of the impact 

of these bullets. The amount of force absorbed is determined by measuring the depth of the 

depression—called back-face deformation— caused to the clay placed behind the body 

armor during ballistic testing: the lower the back-face deformation, the more force that is 

absorbed by the body armor. See figures 4 and 5 for examples of back-face deformation. 

8

 After testers realized they were incorrectly measuring back-face deformation at the point 

of aim rather than at the deepest point, testers began to measure to both points but used 

the point-of-aim measure as the official measure, which according to Army officials was 

necessary to maintain consistency throughout testing and to not disadvantage any vendors. 

These two designs would have failed if the deepest point measure recorded had been used 

as the official measure. Because the deepest point was not measured during the first third 

of testing, additional designs could have improperly passed. 

9

Prior to Preliminary Design Model testing a body armor manufacturer whose design failed 

a prior test made public allegations that PEO Soldier had an unfair bias against its design. 

In an attempt to remove any appearance of bias against that manufacturer, PEO Soldier 

made a decision to not provide an on-site presence during Preliminary Design Model 

testing. 

Page 5 GAO-10-119 Warfighter Support
 

result of Preliminary Design Model testing until after First Article Testing 

was conducted. 

During First Article Testing, while the Army addressed some of the 

problems identified during Preliminary Design Model testing, we observed 

instances in which Army testers did not follow the established testing 

protocols and did not maintain internal controls over the integrity and 

reliability of test data, raising questions as to whether the Army met its 

First Article Testing objective of determining whether each of the 

contracted armor plate designs met performance requirements. The Army 

resolved the problems with shot location and velocity and with the easeof-insertion test. Also, Army technical experts from PEO Soldier who 

served on the Integrated Product Team were charged with testing 

oversight and maintained an on-site presence in the test lanes. However, 

Army testers continued to deviate from established testing protocols with 

respect to clay calibration and back-face deformation measurement as 

follows: 

• For the clay calibration test, the Army testers followed an orally 

agreed-upon set of procedures that deviated from the established 

testing protocols. Specifically, Army testers used clay in testing that 

had failed the initial clay calibration test. The use of clay that has failed 

the calibration test could significantly impact test results. This was 

especially significant on a day with high failure rates when we 

observed clay being exposed to constant heavy, cold rain. The 

established testing protocols require the use of a specific type of nonhardening oil-based clay. Officials from the Army, private NIJ-certified 

ballistics laboratories, and the clay manufacturer told us that water 

exposure may contaminate the clay by changing its chemical bonding 

characteristics as well as by causing rapid and uneven cooling, which 

could affect test results. Although Army Test and Evaluation 

Command officials said covering the clay was not required and its 

exposure to water would not impact testing, these officials were 

unable to provide any documentation to support their position, raising 

concerns that exposure to rain may have impacted the testing results. 

• Army testers rounded down back-face deformation measurements, 

which is not authorized in established testing protocols or consistent 

with their testing practice during Preliminary Design Model testing. 

Army officials said that rounding is a common industry practice and 

that they should have also rounded Preliminary Design Model testing 

results. While we did not validate this assertion, officials we spoke 

with from one private industry ballistics testing facility said that their 

practice was to always round results up, not down, which has the same 

Page 6 GAO-10-119 Warfighter Support
 

effect as not rounding. As a result of the rounding, two designs passed 

First Article Testing that would have failed if the measurements had 

not been rounded. 

• The Army used a laser scanner as a new method to measure back-face 

deformation without adequately certifying that the scanner could 

function: (1) in its operational environment, (2) at the required 

accuracy, (3) in conjunction with its software package, or (4) without 

overstating deformation measurements. Army officials told us they are 

unable to estimate the accuracy of the laser scanner used in testing, 

raising concerns regarding the reliability of back-face deformation 

results. Aberdeen Test Center officials said they initially decided to use 

the laser because they did not believe it was possible to measure backface deformations to the required level of accuracy using the digital 

caliper. However, officials from PEO Soldier and private NIJ-certified 

laboratories have told us that they believe the digital caliper method is 

capable of making these measurements and that the back-face 

deformation measurements in the testing protocols were developed 

using a digital caliper.10 While it is uncertain what impact this issue had 

on the test results, the reliability and accuracy of the measurements 

may be called into question. 

During First Article Testing, Army testers deviated from the established 

testing protocols by improperly scoring a complete penetration as a partial 

penetration. Army testers said they used a method to evaluate the 

penetration results that was discussed internally before First Article 

Testing but that was not described in the testing protocols or otherwise 

documented. As a result of this incident, one design passed First Article 

Testing that would have otherwise failed.11 With respect to internal control 

issues, the Army did not consistently maintain adequate internal controls 

to ensure the integrity and reliability of test data. In one example, during 

ballistic testing, data were lost and testing had to be repeated because an 

 

10 Army and private laboratory officials told us that, on the basis of the limited data they 

had previously collected, they were concerned that the laser scanner may overstate backface deformation measurements by about 2 millimeters as compared with the 

measurements obtained by using the digital caliper. We did not independently verify or 

validate the data provided by these officials. Since standards are based on measurements 

obtained with a digital caliper, results obtained using the laser scanner may be 

inconsistent/different than those obtained using the digital caliper. 

11 This design is also one of the ones that would have failed Preliminary Design Model 

testing had back-face deformations been measured to the deepest point as required by the 

testing protocols. 

Page 7 GAO-10-119 Warfighter Support
 

official accidentally hit the delete button and software controls were not in 

place to protect the integrity of test data. Federal internal control 

standards require that federal agencies maintain effective controls over 

information processing to help ensure completeness, accuracy, 

authorization, and validity of all transactions. Army officials 

acknowledged that before our review they were unaware of the specific 

internal control problems we identified. As a result of the deviations from 

testing protocols that we observed, three of the five designs that passed 

First Article Testing would not have passed under the existing testing 

protocols and one of the remaining two designs that passed would have 

failed Preliminary Design Model testing if those testing protocols had been 

fully followed. Thus, four of the five designs that passed First Article 

Testing and were certified by the Army as ready for full production would 

have instead failed testing at some point during the process, either during 

the initial Preliminary Design Model testing or the subsequent First Article 

testing, if the established testing protocols had been fully followed. As a 

result, the overall reliability and repeatability of the test results are 

uncertain. Although designs passed testing that would not have if the 

testing protocols had been followed, ballistics experts have not assessed 

the impact of the deviations from the testing protocols to determine if 

their effect is sufficient to call into question the ability of those designs to 

meet mission requirements. The Army has ordered 2,500 sets of plates (at 

two plates per set) from those vendors whose designs passed First Article 

Testing to be used for additional ballistics testing and 120,000 sets of 

plates to be put into inventory to address future requirements. However, to 

date, none of these designs have been fielded because, according to Army 

officials, there are adequate quantities of armor plates produced under 

prior contracts already in the inventory to meet current requirements. 

To help ensure that test results are reliable, we are recommending that 

before any body armor plates are fielded to soldiers under the May 2007 

solicitation, an assessment of the First Article Testing test data be 

performed by independent experts to determine whether the issues we 

identified had a significant effect on the test results. We are also making 

several recommendations intended to improve the transparency of testing 

by fully documenting any revised test practices so that their alignment 

with testing protocols is clear. Finally, we are making several 

recommendations to address the specific inconsistencies in test conditions 

we observed and to improve internal controls. 

In written comments on a draft of this report, DOD generally concurred 

with our finding that there were deviations from the established testing 

protocols during Preliminary Design Model testing and First Article 

Page 8 GAO-10-119 Warfighter Support
 

Testing and with our recommendations to fully document revised test 

practices in the testing protocols and to improve internal controls over 

testing. However, DOD did not concur with our recommendation that an 

independent expert assessment of First Article Testing data be performed 

before any body armor plates are fielded to soldiers under contracts 

awarded under this solicitation. In the comments, DOD wrote that the 

deviations we identified have no significant impact on the test results and 

the subsequent contracting actions taken by the Army based on these test 

results. We disagree with DOD’s assertions in this regard and continue to 

state that such an independent assessment is necessary to ensure that the 

body armor plates meet all protection requirements. We were unable to 

determine the full effects of deviations we observed as they relate to the 

quality of the armor designs and believe that such a determination should 

only be made based on a thorough assessment of the testing data by 

independent ballistics-testing experts. In light of such uncertainty and the 

critical need for confidence in the equipment by the soldiers, the Army 

would be taking unacceptable risk if it were to field these armor designs 

without taking additional steps to gain the needed confidence that the 

armor will perform as required. Consequently, we have added a matter for 

congressional consideration to our report suggesting that Congress 

consider directing DOD to either require that an independent external 

review of these body armor test results be conducted or require that DOD 

officially amend its testing protocols to reflect any revised test procedures 

and repeat First Article Testing to ensure that designs are properly tested. 

DOD’s written comments are reprinted in appendix II. 

Background 

Army Solicitation for Body 

Armor 

In May 2007, the Army issued a solicitation for body armor designs to 

replenish stocks and to protect against future threats by developing the 

next generation (X level) of protection. According to Army officials, the 

solicitation would result in contracts that the Army would use for 

sustainment of protective plate stocks for troops in Iraq and Afghanistan. 

The indefinite delivery/indefinite quantity contracts require the Army to 

purchase a minimum of 500 sets per design and allow for a maximum 

purchase of 1.2 million sets over the 5-year period.12

 

12 A set of protective plates comprises two plates—one front and one back plate. 

Page 9 GAO-10-119 Warfighter Support
 

The Army’s solicitation, which closed in February 2008, called for 

preliminary design models in four categories of body armor protective 

plates: 

• Enhanced Small Arms Protective Insert (ESAPI)—plates designed to 

same protection specifications as those currently fielded and to fit into 

currently fielded Outer Tactical Vests. 

• Flexible Small Arms Protective Vest-Enhanced (FSAPV-E)—flexible 

armor system designed to same protection specifications as armor 

currently fielded. 

• Small Arms Protective Insert-X level (XSAPI)—next-generation plates 

designed to defeat higher level threat. 

• Flexible Small Arms Protective Vest-X level (FSAPV-X)—flexible 

armor system designed to defeat higher level threat. 

In figure 1, we show the ESAPI plates inside the Outer Tactical Vest. 

Page 10 GAO-10-119 Warfighter Support
 

Figure 1: ESAPI Plates as Worn inside Outer Tactical Vest 

Source: Army.

Between May of 2007 and February of 2008 the Army established testing 

protocols, closed the solicitation, and provided separate live-fire 

demonstrations of the testing process to vendors who submitted items for 

testing and to government officials overseeing the testing. Preliminary 

Design Model testing was conducted at Aberdeen Test Center between 

February 2008 and June 200813 at an estimated cost of $3 million. 

 

13 After it was discovered that back-face deformation was being measured incorrectly, 

Preliminary Design Model testing was halted for 2 weeks so that Army officials could 

consult with senior Army leadership on how to best resolve the issue. 

Page 11 GAO-10-119 Warfighter Support
 

Additionally, over $6 million was spent on infrastructure and equipment 

improvements at Aberdeen Test Center to support future light armor test 

range requirements, including body armor testing. 

First Article Testing was then conducted at Aberdeen Test Center from 

November 10, 2008, to December 17, 2008,14 on the three ESAPI and five 

XSAPI designs that had passed Preliminary Design Model testing.15 First 

Article Testing is performed in accordance with the Federal Acquisition 

Regulation to ensure that the contractor can furnish a product that 

conforms to all contract requirements for acceptance. First Article Testing 

determines whether the proposed product design conforms to contract 

requirements before or in the initial stage of production. During First 

Article Testing, the proposed design is evaluated to determine the 

probability of consistently demonstrating satisfactory performance and 

the ability to meet or exceed evaluation criteria specified in the purchase 

description. Successful First Article Testing certifies a specific design 

configuration and the manufacturing process used to produce the test 

articles. Failure of First Article Testing requires the contractor to examine 

the specific design configuration to determine the improvements needed 

to correct the performance of subsequent designs. 

Testing of the body armor currently fielded by the Army was conducted by 

private NIJ-certified testing facilities under the supervision of PEO Soldier. 

According to Army officials, not a single death can be attributed to this 

armor’s failing to provide the required level of protection for which it was 

designed. However, according to Army officials, one of the body armor 

manufacturers that had failed body armor testing in the past did not agree 

with the results of the testing and alleged that the testers tested that armor 

to higher–than–required standards. The manufacturer alleged a bias 

against its design and argued that its design was superior to currently 

fielded armor. As a result of these allegations and in response to 

congressional interest, after the June 2007 House Armed Services 

Committee hearing, the Army accelerated completion of the light armor 

ranges to rebuild small arms ballistic testing capabilities at Aberdeen Test 

 

14 Testing was halted for other high-priority tests involving 2,000 plates from Iraq that were 

identified as potentially cracked by nondestructive testing performed by the Army. 

15 From November 14 to November 19 First Article Testing was halted to allow for higherpriority testing to be conducted. Nearly all the ballistic testing was conducted between 

November 10 and December 4. The testing conducted prior to November 10 was mainly 

physical characterization of the plates, and the testing after December 4 was limited to the 

retesting of a single plate that the Army had identified as being tested incorrectly. 

Page 12 GAO-10-119 Warfighter Support
 

Center and to conduct testing under the May 2007 body armor solicitation 

there, without officials from PEO Soldier supervising the testing. 

Furthermore, the decision was made to allow Aberdeen Test Center, 

which is not an NIJ-certified facility, to be allowed to conduct the repeated 

First Article Testing. In February 2009 the Army directed that all future 

body armor testing be performed at Aberdeen Test Center. According to 

Army officials, as of this date, none of the body armor procured under the 

May 2007 solicitation had been fielded. 

Given the significant congressional interest in the testing for this 

solicitation and that these were the first small arms ballistic tests 

conducted at Aberdeen Test Center in years, multiple defense 

organizations were involved in the Preliminary Design Model testing. 

These entities include the Aberdeen Test Center, which conducted the 

testing; PEO Soldier, which provided the technical subject-matter experts; 

and DOD’s office of the Director of Operational Test and Evaluation, 

which combined to form the Integrated Product The Integrated Product 

Team was responsible for developing and approving the test plans used for 

the Preliminary Design Model testing and First Article Testing. Figure 2 

shows a timeline of key Preliminary Design Model testing and First Article 

Testing events. 

Figure 2: Timeline of Key Preliminary Design Model Testing and First Article Testing Events 

5/25/07

Solicitation

issued

9/11/07

Detailed test 

plans signed

2/7/08 

Solicitation 

closed

6/6/07

House Armed 

Services Committee 

Hearing

6/14/07

Purchase 

descriptions signed

2/20-21/08

Live fire 

demonstration 

for vendors

2/08

Start of 

Preliminary Design 

Model testing

3/26/08

Testing halted due 

to back face 

deformation issue

2007 2008

4/10/08 

Testing 

resumed

6/08

End of Preliminary 

Design Model 

testing

8/08

Source 

selections

made

11/14/08

First Article 

Testing 

halted

11/6/08 

Start of First 

Article Testing

11/19/08

First Article 

Testing 

resumed

12/17/08

End of 

First 

Article 

Testing

Source: GAO observation and Army data.

Body Armor Test 

Procedures 

The test procedures to be followed for Preliminary Design Model testing 

were established and identified in the purchase descriptions 

accompanying the solicitation announcement and in the Army’s detailed 

Page 13 GAO-10-119 Warfighter Support
 

test plans (for each of the four design categories), which served as 

guidance to Army testers and were developed by the Army Test and 

Evaluation Command and approved by PEO-Soldier, DOD’s office of the 

Director of Operational Test and Evaluation, and others. Originally, PEO 

Soldier required that testing be conducted at an NIJ-certified facility. 

Subsequently, the decision was made to conduct testing at Aberdeen Test 

Center, which is not NIJ-certified.16

The test procedures for both Preliminary Design Model testing and First 

Article Testing included both (1) physical characterization steps 

performed on each armor design to ensure they met required 

specifications, which included measuring weight, thickness, curvature, 

and size and (2) ballistic testing performed on each design. Ballistics 

testing for this solicitation included the following subtests: (1) ambient 

testing to determine whether the designs can defeat the multiple threats 

assigned in the respective solicitation’s purchase descriptions 100 percent 

of the time; (2) environmental testing of the designs to determine whether 

they can defeat each threat 100 percent of the time after being exposed to 

nine different environmental conditions; and (3) testing, called V50 testing, 

to determine whether designs can defeat each threat at velocities 

significantly higher than those present or expected in Iraq or Afghanistan 

at least 50 percent of the time. 

Ambient and environmental testing seek to determine whether designs can 

defeat each threat 100 percent of the time by both prohibiting the bullet 

from penetrating through the plate and by prohibiting the bullet from 

causing too deep of an indentation in the clay backing behind the plate. 

Preventing a penetration is important because it prevents a bullet from 

entering the body of the soldier. Preventing a deep indentation in the clay 

(called “back-face deformation”) is important because the depth of the 

indentation indicates the amount of blunt force trauma to the soldier. 

Back-face deformation deeper than 43 millimeters puts the soldier at 

higher risk of internal injury and death. 

The major steps taken in conducting a ballistic subtest include: 

 

16 Even though Aberdeen Test Center is not an NIJ-certified facility, Aberdeen Test Center 

officials said they are actively keeping abreast of NIJ standards, have made adjustments to 

their procedures based on those standards, and consider those standards when evaluating 

their own testing practices. Although there remains an active discussion in the Army 

testing community as to whether Aberdeen Test Center should pursue certification, 

Aberdeen Test Center currently has no plans to pursue NIJ certification. 

Page 14 GAO-10-119 Warfighter Support
 

1. For environmental subtests, the plate is exposed to the environmental 

condition tested (e.g., impact test, fluid soaks, temperature extremes, 

etc.). 

2. The clay to be used to back the plate is formed into a mold and is 

placed in a conditioning chamber for at least 3 hours. 

3. The test plate is placed inside of a shoot pack. 

4. The clay is taken out of the conditioning chamber. It is then tested to 

determine if it is suitable for use17 and, if so, is placed behind the test 

plate. 

5. The armor and clay are then mounted to a platform and shot. 

6. If the shot was fired within required specifications,18 the plate is 

examined to determine if there is a complete or partial penetration, 

and the back-face deformation is measured. 

7. The penetration result and back-face deformation are scored as a 

pass,19 a limited failure,20 or a catastrophic failure.21 If the test is not 

conducted according to the testing protocols, it is scored as a no-test. 

 

17 Testing protocols require that clay be calibrated by dropping a 1-kilogram cylindrical 

weight on the clay in three locations. If all drops cause indentions between 22 and 28 

millimeters, the clay is acceptable for use. 

18 Specifications include factors such as firing the shot at proper velocity, in conditions 

with correct humidity and temperature, and using properly conditioned clay. 

19 A “pass” is any plate that is not a limited or catastrophic failure. 

20 A “limited failure” for threats A, B, C, F, and Y is either (1) complete penetration of hard 

armor (the plate), but partial penetration of the soft armor (shoot pack) on any shot or (2) 

a back-face deformation greater than 43 millimeters but less than 48 millimeters. A limited 

failure on threats D and X is either (1) a complete penetration of hard armor (plate), but a 

partial penetration of the soft armor (shoot pack) on the first shot or (2) a complete 

penetration of both the hard armor (plate) and the soft armor (shoot pack) on a second 

shot or (3) back-face deformation on the first shot greater than 43 millimeters, but less than 

48 millimeters or (4) back-face deformation greater than 43 millimeters on a second shot. 

21 A “catastrophic failure” for threats A, B, C, F, and Y is either (1) a complete penetration 

of the hard armor (plate) and the soft armor (shoot pack) on any shot or (2) a back-face 

deformation on any shot greater than or equal to 48 millimeters. A catastrophic failure for 

threats D and X is either (1) a complete penetration of both the hard armor (plate) and soft 

armor (shoot pack) on a first shot or (2) a first shot back-face deformation greater or equal 

to than 48 millimeters. 

Page 15 GAO-10-119 Warfighter Support
 

Army Took Significant 

Steps during 

Preliminary Design 

Model Testing to Run 

a Controlled Test and 

Maintain Consistency 

but Did Not 

Consistently Follow 

Established Testing 

Protocols and, as a 

Result, Did Not 

Achieve the Intended 

Test Objective 

Army Took Significant 

Steps to Run a Controlled 

Test and Maintain 

Consistency 

Following are significant steps the Army took to run a controlled test and 

maintain consistency throughout Preliminary Design Model testing: 

• The Army developed testing protocols for the hard-plate (ESAPI and 

XSAPI) and flexible-armor (FSAPV-E and FSAPV-X) preliminary 

design model categories in 2007. These testing protocols were 

specified in newly created purchase descriptions, detailed test plans, 

and other documents. For each of the four preliminary design model 

categories, the Army developed new purchase descriptions to cover 

both hard-plate and flexible designs. These purchase descriptions 

listed the detailed requirements for each category of body armor in the 

solicitation issued by the Army. Based on these purchase descriptions, 

the Army developed detailed test plans for each of the four categories 

of body armor. These detailed test plans provided additional details on 

how to conduct testing and provided Army testers with the 

requirements that each design needed to pass. After these testing 

protocols were developed, Army testers then conducted a pilot test in 

which they practiced test activities in preparation for Preliminary 

Design Model testing, to help them better learn and understand the 

testing protocols. 

Page 16 GAO-10-119 Warfighter Support
 

• The Army consistently documented many testing activities by using 

audio, video, and other electronic means. The use of cameras and 

microphones to provide 24-hour video and audio surveillance of all of 

the major Preliminary Design Model testing activities provided 

additional transparency into many testing methods used and allowed 

for enhanced oversight by Army management, who are unable to 

directly observe the lanes on a regular basis but who wished to view 

select portions of the testing. The Army utilized an electronic database 

to maintain a comprehensive set of documentation for all testing 

activities. This electronic database included a series of data reports 

and pictures for each design including: physical characterization 

records, X-ray pictures, pre- and post-shot pictures, ballistics testing 

results, and details on the condition of the clay backing used for the 

testing of those plates. 

The Army took a number of additional actions to promote a consistent and 

unbiased test. For example, the Army disguised vendor identity for each 

type of solution by identifying vendors with random numbers to create a 

blind test. The Army further reduced potential testing variance by shooting 

subtests in the same shooting lane. The Army also made a good faith effort 

to use consistent and controlled procedures to measure the weight, 

thickness, and curvature of the plates. Additionally, the Army made 

extensive efforts to consistently measure and maintain room temperature 

and humidity within desired ranges. We also observed that projectile yaw22

was consistently monitored and maintained. We also found no deviations 

in the monitoring of velocities for each shot and the re-testing of plates in 

cases where velocities were not within the required specifications. We 

observed no instances of specific bias against any design, nor did we 

observe any instances in which a particular vendor was singled out for 

advantage or disadvantage. 

Army Did Not Consistently 

Follow All Testing 

Protocols 

We identified several instances in which the Aberdeen Test Center did not 

follow established testing protocols. For example, during V50 testing, 

testers failed to properly adjust shot velocities. V50 testing is conducted to 

discern the velocity at which 50 percent of the shots of a particular threat 

would penetrate each of the body armor designs. The testing protocols 

require that after every shot that is defeated by the body armor the 

 

22 The ”yaw card” is a piece of paper placed in the intended path of the ballistic and is 

meant to measure the amount of yaw, or wobble, of the ballistic as it travels through the 

air. 

Page 17 GAO-10-119 Warfighter Support
 

velocity of the next shot be increased. Whenever a shot penetrates the 

armor, the velocity should be decreased for the next shot. This increasing 

and decreasing of the velocities is supposed to be repeated until testers 

determine the velocity at which 50 percent of the shots will penetrate. In 

cases in which the armor far exceeds the V50 requirements and is able to 

defeat the threat for the first six shots, the testing may be halted without 

discerning the V50 for the plate, and the plate is ruled as passing the 

requirements. During Preliminary Design Model testing, in cases in which 

plates defeated the first three shots, Army testers failed to increase shot 

velocities, but rather continued to shoot at approximately the same 

velocity or lower for shots four, five, and six in order to obtain six partial 

penetrations and conclude the test early. Army officials told us that this 

deviation was implemented by Aberdeen Test Center to conserve plates 

for other tests that needed repeating as a result of no-test events, 

according to Aberdeen Test Center officials—but was a practice not 

described in the protocols. Army officials told us that this practice had no 

effect on which designs passed or failed; however, this practice made it 

impossible to discern the true V50s for these designs and was a deviation 

from the testing protocols that require testers to increase velocities for 

shots after the armor defeats the threat. 

In another example, Aberdeen Test Center testers did not consistently 

follow testing protocols in the ease-of-insertion test. According to the 

testing protocols, one barehanded person shall demonstrate insertion and 

removal of the ESAPI/XSAPI plates in the Outer Tactical Vest23 pockets 

without tools or special aids. Rather than testing the insertion of both the 

front and the rear pockets as required, testers only tested the ability to 

insert into the front pocket. Testing officials told us that they did not test 

the ability to insert the plates into the rear pocket because they were 

unable to reach the rear pocket while wearing the Outer Tactical Vest. The 

cause for this deviation is that the testers misinterpreted the testing 

protocols, as there is no requirement in the established testing protocols to 

wear the Outer Tactical Vest when testing the ability to insert the plates in 

the rear pocket of the Outer Tactical Vest. Officials from PEO Soldier told 

us that, had they been present to observe this deviation during testing, 

they would have informed testers that the insertion test does not require 

that the Outer Tactical Vest be worn, which would have resulted in testers 

conducting the insertion test as required. According to Aberdeen Test 

Center officials, this violation of the testing protocols had no impact on 

 

23 Testing protocols use the acronym OTV for Outer Tactical Vest. 

Page 18 GAO-10-119 Warfighter Support
 

test results. While we did not independently verify this assertion, 

Aberdeen Test Center officials told us that the precise physical 

characterization measurements of the plate’s width and dimensions are, 

alone, sufficient to ensure the plate will fit. 

In addition, testers deviated from the testing protocols by placing shots at 

the wrong location on the plate. The testing protocols require that the 

second shot for one of the environmental sub-tests, called the impact test, 

be taken approximately 1.5 inches from the edge of the armor. However, 

testers mistakenly aimed closer to the edge of the armor for some of the 

designs tested. Army officials said that the testing protocols were unclear 

for this test because they did not prescribe a specific hit zone (e.g., 1.25 – 

1.75 inches), but rather relied upon testers’ judgment to discern the 

meaning of the word “approximately.” One of the PEO Soldier technical 

advisors on the Integrated Product Team told us he was contacted by the 

Test Director after the plates had been shot and asked about the shot 

location. He told us that he informed the Test Director that the plates had 

been shot in the wrong location. The PEO Soldier Technical advisor told 

us that, had he been asked about the shot location before the testing was 

conducted, he could have instructed testers on the correct location at 

which to shoot. For 17 of the 47 total designs that we observed and 

measured,24 testers marked target zones that were less than the required 

1.5 inches from the plate’s edge, ranging from .75 inches to 1.25 inches 

from the edge. Because 1.5 inches was outside of the marked aim area for 

these plates, we concluded that testers were not aiming for 1.5 inches. For 

the remaining 30 designs tested that we observed and measured, testers 

used a range that included 1.5 inches from the edge (for example, aiming 

for 1 to 1.5 inches). It is not clear what, if any, effect this deviation had on 

the overall test results. While no design failed Preliminary Design Model 

testing due to the results of this subtest, there is no way to determine if a 

passing design would have instead failed if the testing protocol had been 

correctly followed. However, all designs that passed this testing were later 

subject to First Article Testing, where these tests were repeated in full 

using the correct shot locations.25

 

24 Nine test items could not be measured either because they were marked in a way that 

could not be measured or because the impact of the bullet deformed the plate too severely. 

25 One of the designs that passed Preliminary Design Model testing later failed First Article 

Testing because of its results during the impact test. Thus, it is possible that this design 

may have passed Preliminary Design Model testing due to shooting the plate at the wrong 

location, resulting in additional testing costs. 

Page 19 GAO-10-119 Warfighter Support
 

Of potentially greater consequence to the final test results is our 

observation of deviations from testing protocols regarding the clay 

calibration tests. According to testing protocols, the calibration of the clay 

backing material was supposed to be accomplished through a series of 

pre-test drops.26 The depths of the pre-test drops should have been 

between 22 and 28 millimeters. Aberdeen Test Center officials told us that 

during Preliminary Design Model testing they did not follow a consistent 

system to determine if the clay was conditioned correctly. According to 

Aberdeen Test Center officials, in cases in which pre-test drops were 

outside the 22- to 28-millimeter range, testers would sometimes repeat one 

or all of the drops until the results were within range—thus resulting in the 

use of clay backing materials that should have been deemed unacceptable 

for use. These inconsistencies occurred because Army testers in each test 

lane made their own, sometimes incorrect, interpretation of the testing 

protocols. Members of the Integrated Product Team expressed concerns 

about these inconsistencies after they found out how calibrations were 

being conducted. In our conversations with Army and private body armor 

testing officials, consistent treatment and testing of clay was identified as 

critical to ensure consistent, accurate testing. According to those officials 

if the clay is not conditioned correctly it will impact the test results. Given 

that clay was used during Preliminary Design Model testing that failed the 

clay calibration tests, it is possible that some shots may have been taken 

under test conditions different than those stated in the testing protocols, 

potentially impacting test results. Figure 3 shows an Army tester 

calibrating the clay with pre-test drops. 

 

26 These drops comprise dropping a cylindrical metal apparatus onto the clay backing 

material and measuring the amount of depression caused by the drop. 

Page 20 GAO-10-119 Warfighter Support
 

Figure 3: Clay Being Calibrated with Pre-Test Drops 

Source: Army.

The most consequential of the deviations from testing protocols we 

observed involved the measurement of back-face deformation, which did 

affect final test results. According to testing protocol, back-face 

Page 21 GAO-10-119 Warfighter Support
 

deformation is to be measured at the deepest point of the depression in 

the clay backing. This measure indicates the most force that the armor will

allow to be exerted on an individual struck by a bullet. According to A

officials, the deeper the back-face deformation measured in the cla

backing, the higher the risk of internal injury or death. During 

approximately the first one-third of testing, however, Army testers 

incorrectly measured deformation at the point of aim, rather than at the 

deepest point of depression. This is significant because, in many instances

measuring back-face deformation at the point of aim results in measuri

at a point upon which less ballistic force is exerted, resulting in lower 

back-face deformation measurements and overestimating the effectiveness 

of the armor. The Army’s subject matter experts on the Integrated Product 

Team were not on the test lanes during testing and thus not made aware of 

rmy 

y 

, 

ng 

the error until approximately one-third of the testing had been completed. 

eeks 

ut 

 call 

two 

nd 

strate the 

difference between the point of aim and the deepest point. 

When members of the Integrated Product Team overseeing the testing 

were made aware of this error, the Integrated Product Team decided to 

begin measuring at the deepest point of depression. When senior Army 

leadership was made aware of this error, testing was halted for 2 w

while Army leadership considered the situation. Army leadership 

developed many courses of action, including restarting the entire 

Preliminary Design Model testing with new armor plate submissions, b

ultimately decided to continue measuring and scoring officially at the 

point of aim, since this would not disadvantage any vendors. The Army 

then changed the test plans and modified the contract solicitation to

for measuring at the point of aim. The Army also decided to collect 

deepest point of depression measurements for all shots from that point 

forward, but only as a government reference. During the second two-thirds 

of testing, we observed significant differences between the measurements 

taken at the point of aim and those taken at the deepest point, as much as 

a 10-millimeter difference between measurements. As a result, at least 

of the eight designs that passed Preliminary Design Model testing a

were awarded contracts would have failed if the deepest point of 

depression measurement had been used. Figures 4 and 5 illu

Page 22 GAO-10-119 Warfighter Support
 

Figure 4: Graphic Representation of the Difference between the Point of Aim and the Deepest Point 

Guide plane

Point of aim 

depression

Deepest point 

of depression

Clay backing

Armor plate

Back-face 

deformation

Point of aim Aim laser

Shooting barrel

Line of fire

Source: GAO analysis.

Page 23 GAO-10-119 Warfighter Support
 

Figure 5: Photographic Representation of the Difference between the Point of Aim 

and the Deepest Point 

Source: Army.

Point of aim (laser)

Deepest point

Army Decided to Repeat 

Testing in First Article 

Testing in an Attempt to 

Address Back-Face 

Deformation Measurement 

Problem Identified during 

Preliminary Design Model 

Testing 

Before Preliminary Design Model testing began at Aberdeen Test Center, 

officials told us that Preliminary Design Model testing was specifically 

designed to meet all the requirements of First Article Testing. However, 

Preliminary Design Model testing failed to meet its goal of determining 

which designs met requirements, because of the deviations from 

established testing protocols described earlier in this report. Those 

deviations were not reviewed or approved by officials from PEO Soldier, 

the office of the Director of Operational Test and Evaluation, or by the 

Integrated Product Team charged with overseeing the test. PEO Soldier 

officials told us that the reason for a lack of PEO Soldier on-site presence 

during this testing was because of a deliberate decision made by PEO 

Soldier management to be as removed from the testing process as possible 

in order to maximize the independence of the Aberdeen Test Center. PEO 

Soldier officials told us that it was important to demonstrate the 

independence of the Aberdeen Test Center to quash allegations of bias 

made by a vendor whose design had failed prior testing and that this 

choice may have contributed to some of the deviations not being identified 

Page 24 GAO-10-119 Warfighter Support
 

by the Army earlier during testing. After the conclusion of Preliminary 

Design Model testing, PEO Soldier officials told us that they should have 

been more involved in the testing and that they would be more involved in 

future testing. 

After the completion of Preliminary Design Model testing, the 

Commanding General of PEO Soldier said that, as the Milestone Decision 

Authority27 for the program, he elected to repeat the testing conducted 

during Preliminary Design Model testing through First Article Testing 

before any body armor was fielded based on the solicitation. According to 

PEO Soldier officials, at the beginning of Preliminary Design Model 

testing, there was no intention or plan to conduct First Article Testing 

following contract awards given that the Preliminary Design Model testing 

was to follow the First Article Testing protocol. However, because of the 

fact that back-face deformation was not measured to the deepest point, 

PEO-Soldier and Army Test and Evaluation and Command acknowledged 

that there was no longer an option of forgoing First Article Testing. PEO 

Soldier also expressed concerns that Aberdeen Test Center test facilities 

have not yet demonstrated that they are able to test to the same level as 

NIJ-certified facilities. However, officials from Army Test and Evaluation 

Command and DOD’s office of the Director of Operational Test and 

Evaluation asserted that Aberdeen Test Center was just as capable as NIJcertified laboratories, and Army leadership eventually decided that First 

Article Testing would be performed at Aberdeen. 

 

27 The Milestone Decision Authority (MDA) is the designated individual with overall 

responsibility for a program. According to DOD Directive 5000.01, the MDA shall have the 

authority to approve entry of an acquisition program into the next phase of the acquisition 

process and shall be accountable for cost, schedule, and performance reporting to higher 

authority, including congressional reporting. 

Page 25 GAO-10-119 Warfighter Support
 

During First Article 

Testing the Army 

Addressed Some of 

the Problems 

Identified in 

Preliminary Design 

Model Testing, but 

Army Testers Did Not 

Always Follow 

Established Testing 

Protocols and Did Not 

Maintain Some 

Internal Controls 

During First Article 

Testing, the Army 

Addressed Some of the 

Problems Identified during 

Preliminary Design Model 

Testing 

PEO Soldier maintained an on-site presence in the test lanes and the Army 

technical experts on the Integrated Product Team charged with testing 

oversight resolved the following problems during First Article Testing: 

• The Army adjusted its testing protocols to clarify the required shot 

location for the impact test, and Army testers correctly placed these 

shots as required by the protocols. 

• After the first few days of First Article Testing, in accordance with 

testing protocols, Army testers began to increase the velocity after 

every shot defeated by the armor required during V50 testing. 

• As required by the testing protocols, Army testers conducted the easeof-insertion tests for both the front and rear pockets of the outer 

protective vest, ensuring that the protective plates would properly fit 

in both pockets. 

The Army began to address the problems identified during Preliminary 

Design Model testing with the clay calibration tests and back-face 

deformation measurements. Army testers said they developed an informal 

set of procedures to determine when to repeat failed clay calibration tests. 

The procedures, which were not documented, called for repeating the 

Page 26 GAO-10-119 Warfighter Support
 

entire series of clay calibration drops if one of the calibration drops 

showed a failure.28 If the clay passes either the first or second test, the clay 

is to be used in testing. If the clay fails both the first and the second series 

of drops, the clay is to then be placed back in conditioning and testers get 

a new block of clay. With respect to back-face deformation measurements, 

Army testers measured back-face deformation at the deepest point, rather 

than at the point of aim.29

Army Did Not Follow All 

Established Testing 

Protocols during First 

Article Testing 

Although the Army began to address problems relating to the clay 

calibration tests and back-face deformation measurements, Army testers 

still did not follow all established testing protocols in these areas. As a 

result, the Army may not have achieved the objective of First Article 

Testing—to determine if the designs tested met the minimum 

requirements for ballistic protection. First, the orally agreed-upon 

procedures used by Army testers to conduct the clay calibration tests were 

inconsistent with the established testing protocols. Second, with respect 

to back-face deformation measurements, Army testers rounded back-face 

deformation measurements to the nearest millimeter, a practice that was 

neither articulated in the testing protocols nor consistent with Preliminary 

Design Model testing. Third, also with respect to back-face deformation 

measurements, Army testers introduced a new, unproven measuring 

device. 

Although Army testers told us that they had orally agreed upon an 

informal set of procedures to determine when to repeat failed clay 

calibration tests, those procedures are inconsistent with the established 

testing protocols. The Army deviated from established testing protocols by 

using clay that had failed the calibration test as prescribed by the testing 

protocols. The testing protocols specify that a series of three pre-test 

drops of a weight on the clay must be within specified tolerances before 

the clay is used. However, in several instances, the Army repeated the 

calibration test on the same block of clay after it had initially failed until 

the results of a subsequent series of three drops were within the required 

specifications. Army officials told us that the testing protocols do not 

specify what procedures should be performed when the clay does not pass 

 

28 When asked, Aberdeen Test Center officials could not produce a memo documenting this 

procedure or how they knew that it was consistently applied during the test. 

29 The Aberdeen Test Center dropped the procedure that measures depression at the aim 

point location used during Preliminary Design Model testing. 

Page 27 GAO-10-119 Warfighter Support
 

the first series of calibration drops, so Army officials stated they 

developed the procedure they followed internally prior to First Article 

Testing and provided oral guidance on those procedures to all test 

operators to ensure a consistent process. Officials we spoke with from the 

Army, private NIJ-certified laboratories, and industry had mixed opinions 

regarding the practice of re-testing failed clay, with some expressing 

concerns that performing a second series of calibration drops on clay that 

had failed might introduce risk that the clay may not be at the proper 

consistency for testing because as the clay rests it cools unevenly, which 

could affect the calibration.30 Aberdeen Test Center’s Test Operating 

Procedure states that clay should be conditioned so that the clay passes 

the clay calibration test, and Army officials, body armor testers from 

private laboratories, and body armor manufacturers we spoke to agreed 

that when clay fails the calibration test, this requires re-evaluation and 

sometimes adjustment of the clay calibration procedures used. After 

several clay blocks failed the clay calibration test on November 13, 2008, 

Army testers recognized that the clay conditioning process used was 

yielding clay that was not ideal and, as a result, Army testers adjusted their 

clay conditioning process by lowering the temperature at which the clay 

was stored. 

On that same day of testing, November 13, 2008, we observed heavy, cold 

rain falling on the clay blocks that were being transported to test lanes. 

These clay blocks had been conditioned that day in ovens located outside 

of the test ranges at temperatures above 100 degrees Fahrenheit to 

prepare them for testing, and then were transported outside uncovered on 

a cold November day through heavy rain on the way to the temperatureand humidity-controlled test lane. We observed an abnormally high level of 

clay blocks failing the clay calibration test and a significantly higher-thannormal level of failure rates for the plates tested on that day. The only 

significant variation in the test environment we observed that day was 

constant heavy rain throughout the day. Our analysis of test data31 also 

showed that 44 percent (4 of 9) of the first shots and 89 percent (8 of 9) of 

the second shots taken on November 13, 2008, resulted in failure 

 

30 We did not conduct an independent assessment of the appropriateness of re-testing failed 

clay. 

31 We analyzed all V0, threat D shots. We excluded V50 shots, as well as shots from all other 

threats either because those tests consistently did not result in penalty points or because 

those threats were not tested on November 13, 2008. 

Page 28 GAO-10-119 Warfighter Support
 

penalties.32 On all of the other days of testing only 14 percent (10 of 74) of 

the first shots and 42 percent (31 of 74) of the second shots resulted in 

failure penalties. Both of these differences are statistically significant, and 

we believe the differences in the results may be attributable to the 

different test condition on that day. The established testing protocols 

require the use of a specific type of non-hardening oil-based clay.33 Body 

armor testers from NIJ-certified private laboratories, Army officials 

experienced in the testing of body armor, body armor manufacturers, and 

the clay manufacturer we spoke with said that the clay used for testing is a 

type of sculpting clay that naturally softens when heat is added and that 

getting water on the clay backing material could cause a chemical bonding 

change on the clay surface.34 Those we spoke with further stated that the 

cold water could additionally cause the outside of the clay to cool 

significantly more rapidly than the inside causing the top layer of clay to 

be harder than the middle. They suggested that clay be conditioned inside 

the test lanes and said that clay exposed to water or extreme temperature 

changes should not be used. Army Test and Evaluation Command officials 

we spoke with said that there is no prohibition in the testing protocols on 

allowing rain to fall onto the clay backing material and that its exposure to 

water would not impact testing. However, these officials were unable to 

provide data to validate their assertion that exposure to water would not 

affect the clay used during testing or the testing results. Army test officials 

also said that, since the conclusion of First Article Testing, Aberdeen Test 

Center has procured ovens to allow clay to be stored inside test lanes, 

rather than requiring that the clay be transported from another room 

where it would be exposed to environmental conditions, such as rain. 

With respect to the issue of the rounding of back-face deformation 

measurements, during First Article Testing Army testers did not award 

penalty points for shots with back-face deformations between 43.0 and 

43.5 millimeters. This was because the Army decided to round back-face 

 

32 Testing officials disputed the inclusion of one of the plates in our analysis because it was 

ruled a no-test. We included this plate because we had a complete set of data for the test 

item and it was ruled a valid test on the lane, only to be discarded several days later 

because testing officials believed one of the shots was “questionable.” Based on the Army’s 

objection, we analyzed the data without this plate or two other no-test plates and found 38 

percent of first shots resulted in penalties. Furthermore, our analysis revealed that the 

proportion of failures or penalties on November 13, 2008, still differed substantially and/or 

significantly from the proportion on all other days. 

33 Roma Plastilina Number 1, manufactured by Chavant, Inc. 

34 We did not independently validate the information provided by these officials. 

Page 29 GAO-10-119 Warfighter Support
 

deformation measurements to the nearest millimeter—a practice that is 

inconsistent with the Army’s established testing protocols, which require 

that back-face deformation measurements in the clay backing not exceed 

43 millimeters and that is inconsistent with procedures followed during 

Preliminary Design Model testing. Army officials said that a decision to 

round the measurements for First Article Testing was made to reflect 

testing for past Army contract solicitations and common industry 

practices of recording measurements to the nearest millimeter.35 While we 

did not validate this assertion that rounding was a common industry 

practice, one private industry ballistics testing facility said that its practice 

was to always round results up, not down, which has the same effect as 

not rounding at all. Army officials further stated that they should have also 

rounded Preliminary Design Model results but did not realize this until 

March 2008—several weeks into Preliminary Design Model testing—and 

wanted to maintain consistency throughout Preliminary Design Model 

testing. The Army’s decision to round measurement results had a 

significant outcome on testing because two designs that passed First 

Article Testing would have instead failed if the measurements had not 

been rounded. 

With respect to the introduction of a new device to measure back-face 

deformation, the Army began to use a laser scanner to measure back-face 

deformation without adequately certifying that the scanner could measure 

against the standard established when the digital caliper was used as the 

measuring instrument. Although Army Test and Evaluation Command 

certified36 the laser scanner as accurate for measuring back-face 

deformation, we observed the following certification issues: 

• The laser was certified based on testing done in a controlled laboratory 

environment that is not similar to the actual conditions on the test 

lanes. For example, according to the manufacturer of the laser 

scanner, the scanner is operable in areas of vibration provided the area 

scanned and the scanning-arm are on the same plane or surface.37 This 

 

35 We reviewed a few test reports for body armor testing and found instances where backface deformation results were rounded and instances where they were not rounded. 

36 We did not evaluate the validity of the certification; however, it is worthy of note that the 

certification report states that the method of analysis used was somewhat unusual, that 

some of the results were discarded because of problems with the laser, and that changes 

were made to the laser during the testing process. 

37 We did not independently evaluate the manufacturer’s description of the capabilities of 

the laser scanner. 

Page 30 GAO-10-119 Warfighter Support
 

was not the case during testing, and thus it is possible the impact of 

the bullets fired may have thrown the scanner out of alignment or 

calibration. 

• The certification is to a lower level of accuracy than required by the 

testing protocols. The certification study says that the laser is accurate 

to 0.2 millimeters; however, the testing protocols require an accuracy 

of 0.1 millimeters or better. Furthermore, the official letter from the 

Army Test and Evaluation Command certifying the laser for use 

incorrectly stated the laser meets an accuracy requirement of 1.0 

millimeter rather than 0.1 millimeters as required by the protocols. 

Officials confirmed that this was not a typographical error. 

• The laser certification was conducted before at least three38 major 

software upgrades were made to the laser, which according to Army 

officials may have significantly changed the accuracy of the laser. 

Because of the incorporation of the software upgrades, Army testers 

told us that they do not know the accuracy level of the laser as it was 

actually used in First Article Testing. 

• In evaluating the use of the laser scanner, the Army did not compare 

the actual back-face deformation measurements taken by the laser 

with those taken by digital caliper, previously used during Preliminary 

Design Model testing and by NIJ-certified laboratories. According to 

vendor officials and Army subject matter experts, the limited data they 

had previously collected have shown that back-face deformation 

measurements taken by laser have generally been deeper by about 2 

millimeters than those taken by digital caliper. Given those preliminary 

findings, there is a significant risk that measurements taken by the 

laser may represent a significant change in test requirements. 

Although Army testing officials acknowledged that they were unable to 

estimate the exact accuracy of the laser scanner as it was actually used 

during testing, they believed that based on the results of the certification 

study, it was suitable for measuring back-face deformation. These test 

officials further stated that they initially decided to use the laser because 

they did not believe it was possible to measure back-face deformations to 

the required level of accuracy using the digital caliper. However, officials 

 

38 In addition, after First Article Testing was concluded Aberdeen Test Center installed 

additional software upgrades needed to correct errors discovered in subsequent tests. 

These errors, which were not identified until after First Article Testing was concluded, 

affected the First Article Testing results. 

Page 31 GAO-10-119 Warfighter Support
 

from PEO Soldier and private NIJ-certified laboratories have told us that 

they believe the digital caliper method is capable of making these 

measurements with the required level of accuracy39 and have been using 

this technique successfully for several years. PEO Soldier officials also 

noted that the back-face deformation measurements in the testing 

protocols were developed using this digital caliper method. Army testing 

officials noted that the laser certification study confirmed their views that 

the laser method was more accurate than the digital caliper. However, 

because of the problems with the study that we have noted in this report, 

it is still unclear whether the laser is the most appropriate and accurate 

technique for measuring back-face deformation. 

Although we did not observe problems in the Army’s determination of 

penetration results during Preliminary Design Model testing, during First 

Article Testing we observed that the Army did not consistently follow its 

testing protocols in determining whether a shot was a partial or a 

complete penetration. Army testing protocols require that penalty points 

be awarded when any fragment of the armor material is imbedded or 

passes into the soft under garment used behind the plate; however, the 

Army did not score the penetration of small debris through a plate as a 

complete penetration of the plate in at least one case that we observed. In 

this instance, we observed small fragments from the armor three layers 

deep inside the Kevlar backing behind the plate. This shot should have 

resulted in the armor’s receiving 1.5 penalty points, which would have 

caused the design to fail First Article Testing.40 Army officials said that 

testers counted the shot as only a partial penetration of the plate because 

it was determined that fibers of the Kevlar backing placed behind the plate 

were not broken,41 which they stated was a requirement for the shot to be 

counted as a complete penetration of the plate. This determination was 

made with the agreement of an Army subject-matter expert from PEOSoldier present on the lane. However, the requirement for broken fibers is 

inconsistent with the written testing protocols. Army officials 

acknowledged that the requirement for broken fibers was not described in 

 

39 We did not independently verify the level of accuracy of the digital caliper. However, the 

manufacturer’s stated accuracy is .01 millimeters for the digital caliper specifications we 

obtained. 

40 This solution is also one of the ones that would have failed had back-face deformations 

been scored at the deepest point, rather than at the point of aim, during Preliminary Design 

Model Testing 

41 We observed Kevlar fibers that were frayed and tattered. 

Page 32 GAO-10-119 Warfighter Support
 

the testing protocols or otherwise documented but said that Army testers 

discussed this before First Article Testing began. Figure 6 shows the tear 

in the fibers of the rear of the plate in question. 

Figure 6: Tears in Kevlar Backing Material after a Penetration of the Plate 

Source: Army.

Page 33 GAO-10-119 Warfighter Support
 

Federal internal control standards require that federal agencies maintain 

effective controls over information processing to help ensure 

completeness, accuracy, authorization, and validity of all transactions.42

However, the Army did not consistently maintain adequate internal 

controls to ensure the integrity and reliability of its test data. For example, 

in one case bullet velocity data were lost because the lane Test Director 

accidentally pressed the delete button on the keyboard, requiring a test to 

be repeated. Additionally, we noticed that the software being used with 

the laser scanner to calculate back-face deformation measurements lacked 

effective edit controls, which could potentially allow critical variables to 

be inappropriately modified during testing. We further observed a few 

cases in which testers attempted to memorize test data for periods of time, 

rather than writing that data down immediately. In at least one case, this 

practice resulted in the wrong data being reported and entered into the 

test records. 

Army Did Not Maintain 

Internal Controls over the 

Integrity and Reliability of 

Test Data at All Times 

Army Did Not Formally 

Document Significant 

Procedures That Deviated 

from Established Testing 

Protocols or Assess the 

Impact of These 

Deviations 

According to Army officials, decisions to implement those procedures that 

deviated from testing protocols were reviewed and approved by 

appropriate officials. However, these decisions were not formally 

documented, the testing protocols were not modified to reflect the 

changes, and vendors were not informed of the procedures. At the 

beginning of testing, the Director of Testing said that any change to the 

testing protocols has to be approved by several Army components; 

however, the Army was unable to produce any written documentation 

indicating approval of the deviations we observed by those components. 

With respect to internal control issues, Army officials acknowledged that 

before our review they were unaware of the specific internal control 

problems we identified. We noted during our review that in industry, as 

part of the NIJ certification process, an external peer review process is 

used to evaluate testing processes and procedures of ballistics testing 

facilities to ensure that effective internal controls are in place. However, 

we found that the Aberdeen Test Center has conducted no such reviews, a 

contributing factor to the Army’s lack of unawareness of the control 

problems we noted. 

As a result of the deviations from testing protocols that we observed, three 

of the five designs that passed First Article Testing would not have passed 

 

42 GAO, Internal Control: Standards for Internal Control in the Federal Government, 

GAO/AIMD-00-21.3.1 (Washington, D.C. 1999). 

Page 34 GAO-10-119 Warfighter Support
 

under the existing testing protocols. Furthermore, one of the remaining 

two designs that passed First Article Testing was a design that would have 

failed Preliminary Design Model testing if back-face deformation was 

measured in accordance with the established protocols for that test. Thus, 

four of the five designs that passed First Article Testing and were certified 

by the Army as ready for full production would have instead failed testing 

at some point during the process, either during the initial Preliminary 

Design Model testing or the subsequent First Article testing, if all the 

established testing protocols had been followed.43 As a result, the overall 

reliability and repeatability of the test results are uncertain. However, 

because ballistics experts from the Army or elsewhere have not assessed 

the impact of the deviations from the testing protocols we observed during 

First Article Testing, it is not certain whether the effect of these deviations 

is sufficient to call into question the ability of the armor to meet mission 

requirements. Although it is certain that some armor passed testing that 

would not have if specific testing protocols had been followed, it is 

unclear if there are additional factors that would mean the armor still 

meets the required performance specifications. For example, the fact that 

the laser scanner used to measure back-face deformation may not be as 

accurate as what the protocol requires may offset the effects of rounding 

down back-face deformations. Likewise, it is possible that some of the 

deviations that did not on their own have a visible effect on testing results 

could, when taken together with other deviations, have a combined effect 

that is greater. In our opinion, given the significant deviations in the 

testing protocols, independent ballistics testing expertise would be 

required to determine whether or not the body armor designs procured 

under this solicitation provide the required level of protection. The Army 

has ordered 2,500 sets of plates (at two plates per set) from those vendors 

whose designs passed First Article Testing to be used for additional 

ballistics testing and 120,000 sets of plates to be put into inventory to 

address future requirements. However, to date, none of these designs have 

been fielded because, according to Army officials, there are adequate 

quantities of armor plates produced under prior contracts already in the 

inventory to meet current requirements. 

 

43 The one design that would have passed both the Preliminary Design Model testing and 

the First Article Testing actually suffered a catastrophic first-shot penalty during First 

Article Testing, on November 13, 2008. However, Army testers later deemed this a 

“questionable” shot and ruled it a no-test. The design subsequently passed its re-test. 

Page 35 GAO-10-119 Warfighter Support
 

Body armor plays a critical role in protecting our troops, and the testing 

inconsistencies we identified call into question the quality and 

effectiveness of testing performed at Aberdeen Test Center. Because we 

observed several instances in which actual test practices deviated from the 

established testing protocols, it is questionable whether the Army met its 

First Article Testing objectives of ensuring that armor designs fully met 

Army’s requirements before the armor is purchased and used in the field. 

While it is possible that the testing protocol deviations had no significant 

net effect or may have even resulted in armor being tested to a more 

rigorous standard, it is also possible that some deviations may have 

resulted in armor being evaluated against a less stringent standard than 

required. We were unable to determine the full effects of these deviations 

as they relate to the quality of the armor designs and believe such a 

determination should only be made based on a thorough assessment of the 

testing data by independent ballistics testing experts. In light of such 

uncertainty and the critical need for confidence in the equipment by the 

soldiers, the Army would take an unacceptable risk if it were to field these 

designs without taking additional steps to gain the needed confidence that 

the armor will perform as required. 

Conclusions 

The Army is now moving forward with plans to conduct all future body 

armor testing at Aberdeen Test Center. Therefore, it is essential that the 

transparency and consistency of its program be improved by ensuring that 

all test practices fully align with established testing protocols and that any 

modifications in test procedures be fully reviewed and approved by the 

appropriate officials, with supporting documentation, and that the testing 

protocols be formally changed to reflect the revised or actual procedures. 

Additionally, it is imperative that all instrumentation, such as the laser 

scanner, used for testing be fully evaluated and certified to ensure its 

accuracy and applicability to body armor testing. Furthermore, it is 

essential that effective internal controls over data and testing processes be 

in place. The body armor industry has adopted the practice, through the 

NIJ certification program, of using external peer reviews to evaluate and 

improve private laboratories’ test procedures and controls. This type of 

independent peer review could be equally beneficial to the Aberdeen Test 

Center. Without all of these steps, there will continue to be uncertainty 

with regard to whether future testing data are repeatable and reliable and 

can be used to accurately evaluate body armor designs. Until Aberdeen 

Test Center has effectively honed its testing practices to eliminate the 

types of inconsistencies we observed, concerns will remain regarding the 

rigor of testing conducted at that facility. 

Page 36 GAO-10-119 Warfighter Support
 

To determine what effect, if any, the problems we observed had on the test 

data and on the outcomes of First Article Testing, we recommend the 

Secretary of Defense direct the Secretary of the Army to provide for an 

independent evaluation of the First Article Testing results by ballistics and 

statistical experts external to DOD before any armor is fielded to soldiers 

under this contract solicitation and that the Army report the results of that 

assessment to the office of the Director of Operational Test and 

Evaluation and the Congress. In performing this evaluation, the 

independent experts should specifically evaluate the effects of the 

following practices observed during First Article Testing: 

Recommendations for 

Executive Action 

• the rounding of back-face deformation measurements; 

• not scoring penetrations of material through the plate as a complete 

penetration unless broken fibers are observed in the Kevlar backing 

behind each plate; 

• the use of the laser scanner to measure back-face deformations 

without a full evaluation of its accuracy as it was actually used during 

testing, to include the use of the software modifications and operation 

under actual test conditions; 

• the exposure of the clay backing material to rain and other outside 

environmental conditions as well as the effect of high oven 

temperatures during storage and conditioning; and 

• the use of an additional series of clay calibration drops when the first 

series of clay calibration drops does not pass required specifications. 

To better align actual test practices with established testing protocols 

during future body armor testing, we recommend that the Secretary of the 

Defense direct the Secretary of the Army to document all key decisions 

made to clarify or change the testing protocols. With respect to the 

specific inconsistencies we identified between the test practices and 

testing protocols, we recommend that the Secretary of the Army, based on 

the results of the independent expert review of the First Article Test 

results, take the following actions: 

• Determine whether those practices that deviated from established 

testing protocols during First Article Testing will be continued during 

future testing and change the established testing protocols to reflect 

those revised practices. 

• Evaluate and re-certify the accuracy of the laser scanner to the correct 

standard with all software modifications incorporated and include in 

this analysis a side-by-side comparison of the laser measurements of 

the actual back-face deformations with those taken by digital caliper to 

Page 37 GAO-10-119 Warfighter Support
 

determine whether laser measurements can meet the standard of the 

testing protocols. 

To improve internal controls over the integrity and reliability of test data 

for future testing as well as provide for consistent test conditions and 

comparable data between tests, we recommend that the Secretary of 

Defense direct the Secretary of the Army to provide for an independent 

peer review of Aberdeen Test Center’s body armor testing protocols, 

facilities, and instrumentation to ensure that proper internal controls and 

sound management practices are in place. This peer review should be 

performed by testing experts external to the Army and DOD. 

DOD did not concur with our recommendation for an independent 

evaluation of First Article Testing results and accordingly plans to take no 

action to provide such an assessment. DOD asserted that the issues we 

identified do not alter the effects of testing. However, based on our 

analysis and findings there is sufficient evidence to raise questions as to 

whether the issues we identified had an impact on testing results. As a 

result, we continue to believe it is necessary to have an independent 

external expert review these test results and the overall effect of the 

testing deviations we observed on those results before any armor is fielded 

to military personnel. Without such an independent review, the First 

Article Test results remain questionable, undermining the confidence of 

the public and those who might rely on the armor for protection. 

Consequently, Congress should consider directing the Office of the 

Secretary of Defense to either require that an independent external review 

of these body armor test results be conducted or that DOD officially 

amend its testing protocols to reflect any revised test procedures and 

repeat First Article Testing to ensure that only properly tested designs are 

fielded. 

In written comments on a draft of this report, DOD takes the position that 

our findings had no significant impact on the test results and on the 

subsequent contracting actions taken by the Army. DOD also does not 

concur with what it perceives as our two overarching conclusions: (1) that 

Preliminary Design Model testing did not achieve its intended objective of 

determining, as a basis for contract awards, which designs met 

performance requirements and (2) that First Article Testing may not have 

met its objective of determining whether each of the contracted plate 

designs met performance requirements. DOD commented that it 

recognizes the importance of personal protection equipment such as body 

Matter for 

Congressional 

Consideration 

Agency Comments 

and Our Evaluation 

Page 38 GAO-10-119 Warfighter Support
 

armor and provided several examples of actions DOD and the Army have 

taken to improve body armor testing. DOD generally concurred with our 

findings that there were deviations from the testing protocols during 

Preliminary Design Model testing and First Article Testing. We agree that 

DOD has taken positive steps to improve its body armor testing program 

and to address concerns raised by Congress and others. DOD also 

concurred with our second recommendation to document all key 

decisions made to clarify or change the testing protocols. DOD did not 

concur with our first recommendation that an independent evaluation of 

First Article Testing results be performed by independent ballistics and 

statistical experts before any of the armor is fielded to soldiers under 

contracts awarded under this solicitation. Similarly, DOD did not agree 

with our conclusions that Preliminary Design Model testing did not meet 

its intended objectives and that First Article Testing may not have met its 

intended objectives. In supporting its position, DOD cited, for example, 

that rounding back-face deformation measurements during First Article 

Testing was an acceptable test practice because rounding is a practice that 

has been used historically. It was the intent of PEO Soldier to round backface deformations for all testing associated with this solicitation, and the 

Integrated Product Team decided collectively to round back-face 

deformations during First Article Testing. However, as stated in our report 

and acknowledged by DOD, the rounding down of back-face deformations 

was not spelled out or provided for by any of the testing protocol 

documents. Additionally, it created an inconsistency between Preliminary 

Design Model testing, where back-face deformations were not rounded 

down and in First Article Testing, where back-face deformations were 

rounded down. Of greatest consequence, rounding down back-face 

deformations lowered the requirements that solutions had to meet to pass 

testing. Two solutions passed First Article Testing because back-face 

deformations were rounded down, meaning that the Army may be taking 

unacceptable risk if plates are fielded without an additional, independent 

assessment by experts. 

DOD also did not agree with our finding that a penetration of a plate was 

improperly scored. DOD did agree that figure 6, which shows the tear in 

the Kevlar fibers of the rear of the plate in question, appears to show 

evidence of a perforation and that an Aberdeen Test Center ballistics 

subject matter expert found particles in the soft backing material behind 

the plate. Nevertheless, DOD did not concur with our finding because it 

asserted that no threads were broken on the first layer of Kevlar. However, 

Page 39 GAO-10-119 Warfighter Support
 

as we stated in the report, the protocols define a complete penetration44 as 

having occurred when the projectile, fragment of the projectile, or 

fragment of the armor material is imbedded or passes into the soft under 

garment used behind the protective inserts plates, not when threads of the 

Kevlar are broken. The fragments found by the Aberdeen Test Center 

subject matter expert, as well as the three frayed, tattered, and separated 

Kevlar layers that we and Army testers observed, confirm our observations 

during testing. DOD also stated that the first layer of soft armor behind the 

plate under test serves as a witness plate during testing and if that first 

layer of soft armor is not penetrated, as determined by the breaking of 

threads on that first layer of soft armor, the test shot is not scored as a 

complete penetration in accordance with the PEO Soldier’s scoring 

criteria. We disagree with DOD’s position because the protocols do not 

require the use of a “witness plate” during testing to determine if a 

penetration occurred. If this shot would have been ruled a complete 

penetration rather than a partial penetration, this design would have 

accrued additional point deductions causing it to fail First Article Testing. 

DOD did not agree that the certification of the laser scanner was 

inadequate and made several statements in defense of both the laser and 

its certification. Among these is the fact that the laser removes the human 

factor of subjectively trying to find the deepest point, potentially pushing 

the caliper into the clay, and removing the need to use correction factors, 

all of which we agree may be positive things. However, we maintain that 

the certification of the laser was not adequately performed. As indicated in 

the certification letter, the laser was certified to a standard that did not 

meet the requirement of the testing protocols. Additionally, DOD stated 

that software modifications added to the laser after certification did not 

affect measurements; however, Army testers told us on multiple occasions 

that the modifications were designed to change the measurements 

reported by the laser. DOD added that the scanner does not artificially 

overstate back-face deformations and relies on the verified accuracy of the 

scanner and the study involving the scanning of clay replicas to support its 

claim. Based on our observations, the scanner was certified to the wrong 

standard and the certification study was not performed in the actual test 

environment using actual shots. DOD asserts that the scanner does not 

overstate back-face deformations and that it does not establish a new 

requirement. However, DOD cannot properly validate these assertions 

 

44 NIJ Standard, Section 3.34 is consistent with this definition, Ballistic Resistance of Body 

Armor, NIJ Standard-0101.06, July 2008. 

Page 40 GAO-10-119 Warfighter Support
 

without a side-by-side comparison of the laser scanner and the digital 

caliper in their operational environment. Given the numerous issues 

regarding the laser and its certification, we maintain that its effect on First 

Article Testing should be examined by an external ballistics expert. 

DOD also stated that it did not agree with our finding that exposure of the 

clay backing to heavy rain on one day may have affected test results. DOD 

challenged our statistical analysis and offered its own statistical analysis 

as evidence that it was the poor designs themselves that caused unusual 

test results that day. We stand by our analysis, in combination with 

statements made by DOD and non-DOD officials with testing expertise and 

by the clay manufacturer, that exposure of the clay to constant, heavy cold 

rain may have had an effect on test results. Further, in analyzing the 

Army’s statistical analysis presented in DOD’s comments, we did not find 

this information to demonstrate that the designs were the factor in 

unusual test results that day or that the rain exposure could not have had 

an effect on the results. More detailed discussions of the Army’s analysis 

and our conclusions are provided in comments 13 and 24 of appendix II. 

DOD partially disagreed that the use of an additional series of clay 

calibration drops when the first series of drops were outside specifications 

did not meet First Article Test requirements and added that all clay used in 

testing passed the clay calibration in effect at the time. However, we 

witnessed several clay calibration drops that were not within 

specifications. These failed clay boxes were repaired, re-dropped, and 

either used if they passed the subsequent drop calibration series or 

discarded if they failed. The protocols only allow for one series of drops 

per clay box, which is the methodology that Army testers should have 

followed. DOD stated that NIJ standards do permit the repeating of failed 

calibration drops. However, our review of the NIJ standards45 reveals that 

there is no provision that allows repeat calibration drops. DOD states in its 

comments that NIJ standards are inappropriate for its test facilities, 

stating that these standards are insufficient for the U.S. Army given the 

expanded testing required to ensure body armor meets U.S. Army 

requirements. NIJ standards were not the subject of our review, but rather 

Aberdeen Test Center’s application of the Army’s current solicitation’s 

protocols during testing. Further, DOD acknowledged in its comments that 

 

45 Army protocols require only a series of three pre-shot calibration drops. NIJ Section 

4.2.5.6 requires that a series of five pre-shot and a series of five post-shot calibration drops 

be within specification or a new conditioned and calibration drop validated clay be used—

Ballistic Resistance of Body Armor, NIJ Standard-0101.06, July 2008. 

Page 41 GAO-10-119 Warfighter Support
 

National Institute of Standards and Technology officials recommended 

only one series of drops for clay calibration. However, DOD stated that it 

will partner with the National Institute of Standards and Technology to 

study procedures for clay calibration, to include repeated calibration 

attempts, and document any appropriate procedural changes, which we 

agree is a good step. 

Based on our analyses as described in our report and in our above 

responses to DOD’s comments, we believe there is sufficient evidence to 

raise questions as to whether the issues we identified had an impact on 

testing results. As a result, we continue to believe that it is necessary that 

DOD allow an independent external expert to review these test results and 

the overall effect of DOD’s deviations on those results before any armor is 

fielded to military personnel. Without such an independent review, it is our 

opinion that the First Article Testing results will remain questionable. 

Consequently, we have added a matter for congressional consideration to 

our report suggesting that Congress consider either directing DOD to 

require that an independent external review of these body armor test 

results be conducted or require that DOD officially amend its testing 

protocols to reflect any revised test procedures and repeat First Article 

Testing to ensure properly tested designs. 

DOD partially concurred with our third recommendation to determine 

whether those procedures that deviated from established testing protocols 

during First Article Testing should be continued during future testing and 

to change the established testing protocols to reflect those revised 

procedures. DOD recognized the need to update testing protocols and 

added that when the office of the Director of Operational Test and 

Evaluation promulgates standard testing protocols across DOD, these 

standards will address issues that we identified. As long as DOD 

specifically addresses all the inconsistencies and deviations that we 

observed prior to any future body armor testing, this would satisfy our 

recommendation. 

DOD stated that it partially concurs with our fourth recommendation to 

evaluate and recertify the accuracy of the laser scanner to the correct 

standard with all software modifications incorporated, based on the 

results of the independent expert review of the First Article Testing 

results. We also recommended that this process include a side-by-side 

comparison of the laser’s measurement of back-face deformations and 

those taken by digital caliper. DOD concurred with the concept of an 

independent evaluation, but it did not concur that one is needed in this 

situation because according to DOD its laser certification was sufficient. 

Page 42 GAO-10-119 Warfighter Support
 

We disagree that the laser certification was performed correctly. As 

discussed in the body of our report and further in appendix II, 

recertification of the laser is critical because (1) the laser was certified to 

the wrong standard, (2) software modifications were added after the 

certification of the laser, and (3) these modifications did change the way 

the laser scanner measured back-face deformations. DOD did not 

explicitly state whether it concurred with our recommendation for a sideby-side comparison of the laser scanner and the digital caliper in their 

operational environment. We assert that such a study is important because 

without it the Army and DOD do not know the effect the laser scanner may 

have on the back-face deformation standard that has been used for many 

years and was established with the intention of being measured with a 

digital caliper. If the comparison reveals a significant difference between 

the laser scanner and the digital caliper, DOD and the Army may need to 

revisit the back-face deformation standard of its requirements with the 

input of industry experts and the medical community. 

DOD generally concurred with our fifth recommendation to conduct an 

independent evaluation of the Aberdeen Test Center’s testing protocols, 

facilities, and instrumentation and stated that such an evaluation would be 

performed by a team of subject matter experts that included both DOD 

and non-DOD members. We agree that in principal this approach meets the 

intent of our recommendation as long as the DOD46 members of the 

evaluation team are independent and not made up of personnel from those 

organizations involved in the body armor testing such as office of the 

Director of Operational Test and Evaluation, the Army Test and Evaluation 

Command, or PEO Soldier. DOD’s comments and our specific responses to 

them are provided in appendix II. 

We are sending copies of this report to the appropriate congressional 

committees, the Secretary of Defense, and the Secretary of the Army. In 

addition, the report will be available at no charge on GAO’s Web site at 

http://www.gao.gov. If you or your staff has any questions about this 

report, please contact me at (202) 512-8365 or solisw@gao.gov. Contact 

 

46 DOD members should not have veto power over non-DOD members. 

Page 43 GAO-10-119 Warfighter Support
 

points for our Offices of Congressional Relations and Public Affairs may 

be found on the last page of this report. GAO staff who made major 

William M. Solis 

contributions to this report are listed in appendix III. 

Director, Defense Capabilities and Management 

Page 44 GAO-10-119 Warfighter Support
 

List of Requesters 

The Honorable Carl Levin 

Chairman 

The Honorable John McCain 

Ranking Member 

Committee on Armed Services 

United States Senate 

The Honorable Jim Webb 

United States Senate 

The Honorable Ike Skelton 

Chairman 

The Honorable Howard McKeon 

Ranking Member 

Committee on Armed Services 

United States House of Representatives 

The Honorable Neil Abercrombie 

Chairman 

The Honorable Roscoe Bartlett 

Ranking Member 

Subcommittee on Air and Land Forces 

Committee on Armed Services 

United States House of Representatives 

The Honorable Joe Courtney 

United States House of Representatives 

Page 45 GAO-10-119 Warfighter Support
Appendix I: Scope and Methodology Appendix I: Scope and Methodology 

Our review of body armor testing focused on testing conducted by the 

Army in response to specific concerns raised by the House and Senate 

Armed Services Committees and multiple members of Congress. During 

our review, we were present during two rounds of testing of body armor 

designs that were submitted in response to a May 2007-February 2008 

Army contract solicitation. The first round of testing, called Preliminary 

Design Model testing, was conducted from February 2008 through June 

2008 with the objective of determining whether designs submitted under 

the contract solicitation met the required ballistic performance 

specifications and were eligible for contract award. The second round of 

testing, called First Article Testing, was conducted between November 

2008 and December 2008 on the body armor designs that passed the 

Preliminary Design Model testing. Both tests were conducted at Aberdeen 

Proving Grounds in Aberdeen, Md., and were performed by Aberdeen Test 

Center. During the course of our review, we observed how the Army 

conducted its body armor testing and compared our observations with the 

established body armor testing protocols. We did not verify the accuracy 

of the Army’s test data and did not provide an expert evaluation of the 

results of testing. To understand the practices the Army used and the 

established testing protocols we were comparing the practices with, we 

met with and/or obtained data from officials from the Department of 

Defense (DOD) organizations and the industry experts listed in table 1: 

Table 1: Organizations Contacted for Information about Body Armor Testing 

DOD acquisition organization DOD testing organization Industry expert 

Program Executive Office Soldier Army Test and Evaluation Command U.S. Laboratories 

Developmental Test Command H.P. White Laboratories 

Aberdeen Test Center Various body armor manufacturers 

Army Research Laboratory 

DOD’s office of the Director of Operational Test and 

Evaluation 

Source: GAO. 

To determine the degree to which the Army followed established testing 

protocols during the Preliminary Design Model testing of body armor 

designs, we were present and made observations during the entire period 

of testing, compared our observations with established testing protocols, 

and interviewed numerous DOD and other experts about body armor 

testing. We observed Army testers as they determined whether designs 

met the physical and ballistics specifications described in the contract 

solicitation, and as encouraged by Aberdeen Test Center officials, we 

Page 46 GAO-10-119 Warfighter Support 
Appendix I: Scope and Methodology 

observed the ballistics testing from inside a viewing room equipped with 

video and audio connections to the firing lanes. We also were present and 

observed the physical characterization of the test items and visited the 

environmental conditioning chambers, the weathering chamber, and the Xray facility. We were at Aberdeen Test Center when the designs were 

delivered for testing on February 7, 2008, and were on-site every day of 

physical characterization, which comprises the steps performed to 

determine whether each design meets the required weight and 

measurement specifications. We systematically recorded our observations 

of physical characterization on a structured, paper data-collection 

instrument that we developed after consulting with technical experts from 

Program Executive Office (PEO) Soldier before testing started. We were 

also present for every day except one of the ballistics testing, observing 

and collecting data on approximately 80 percent of the tests from a video 

viewing room that was equipped with an audio connection to each of the 

three firing lanes. To gather data from the day that we were not present to 

observe ballistic testing, we viewed that day’s testing on video playback. 

We systematically recorded our observations of ballistics testing using a 

structured, electronic data-collection instrument that we developed to 

record relevant ballistic test data—such as the shot velocity, penetration 

results, and the amount of force absorbed (called “back-face 

deformation”) by the design tested. Following testing, we supplemented 

the information we recorded on our data collection instrument with some 

of the Army’s official test data and photos from its Vision Digital Library 

System. We developed the data collection instrument used to collect 

ballistics testing data by consulting with technical experts from Program 

Executive Office Soldier and attending a testing demonstration at 

Aberdeen Test Center before Preliminary Design Model testing began. 

After capturing the Preliminary Design Model testing data in our data 

collection instruments, we compared our observations of the way the 

Aberdeen Test Center conducted testing with the testing protocols that 

Army officials told us served as the testing standards at the Aberdeen Test 

Center. According to these officials, these testing protocols comprised the 

(1) test procedures described in the contract solicitation announcement’s 

purchase descriptions and (2) Army’s detailed test plans and Test 

Operating Procedure that serve as guidance to the Aberdeen Test Center 

testers and that were developed by the Army Test and Evaluation 

Command and approved by Program Executive Office Soldier, the office 

of the Director of Operational Test and Evaluation, the Army Research 

Labs, and cognizant Army components. We also reviewed National 

Institute of Justice testing standards because Aberdeen Test Center 

officials told us that, although Aberdeen Test Center is not a National 

Institute of Justice-certified testing facility, they have made adjustments to 

Page 47 GAO-10-119 Warfighter Support
Appendix I: Scope and Methodology 

their procedures based on those standards and consider them when 

evaluating Aberdeen Test Center’s test practices. Regarding the edge shot 

locations for the impact test samples, we first measured the area of 

intended impact on an undisturbed portion of the test item on all 56 test 

samples after the samples had already been shot.1

 The next day we had 

Aberdeen Test Center testers measure the area of intended impact on a 

random sample of the impact test samples to confirm our measurements. 

Throughout testing we maintained a written observation log and compiled 

all of our ballistic test data into a master spreadsheet. Before, during, and 

after testing, we interviewed representatives from numerous Army 

entities—including the Assistant Secretary of the Army for Acquisition, 

Technology and Logistics; Aberdeen Test Center; Developmental Test 

Command; Army Research Laboratories; and Program Executive Office 

Soldier—and also attended Integrated Product Team meetings. 

To determine the degree to which the Army followed established testing 

protocols during First Article Testing of the body armor designs that 

passed Preliminary Design Model testing, we were present and made 

observations during the entire period of testing, compared our 

observations with established testing protocols, and interviewed 

numerous DOD and industry experts about body armor testing. As during 

Preliminary Design Model testing, we observed Army testers as they 

determined whether designs met the physical and ballistics specifications 

described in the contract solicitation. However, different from our review 

of Preliminary Design Model testing, we had access to the firing lanes 

during ballistic testing. We also still had access to the video viewing room 

used during Preliminary Design Model testing, so we used a bifurcated 

approach of observing testing from both the firing lanes and the video 

viewing room. We were present for every day except one of First Article 

Testing—from the first day of ballistics testing on November 11, 2008, until 

the final shot was fired on December 17, 2008.2

 We noted the weights and 

 

1

 Testers marked an area of intended impact by drawing two long lines, one marking the 

inner shot tolerance and the other marking the outer shot tolerance. Both our 

measurements and those taken by Aberdeen Test Center testers were taken by measuring 

the distance between the two lines and the edge on a part of the test sample significantly 

removed from where the shot actually impacted. We could only take these measurements 

on hard plate samples because the flexible samples were marked differently, in a way that 

we could not obtain an accurate measurement. 

2

 Ballistics testing was stopped on one occasion because of a higher priority Army test that 

needed to be conducted at Aberdeen Test Center that involved cracked plates shipped from 

Iraq as part of PEO Soldier’s non-destructive X-ray life cycle testing. Most of the First 

Article Testing concluded on December 4, but one retest was conducted on December 17. 

Page 48 GAO-10-119 Warfighter Support
Appendix I: Scope and Methodology 

measures of plates during physical characterization on the same data 

collection instrument that we used during Preliminary Design Model 

testing. For the ballistics tests, we revised our Preliminary Design Model 

testing data collection instrument so that we could capture data while in 

the firing lane—data that we were unable to confirm first hand during 

Preliminary Design Model testing. For example, we observed the pre-shot 

measurements of shot locations on the plates and the Aberdeen Test 

Center’s method for recording data and tracking the chain of custody of 

the plates; we also recorded the depth of the clay calibration drops (the 

series of pre-test drops of a weight on clay that is to be placed behind the 

plates during the shots), the temperature of the clay, the temperature and 

humidity of the firing lane, the temperatures in the fluid soak conditioning 

trailer, and the time it took to perform tests. We continued to record all of 

the relevant data that we had recorded during Preliminary Design Model 

testing, such as the plate number, type of ballistic subtest, the charge 

weight of the shot, the shot velocity, the penetration results, and the backface deformation. Regarding the new laser arm that Aberdeen Test Center 

acquired to measure back-face deformation during First Article Testing, 

we attended a demonstration of the arm’s functionality performed by 

Aberdeen Test Center and also acquired documents related to the laser 

arm’s certification by Army Test, Measurement, and Diagnostic Equipment 

activity. With a GAO senior methodologist and a senior technologist, we 

made observations related to Aberdeen Test Center’s methods of handling 

and repairing clay, calibrating the laser guide used to ensure accurate 

shots, and measuring back-face deformation. Throughout testing we 

maintained a written observation log and compiled all of our ballistic test 

data into a master spreadsheet. Following testing, we supplemented the 

information we recorded on our data collection instrument with some of 

the Army’s official test data and photos from its Vision Digital Library 

System to complete our records of the testing. After capturing the testing 

data in our data collection instruments, we compared our observations of 

the way Aberdeen Test Center conducted testing with the testing protocols 

that Army officials told us served as the testing standards at the Aberdeen 

Test Center. In analyzing the potential impact of independent variables on 

testing, such as the potential impact of the November 13th rain on the clay, 

we conducted statistical tests including chi-square and Fisher’s Exact Test 

methods to accommodate small sample sizes. Before, during, and after 

testing, we interviewed representatives from numerous Army agencies, 

including Aberdeen Test Center, Developmental Test Command, Army 

Research Laboratories, and Program Executive Office Soldier. We also 

spoke with vendor representatives who were present and observing the 

First Article Testing, as well as with Army and industry subject matter 

experts. 

Page 49 GAO-10-119 Warfighter Support
Appendix I: Scope and Methodology 

We conducted this performance audit from July 2007 through October 

2009 in accordance with generally accepted government auditing 

standards. Those standards require that we plan and perform the audit to 

obtain sufficient, appropriate evidence to provide a reasonable basis for 

our findings and conclusions based on our audit objectives. We believe 

that the evidence obtained provides a reasonable basis for our findings 

and conclusions based on our audit objectives. 

Page 50 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department Appendix II: Comments from the Department of Defense 

of Defense 

Note: GAO comments 

supplementing those in 

the report text appear at 

the end of this appendix. 

Page 51 GAO-10-119 Warfighter Support 
Appendix II: Comments from the Department 

of Defense 

See comment 1. 

Page 52 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

See comment 2. 

See comment 3. 

Page 53 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

See comment 4. 

See comment 5. 

Page 54 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

See comment 6. 

See comment 7. 

Page 55 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

See comment 8. 

Page 56 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

Page 57 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

See comment 9. 

See comment 10. 

Page 58 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

See comment 11. 

See comment 12. 

Page 59 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

Page 60 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

See comment 13. 

Page 61 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

See comment 14. 

Page 62 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

See comment 15. 

Page 63 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

See comment 16. 

Page 64 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

See comment 17. 

See comment 18. 

See comment 19. 

See comment 20. 

Page 65 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

See comment 21. 

See comment 22. 

Page 66 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

Page 67 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

Page 68 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

Page 69 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

Page 70 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

See comment 23. 

See comment 24. 

Page 71 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

See comment 25. 

Page 72 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

See comment 26. 

See comment 27. 

Page 73 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

See comment 28. 

Page 74 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

See comment 29. 

Page 75 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

Page 76 GAO-10-119 

1. The Department of Defense (DOD) stated that undertakings of this 

magnitude are not without flaws and that what was most important was 

fielding body armor plates that defeated the threat. While DOD may have 

identified some flaws that may not be serious enough to call the testing 

results into question, several of the deviations to the testing protocols that 

we observed do call the testing results into question for the reasons stated 

in our report. An independent expert has not evaluated the impact of these 

deviations on the test results and, until such a study is conducted, DOD 

cannot be assured that the plates that passed testing can defeat the threat. 

DOD also noted several actions DOD and the Army have taken to improve 

procedures associated with body armor testing. Our responses to these 

actions are included in comments 2 through 6. 

GAO Comments 

2. The office of the Director of Operational Test and Evaluation’s efforts to 

respond to members of the Armed Services Committees and to address 

issues raised by the Department of Defense Inspector General were 

outside the scope of our audit. Therefore, we did not validate the 

implementation of the actions DOD cited or evaluate their effectiveness in 

improving test procedures. With regard to the office of the Director of 

Operational Test and Evaluation’s establishing a policy to conduct First 

Article Testing at government facilities, using a government facility to 

conduct testing may not necessarily produce improved test results. 

3. Regarding the office of the Director of Operational Test and Evaluation’s 

oversight of testing, the office of the Director of Operational Test and 

Evaluation led the Integrated Product Team and approved the test plans. 

However, while we were present at the Aberdeen Test Center during 

Preliminary Design Model testing and First Article Testing, we did not 

observe on-site monitoring of the testing by the office of the Director of 

Operational Test and Evaluation staff beyond incidental visits during VIP 

events and other demonstrations. 

4. Regarding the procedures and policies DOD stated were implemented 

by the Army Test and Evaluation Command to improve testing: 

• Only two of the test ranges were completed prior to Preliminary 

Design Model testing. Two additional test ranges were completed after 

Preliminary Design Model testing. 

• Regarding the certification of the laser scanner measurement device, 

as noted in our report, the Army had not adequately certified that it 

was an appropriate tool for body armor testing (see our comment 12). 

Warfighter Support 
Appendix II: Comments from the Department 

of Defense 

• The Army’s Test Operating Procedure was not completed or 

implemented until after Preliminary Design Model testing.1

• New clay conditioning chambers inside each test range were not 

constructed until after all testing was completed (see our comment 

13). 

• The improved velocity measurement accuracy study was not 

conducted until after all testing was completed. 

• Regarding the implementation of electronic data collection and 

processing for body armor testing, as stated in our report, we observed 

that not all data are electronically collected. Many types of data are 

manually collected and are later converted to electronic data storage. 

5. Regarding Program Executive Office (PEO) Soldier’s efforts to improve 

the acquisition of personal protection equipment: 

• The contract solicitation allowed all prospective body armor 

manufacturers to compete for new contracts. 

• We observed that PEO Soldier did transfer expertise and experience to 

support Army Acquisition Executive direction that all First Article 

Testing and lot-acceptance testing be conducted by the Army Test and 

Evaluation Command. 

• The task force that focused on soldier protection was not initiated 

until February 2009, after all Preliminary Design Model testing and 

First Article Testing was completed. 

• According to Army officials, PEO Soldier instituted a non-destructive 

test capability that became operational after Preliminary Design Model 

testing, but prior to First Article Testing. 

• PEO Soldier’s personal protection evaluation process was described in 

our previous report—GAO-07-662R. Although we recognized the 

strength of PEO Soldier’s personal protection evaluation process in 

our earlier report, not all the protections that were in place at that time 

remain in place. For example, the requirement that testing be 

 

1

 During Preliminary Design Model testing, the most current Army Test Operating 

Procedure for testing body armor had not been updated since 1975. Test Operations 

Procedure (TOP), 10-2-506 Ballistic Testing of Personnel Armor Materials. January 6, 1975. 

Page 77 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

conducted at a National Institute of Justice (NIJ)-certified facility was 

waived. 

6. DOD stated that many of the actions by Army Test and Evaluation 

Command and PEO Soldier were initiated and improved upon during the 

course of our review. However, as discussed above, several of these 

actions were initiated before and during testing, but many of them were 

not completed until after testing was completed. 

7. DOD and the Army stated that Preliminary Design Model testing had 

achieved its objective to identify those vendor designs that met the 

performance objectives stated in PEO Soldier’s purchase description and 

that “it is incorrect to state that ‘at least two’ of the preliminary design 

models should have failed as they passed in accordance with the modified 

solicitation.” We disagree with these statements. As stated in our report, 

the most consequential of the deviations from testing protocols we 

observed involved the measurement of back-face deformation, which did 

affect final test results. According to original testing protocols, back-face 

deformation was to be measured at the deepest point of the depression in 

the clay backing. This measure indicates the most force that the armor 

will allow to be exerted on an individual struck by a bullet. According to 

Army officials, the deeper the back-face deformation measured in the clay 

backing, the higher the risk of internal injury or death. DOD and the Army 

now claim that these solutions passed in accordance with the modified 

solicitation, which overlooks the fact that the reason the solicitation had 

to be modified was that Army testers deviated from the testing protocols 

laid out in the purchase descriptions and did not measure back-face 

deformation at the deepest point. DOD and the Army also stated in their 

response that they decided to use the point of aim because they 

determined it was an accurate and repeatable process. Yet in DOD’s 

detailed comments regarding edge shot locations, DOD acknowledged that 

there were “potential variances between the actual aim point and impact 

point during testing.” Army Research Laboratory2

 and NIJ-certified 

laboratories use the benchmark process of measuring back-face 

deformation at the deepest point, not at the point of aim. 

As set forth in our report, at least two solutions passed Preliminary Design 

Model testing that would have failed if back-face deformation had been 

measured to the deepest point. This statement came directly from 

 

2

 Member of the Integrated Product Team (IPT). 

Page 78 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

Aberdeen Test Center officials during a meeting in July 2008, where they 

specifically told us which two solutions would have failed. We said “at 

least” two because Army testers did not record deepest point back-face 

deformation data for the first 30 percent of testing, and therefore there 

could be more solutions that would have failed had the deepest point been 

measured during this first portion of the test. Because the Army did not 

measure back-face deformation to the deepest point, it could not identify 

whether these two solutions in particular and all the solutions in general 

met performance requirements. As a result, Army could not waive First 

Article Testing for successful candidates and was forced to repeat the test 

to ensure that all solutions did indeed meet requirements. By repeating 

testing, the Army incurred additional expense and further delays in 

fielding armor from this solicitation to the soldiers. During the course of 

our audit, the Army also acknowledged that the Preliminary Design Model 

testing did not meet its objective because First Article Testing could not be 

waived without incurring risk to the soldiers. 

DOD and the Army stated that, upon discovery of the back-face 

deformation deviation from the testing protocols described in the 

purchase descriptions, the Army stopped testing. The Army’s Contracting 

Office was informed of this deviation through a series of questions posed 

by a vendor who was present at the Vendor Demonstration Day on 

February 20, 2008. This vendor sent questions to the Contracting Office on 

February 27 asking whether testers were measuring at the aim point or at 

the deepest point. This vendor also raised questions about how damage to 

the soft pack would be recorded and about the location of edge shots. 

Based on our observations, all of these questions involved issues where 

Army testers deviated from testing protocols and are discussed in our 

responses to subsequent comments. The Army did not respond until 

March 19 and replied that its test procedures complied with solicitation 

requirements. It was not until Army leadership learned of the vendor’s 

questions and of the deviation in measuring back-face deformation that 

testing was finally halted on March 27, a full month after the issue came to 

the Army Test and Evaluation Command’s attention. 

8. DOD stated that in 2007, prior to the initiation of Preliminary Design 

Model testing, the Army Test and Evaluation Command, the office of the 

Page 79 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

Director of Operational Test and Evaluation, and Army leadership all3

agreed that First Article Testing would be conducted as part of the Army’s 

body armor testing. However, DOD did not provide any documentation 

dated prior to April 2008—that is, prior to the discovery of the back-face 

deformation deviation—that suggested that DOD intended to conduct First 

Article Testing following Preliminary Design Model testing. In July 2008, 

the Army Test and Evaluation Command and PEO Soldier stated in official 

written responses to our questions regarding Preliminary Design Model 

testing that the conduct of First Article Testing became essential following 

Preliminary Design Model testing because of the Army’s measuring backface deformation at the point of aim as opposed to at the deepest point of 

deformation. In fact, because of this deviation, DOD could not waive First 

Article Testing as originally planned and was forced to conduct 

subsequent tests to verify that the designs that had passed Preliminary 

Design Model testing met testing requirements. DOD asserted that a multiphase concept including Preliminary Design Model testing, First Article 

Testing, and extended ballistic testing to support the development of an 

improved test standard was briefed to a congressional member and 

professional staff on November 14, 2007. We were present at this 

November 14 test overview and strategy/schedule briefing and noted that 

it did not include plans for First Article Testing to be performed in 

addition to Preliminary Design Model testing. Excerpts from the slides 

briefed that day showed Preliminary Design Model (Phase 1) testing and a 

subsequent ballistic and suitability testing (Phase 2). As indicated in the 

slides (see fig. 7 and fig. 8) from that November 14 briefing, the Phase 2 

test was designed to test the form, fit, and function of those solutions that 

had passed Preliminary Design Model testing as well as the ballistic 

statistical confidence tests.4

 According to information we obtained, Phase 

2 was never intended to be First Article Testing and was to have no impact 

on whether or not a solution received a contract. 

 

3

 Omitted from this list of agencies agreeing that First Article Testing was part of the 

original testing plan are (1) PEO Soldier, the Army’s materiel developer and product 

manager for individual protection equipment being tested and the contracting officer, and 

(2) the U.S. Army Research, Development, and Engineering Command’s Contracting 

Agency. Both entities told us that First Article Testing was going to be waived. 

4

 Form, fit, and function and the test for high ballistic statistical confidence were not part of 

First Article Testing. Form, fit and function tests involved having soldiers wear the body 

armor and evaluate its comfort and suitability when performing deployment (war-like) 

activities—egression from armored vehicles, the double-time run, moving through an 

obstacle course, and discharging their weapons. 

Page 80 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

Figure 7: Briefing Slide from DOD’s Test Overview (Nov. 14, 2007) 

Phase I

Phase 2

1. Physical inspection to ensure compliance with contract 

requirements and to document material condition.

2. Ballistic testing and analysis to evaluate test article performance 

pursuant to contract requirements.

3. Service test data and all supporting data provided to the Source 

Selection Technical Factor Chief.

4. Contract awards(s) made based on best value to the government 

as determined by the Source Selection Panel analyses of all available 

data. Bid samples provided by vendors subject to Source Selection 

Testing IAW First Article Test protocol. Material provided under those 

awards also provides test articles for Phase 2.

1. Ballistic testing to provide high statistical confidence in material 

performance. 

2. Suitability testing for form fit, and function. Testing conducted using 

operarational Soldiers (Ft. Benning, GA to support).

3. Service T&E report.

Source: Provided by Director of Operational Test and Evaluation at House Armed Services Committee briefing on

November 14, 2007.

*Phase 1 testing described is per the Army test plan approved by CG ATEC on September 11, 2007 

and by DOT&E on September 19, 2007.

Page 81 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

Figure 8: Briefing Slide from DOD’s Test Strategy and Schedule (Nov. 14, 2007) 

Source: Provided by Director of Operational Test and Evaluation at House Armed Services Committee Briefing on

November 14, 2007. Sensitive ballistic information removed form the "description" box (lower, left-hand-side quadrant).

Phase 1 (Proposal selection testing)

Begin: December 13, 2007

PDM Ballistic testing

 V50 Ballistic limit testing

 V0 Ambient testing

 V50 Environmental conditions

 (9 subtests)

Phase 2 (Testing to gain statistical confidence

and H FE data)

Begin: TBD

 V50 Select environmental conditions

 HFE/Suitability testing

Phase 1 Ballistic test plans approved by DOTE.

Phase 2 Ballistic test plans and suitability test

 plans are written in draft form.

Congressional interest: GAO and DOTE

 oversight.

Path forward:

Execute phase 1: Ballistic test.

Finalize phase 2: Ballistice and suitability test plans.

Test strategy/schedule

Issues/status

PM Soldier Equipment RFP for the following 

 Body Armor items:

ESAPI: Enhanced Small Arms Protective Insert

XSAPI: (X) Small Arms Protective Insert

FSAPV-E: Flexible Small Arms Protective 

 Vesg-Enhanced

FSAPV-X: Flexible Small Arms Protective Vest-X

RFP Closes December 12, 2007

Description

System picture

ESAPI—XSAPI—FSAPV-E—FSAPV-X

Body armor

It was not until after the back-face deformation deviation was discovered 

that briefing slides and other documentation on test plans and schedules 

started describing First Article Testing as following Preliminary Design 

Model testing. For example, as stated by DOD in its comments, the 

October 2008 briefing to a congressional member and professional staff 

clearly showed First Article Testing as following Preliminary Design Model 

testing (Phase 1) and preceding Phase 2. Therefore, it is not clear why 

DOD’s test plan briefings would make no mention of a First Article Testing 

prior to the back-face deformation measurement deviation while including 

First Article Testing in subsequent briefings if the plan had always been to 

conduct both Preliminary Design Model testing and First Article Testing. 

Furthermore, it is not clear why DOD would intentionally plan at the start 

of testing to repeat Preliminary Design Model testing (which was supposed 

Page 82 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

to be performed in accordance with the First Article Testing protocol) 

with an identical test (First Article Testing) given that it has been the 

Army’s practice to use such Preliminary Design Model testing to meet First 

Article Testing requirements – a practice that was also supported by the 

DOD Inspector General and the Army Acquisition Executive after an audit 

of the Army’s body armor testing program.5

 DOD also stated that First 

Article Testing waivers were not permitted under the body armor 

solicitation. However, the solicitation and its amendments are unclear as 

to whether waivers of First Article Testing would be permitted. 

Nonetheless, in written answers to questions we posed to the Army in July 

2008, the Army Test and Evaluation Command and PEO Soldier in a 

combined response stated that due to the fact that back-face deformation 

was not measured to the deepest point of penetration during Phase I tests, 

there would be no waivers of First Article Testing after the contract 

award. 

DOD also stated that it and the Army concluded that First Article Testing 

had achieved its objective of verifying that contracted vendors could 

produce, in a full-rate capacity, plates that had passed Preliminary Design 

Model testing. DOD further stated that it is incorrect to say that First 

Article Testing did not meet its objective and it is incorrect to assert that 

three of five vendor designs should have failed First Article Testing. 

However, our analysis showed that two solutions that passed First Article 

Testing would have failed if back-face deformations had not been rounded 

and had been scored as they were during Preliminary Design Model 

testing.6

 The third solution that passed would have failed if Army testers 

had correctly scored a shot result as a complete penetration in accordance 

with the definition of a complete penetration in the purchase description, 

rather than as a partial penetration. Because questions surround these 

scoring methods and because DOD and the Army cannot confidently 

identify whether these vendors can mass produce acceptable plates, we 

restate that First Article Testing may not have achieved its objective. See 

comments 12, 10, and 11 regarding DOD’s statements about the 

certification of the laser scanning equipment, the rounding of back-face 

deformations, and the Aberdeen Test Center’s scoring procedures, 

respectively. 

 

5

 DODIG Report No. D-2008-067, March 31, 2008, DOD Procurement Policy for Body Armor. 

6

 Depending on the type of design (i.e., ESAPI or XSAPI) a design can accumulate either 6 

or 10 penalty points before being eliminated from consideration. The designs in question 

were 1.0 point, or one penalty, away from failing. 

Page 83 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

We agree with DOD that an open dialog with the DOD Inspector General, 

external test and technology experts, and us will improve the current body 

armor testing. However, we disagree with DOD’s statement that NIJcertified laboratories lack expertise to provide reliable information on 

body armor testing issues. Before the current solicitation, the Army relied7

on these NIJ-certified laboratories for all body armor source selection and 

lot acceptance tests. The Marine Corps also conducts source selection 

tests at these facilities. As these independent laboratories have performed 

numerous tests for the Army conducted in accordance with First Article 

Testing protocol, we assert that the credentials of these laboratories 

warrant consideration of their opinions on body armor testing matters. 

9. DOD did not concur with our recommendation for an independent 

evaluation of First Article Testing results before any armor is fielded to 

soldiers because the First Article Testing achieved its objectives. We 

disagree with DOD’s position that First Article Testing and Preliminary 

Design Model testing achieved their objectives because we found 

numerous deviations from testing protocols that allowed solutions to pass 

testing that otherwise would have failed. Due to these deviations, the 

majority of which seem to make the testing easier to pass and favor the 

vendors, we continue to believe that it is necessary to have an independent 

external expert review the results of First Article Testing and the overall 

effect of DOD’s deviations on those results before the plates are fielded. 

An independent observer, external to DOD, is best suited to determine the 

overall impact of DOD’s many deviations during the testing associated 

with this solicitation. Consequently, we have added a matter for Congress 

to consider directing DOD to either conduct this external review or direct 

that DOD officially amend its testing protocols to reflect any revised test 

procedures and repeat First Article Testing. 

10. DOD did not concur with our recommendation that the practice of 

rounding down back-face deformations should be reviewed by external 

experts because the practice has been used historically by NIJ-certified 

laboratories. Although DOD acknowledged that the practice of rounding is 

not adequately described in the testing protocols, it stated that rounding is 

permitted under American Society for Testing and Materials (ASTM) E-29. 

The purchase descriptions (attachments 01 and 02 of the solicitation) 

referenced five ASTM documents, but ASTM E-29 is not referenced and 

 

7

 The law-enforcement community relies on NIJ-certified laboratories to conduct their body 

armor testing and ensure that their body armor meets law enforcement levels of protection. 

Page 84 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

therefore is not part of the protocol. The detailed test plans state that 

solutions shall incur a penalty on deformations greater than 43 

millimeters, and the Army is correct that neither the purchase description 

nor the detailed test plans provide for rounding. During Preliminary 

Design Model testing, Army testers measured back-face deformations to 

the hundredths place and did not round. Any deformation between 43.00 

and 43.50 received a penalty. During First Article Testing, deformations in 

this range were rounded down and did not incur a penalty, so the decision 

to round effectively changed the standard in favor of the vendors. Two 

solutions passed First Article Testing that would have failed if back-face 

deformations had been scored without rounding as they were during 

Preliminary Design Model testing. We recognize that there are other 

factors, such as the fact that the new laser scanner may overstate backface deformations that might justify the decision to round down back-face 

deformations. However, as a stand-alone event, rounding down 

deformations did change the standard in the middle of the solicitation 

between Preliminary Design Model testing and First Article Testing. That 

is why it is important for an independent external expert to review the 

totality of the test and the Army’s deviations from testing protocols to 

determine the actual effect of this and other deviations. 

11. Regarding the incorrect scoring of a complete penetration as a partial 

penetration, DOD stated that the first layer of soft armor behind the plate 

serves as a witness plate during testing. If that first layer of soft armor is 

not penetrated, as determined by the breaking of threads on that first layer 

of soft armor, the test shot is not scored as a complete penetration in 

accordance with the PEO Soldier’s scoring criteria. However, DOD’s 

position is not consistent with the established testing protocols as 

evidenced by the following: 

(1) We did not observe the use of and the testing protocols do not 

require the use of a witness plate during testing to determine if 

a penetration occurred; and 

(2) The testing protocols do not state that “the breaking of 

threads” is the criterion for determining a penetration. 

The language of the testing protocols, not undocumented criteria, should 

be used in scoring and determining penetration results. The criteria for 

scoring a penetration are found in the current solicitation’s protocols. 

Paragraph 6.6, of each of the purchase descriptions state, under 

“Definitions: Complete Penetration (CP) for Acceptance Testing--

Complete penetrations have occurred when the projectile, fragment of the 

Page 85 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

projectile, or fragment of the armor material is imbedded or passes into 

the soft under garment used behind the protective inserts plates” (ESAPIs 

or XSAPIs). 

Our multiple observations and thorough inspection of the soft armor in 

question revealed that black-grayish particles had penetrated at least three 

Kevlar layers as evidenced by their frayed, fuzz-like and separated 

appearance to the naked eye. The black-grayish particles were stopped by 

the fourth Kevlar layer. DOD acknowledged that figure 6 of our report 

appears to show evidence of a perforation on the rear of the test plate in 

question and that the Aberdeen Test Center’s subject matter expert found 

dust particles. These particles are fragments of the projectile or fragments 

of the armor material that were imbedded and indeed passed into the soft 

undergarment used behind the protective insert; therefore, the shot should 

have been ruled a complete penetration according to the testing protocols, 

increasing the point penalties and causing the design to fail First Article 

Testing. 

DOD’s comments stated that we acknowledged there were no broken 

threads on the first layer of the soft armor. We made no such comment 

and this consideration is not relevant as the requirement for broken fibers 

is not consistent with the written testing protocols as we have stated. Of 

consequence, DOD and Army officials acknowledged that the requirement 

for broken fibers was not described in the testing protocols or otherwise 

documented. 

In addition to the DOD acknowledgement that an Aberdeen Test Center 

subject matter expert found particles on the soft body armor, more 

convincing evidence is the picture of the subject plate. Figure 6 of our 

report clearly shows the tear in the fibers that were placed behind the 

plate in question allowing the penetration of the particles found by the 

Aberdeen Test Center subject matter expert. These particles can only be 

fragments of the projectile or fragments of the armor material that passed 

into the soft under garment used behind the protective inserts (plates), 

confirming our observations of the event and the subsequent incorrect 

scoring. The shot should have been scored a complete penetration, and the 

penalty incurred would have caused the design in question to fail First 

Article Testing. 

12. DOD did not concur with our recommendation that the use of the laser 

scanner needs to be reviewed by experts external to DOD due to the lack 

of a full evaluation of the scanner’s accuracy to measure back-face 

deformations, to include an evaluation of the software modifications and 

Page 86 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

operation under actual test conditions. DOD asserted that the laser 

scanner measurement device provides a superior tool for providing 

accurate, repeatable, defensible back-face deformation measurements to 

the deepest point of depression in the clay. We agree that once it is 

properly certified, tested, and evaluated, the laser may eliminate human 

errors such as incorrectly selecting the location of the deepest point or 

piercing the clay with the sharp edge of the caliper and making the 

depression deeper. However, as we stated, the Army used the laser 

scanner as a new method to measure back-face deformation without 

adequately certifying that the scanner could function: (1) in its operational 

environment, (2) at the required accuracy, (3) in conjunction with its 

software upgrades, and (4) without overstating deformation 

measurements. 

DOD asserted that the software upgrades did not affect the measurement 

system of the laser scanner and that these software changes had no effect 

on the physical measurement process of the back-face deformation 

measurement that was validated through the certification process. The 

software upgrades were added after the certification and do include 

functions8

 to purposely remove spikes and other small crevices on the clay 

and a smoothing algorithm that changed back-face deformation 

measurements. We have reviewed these software functions and they do in 

fact include calculations that change the back-face deformation 

measurement taken. Furthermore, Army officials told us that additional 

upgrades to the laser scanner were made after First Article Testing by 

Aberdeen Test Center to correct a software laser malfunction identified 

during the subsequent lot acceptance testing of its plates. According to 

these officials, this previously undetected error caused an overstatement 

of the back-face deformation measurement taken by several millimeters, 

calling into question all the measurements taken during First Article 

Testing. Also, vendors have told us that they have conducted several 

studies9

 that show that the laser scanner overestimates back-face 

deformation measurements by about 2 millimeters as compared with 

measurements taken by digital caliper, thereby over-penalizing vendors’ 

 

8

 Software upgrades were not part of the certification process. Some of these software 

upgrades eliminate the deepest point of depression measurement. 

9

 A vendor test showed an approximately 2-millimeter overstatement of back-face 

deformation measurements by the laser as compared to the caliper. 

Page 87 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

designs and causing them to fail lot acceptance testing.10 Furthermore, the 

laser scanner was certified to an accuracy of 1.0 millimeters, but section 

4.9.9.3 of the purchase descriptions requires a device capable of measuring 

to an accuracy of ±0.1 millimeters. Therefore, the laser does not meet this 

requirement making the certification invalid. The laser scanner is an 

unproven measuring device that may reflect a new requirement because 

the back-face deformation standards are based on measurements obtained 

with a digital caliper. This raises concerns that results obtained using the 

laser scanner may be more inconsistent than those obtained using the 

digital caliper. As we stated in the report, the Aberdeen Test Center has 

not conducted a side-by-side test of the new laser scanner used during 

First Article Testing and the digital caliper previously used during 

Preliminary Design Model testing. Given the discrepancies on back-face 

deformation measurements we observed and the overstating of the backface deformation alleged by the vendors, the use of the laser is still called 

into question. Thus, we continue to support our recommendation that 

experts independent of DOD review the use of the laser during First 

Article Testing and that a full evaluation of the laser scanner is imperative 

to ensure that the tests are repeatable and can be relied upon to certify 

procurement of armor plates for our military personnel based on results of 

body armor testing at the Aberdeen Test Center using the laser scanner. 

Lastly, DOD stated that the laser scanner is used by the aeronautical 

industry; however, the Army Test and Evaluation Command officials told 

us that the scanner had to be customized for testing through various 

software additions and mounting customizations to mitigate vibrations and 

other environmental factors. These software additions and customizations 

change the operation of the scanner. 

13. DOD does not concur with our recommendation that experts examine, 

among other items, “the exposure of clay backing material to rain and 

other outside environmental conditions as well as the effect of high oven 

temperatures during storage and conditioning,” because it believes that 

 

10 Lot acceptance testing provides additional ballistic testing that ensures that the plates 

delivered meet requirements before they are accepted. Two vendors whose designs passed 

Preliminary Design Model testing and First Article Testing have failed lot acceptance 

testing and in July 2009 submitted to the Army, in one case, a ruling on a request for 

equitable adjustment and, in another case, a request to waive contract penalties for late 

deliveries. These vendors have failed several lot acceptance tests involving tens of 

thousands of plates that have been rejected by the government because they failed this 

testing. One vendor is asking for several millions of dollars in payment to compensate for 

material, labor, and delays as a result of the failed lots. 

Page 88 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

such conditions had no impact upon First Article Testing results. As 

detailed in the report, we observed these conditions at different points 

throughout the testing period. Major variations in materials preparation 

and testing conditions such as exposure to rain and/or violations of testing 

protocols merit consideration when analyzing the effectiveness and 

reliability of First Article Testing. 

As one specific example, we described in this report statistically 

significant differences between the rates of failure in response to one 

threat on November 13 and the failure rates on all other days of testing but 

do not use the statistical analysis as the definitive causal explanation for 

such failure. We observed one major environmental difference in testing 

conditions that day, the exposure of temperature-conditioned clay to 

heavy, cold rain in transit to the testing site. After experts confirmed that 

such variation might be one potential factor relating to overall failure rates 

on that day, we conducted statistical tests to assess whether failures rates 

were different on November 13 compared to other dates.. 

Our assertion that the exposure of the clay to rain may have had an impact 

on test results is based not solely on our statistical analysis of test results 

that day; rather, it is also based on our conversations with industry 

experts, including the clay manufacturer, and on the fact that we 

witnessed an unusually high number of clay calibration failures during 

testing that comprised plate designs of multiple vendors, not just the one 

design that DOD points to as the source for the high failure rate. We 

observed that the clay conditioning trailer was located approximately 25 

feet away from the entrance to the firing lane. The clay blocks, weighing in 

excess of 200 lbs., were loaded face up onto a cart and then a single 

individual pulled the cart over approximately 25 feet of gravel to the firing 

lane entrance. Once there, entry was delayed because the cart had to be 

positioned just right to get through the firing lane door. Army testers 

performed all of this without covering the clay11 to protect it from the rain 

and the cold, and once inside the clay had significant amounts of water 

collected on it. 

With respect to the unusually high number of clay calibration failures on 

November 13, there were seven clay calibration drops that were not within 

 

11According to Army officials, during subsequent lot acceptance testing tests, Aberdeen 

Test Center technicians were covering the clay boxes during transport from the 

conditioning ovens to the lanes. 

Page 89 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

specifications. Some of these failed clay boxes were discarded in 

accordance with the testing protocols; however, others were repaired, redropped, and used if they had passed the second drop series. These 

included one plate that was later ruled a no-test and three plates for which 

the first shot yielded a catastrophic back-face deformation. These were the 

only three first-shot catastrophic back-face deformations during the whole 

test, and they all occurred on the same rainy day and involved two 

different solutions, not just the one that DOD claims performed poorly. 

The failure rates of plates as a whole, across all plate designs, were very 

high this day, and the failures were of both the complete penetration and 

the back-face deformation variety. Water conducts heat approximately 25 

times faster than air, which means the water on the surface cooled the clay 

considerably faster than the clay would have cooled by air exposure alone. 

Moreover, Army testers lowered the temperature of the clay conditioning 

trailers during testing on November 13 and told us that the reason was that 

the ovens and clay were too hot. This is consistent with what Army 

subject matter experts and other industry experts told us— that the 

theoretical effect of having cold rain collecting on hot clay may create a 

situation where the clay is more susceptible to both complete penetrations 

because of the colder, harder top layer and to excessive back-face 

deformations because of the overheated, softer clay beneath the top layer. 

Finally, the clay manufacturer told us that, although this is an oil-based 

clay, water can affect the bonding properties of the clay, making it more 

difficult for wet clay to stick together. This is consistent with what we 

observed on November 13. After the first shot on one plate, as Army 

testers were removing the plate from the clay in order to determine the 

shot result, we observed a large chunk of clay fall to the floor. This clay 

was simply swept off to the side by the testers. In another instance, as 

testers were repairing the clay after the calibration drop, one of the testers 

pulled a long blade over the surface of the clay to smooth it. When he hit 

the spot where one of the calibration drops had occurred and the clay had 

been repaired, the blade pulled up the entire divot and the testers had to 

repair the clay further. 

Regarding our use of no-test data, we were strict in the instances where 

we used this data, see our comment 24. 

DOD stated that it was the poor performance of one solution in particular 

that skewed the results for this day and that this solution failed 70 percent 

of its shots against Threat D during First Article Testing. DOD’s statistic is 

misleading. This solution failed 100 percent of its shots (6 of 6) on 

November 13, but only 50 percent for all other test days (7 of 14). Also, the 

Page 90 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

fact that this solution managed to pass the Preliminary Design Model 

testing but performed so poorly during First Article Testing raises 

questions about the repeatability of DOD’s and the Army’s test practices. 

Finally, DOD’s own analysis confirms that two of the four solutions tested 

on November 13 performed at their worst level in the test on that day. If 

the one solution whose plate was questionably ruled a no-test on this day 

is included in the data, then three of the four solutions performed at their 

worst level in the test on this day. 

DOD said that after testing Aberdeen Test Center completed the planned 

installation of new clay conditioning chambers inside the test ranges 

precluding any external environmental conditioning interacting with the 

clay. We believe it is a step in the right direction that the Aberdeen Test 

Center has corrected this problem for future testing, but we continue to 

believe that an external entity needs to evaluate the impact of introducing 

this new independent variable on this day of First Article Testing. 

14. DOD concurred that it should establish a written standard for 

conducting clay calibration drops but non-concurred that failed blocks 

were used during testing. DOD asserted that all clay backing material used 

during testing passed the calibration drop test prior to use. We disagree 

with this position because the calibration of the clay required by the 

testing protocols calls for “a series of drops,” meaning one series of three 

drops, not multiple series of three drops as we observed on various 

occasions. DOD stated that, as a result of our review and the concerns 

cited in our report, the Aberdeen Test Center established and documented 

a revised procedure stating that only one repeat of calibration attempt can 

be made and, if the clay does not pass calibration upon the second 

attempt, it is reconditioned for later use and a new block of clay is 

substituted for calibration. Based on the testing protocols, this is still an 

incorrect procedure to ensure the proper calibration of the clay prior to 

shooting. The testing protocols do not allow for a repeat series of 

calibration drops. DOD also says that, upon completion of testing under 

the current Army solicitation and in coordination with the National 

Institute of Standards and Technology, the office of the Director of 

Operational Test and Evaluation and the Army will review the procedures 

for clay calibration to include repeated calibration attempts and will 

document any appropriate procedural changes. DOD goes on to say that 

the NIJ standard as verified by personnel at the National Institute of 

Standards and Technology does not address specifically the issue of 

repeating clay calibration tests. However, the Aberdeen Test Center’s 

application of the Army’s current solicitation’s protocols during testing, 

and not the NIJ standards, was the subject of our review. 

Page 91 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

In its comments, DOD acknowledged that the National Institute of 

Standards and Technology officials recommend only one series of drops 

for clay calibration, but the Aberdeen Test Center did multiple drops 

during testing. We are pleased that DOD has agreed to partner with the 

National Institute of Standards and Technology to conduct experiments to 

improve the testing community’s understanding of clay performance in 

ballistic testing, but these conversations and studies in our opinion should 

have occurred prior to testing, not after, as this deviation from testing 

protocols calls the tests results into question. We reassert that an external 

entity needs to evaluate the impact of this practice on First Article Testing 

results. 

15. DOD partially concurred with our recommendation and agreed that 

inconsistencies were identified during testing; however, DOD asserted that 

the identified inconsistencies did not alter the test results. As stated in our 

response to DOD’s comments on our first recommendation, we do not 

agree. Our observations clearly show that (1) had the deepest point been 

used during Preliminary Design Model testing, two designs that passed 

would have failed and (2) had the Army not rounded First Article Testing 

results down, two designs that passed would have failed. Further, if the 

Army had scored the particles (which in their comments to this report 

DOD acknowledges were imbedded in the shoot pack behind the body 

armor) according to the testing protocols, a third design that passed First 

Article Testing would have failed. In all, four out of the five designs that 

passed Preliminary Design Model testing and First Article Testing would 

have failed if testing protocols had been followed. 

16. DOD partially concurred with our recommendation that, based on the 

results of the independent expert review of the First Article Testing 

results, it should evaluate and recertify the accuracy of the laser scanner 

to the correct standard with all software modifications incorporated and 

include in this analysis a side-by-side comparison of the laser 

measurements of the actual back-face deformations with those taken by 

digital caliper to determine whether laser measurements can meet the 

standard of the testing protocols. DOD maintains that it performed an 

independent certification of the laser measurement system and process 

and that the software changes that occurred did not affect the 

measurement system in the laser scanner. However, as discussed in 

comment 12, we do not agree that an adequate, independent certification 

of the laser measurement system and process was conducted. Based on 

our observations, we continue to assert that the software changes added 

after certification did affect the measurement system in the laser. 

Page 92 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

17. DOD partially concurred with our recommendation for the Secretary of 

the Army to provide for an independent peer review of the Aberdeen Test 

Center’s body armor testing protocols, facilities, and instrumentation. We 

agree that a review conducted by a panel of external experts that also 

includes DOD members could satisfy our recommendation. However, to 

maintain the independence of this panel, the DOD members should not be 

composed of personnel from those organizations involved in the body 

armor testing (such as the office of the Director of Operational Test and 

Evaluation, the Army Test and Evaluation Command, or PEO Soldier.12

18. DOD stated that Aberdeen Test Center had been extensively involved 

in body armor testing since the 1990s and has performed several tests of 

body armor plates. We acknowledge that Aberdeen Test Center had 

conducted limited body armor testing for the initial testing on the 

Interceptor Body Armor system in the 1990s and have clarified the report 

to reflect that. However, as acknowledged by DOD, Aberdeen Test Center 

did not perform any additional testing on that system for PEO Soldier 

since the 1990s and this lack of experience in conducting source selection 

testing for that system may have led to the misinterpretations of testing 

protocols and deviations noted on our report. According to a recent Army 

Audit Agency report,13 NIJ testing facilities conducted First Article Testing 

and lot acceptance testing for the Interceptor Body Armor system prior to 

this current solicitation. Another reason Aberdeen Test Center could not 

conduct source selection testing was that in the past Aberdeen Test Center 

lacked a capability for the production testing of personnel armor systems 

in a cost-effective manner; the test facilities were old and could not 

support test requirements for a temperature- and humidity-controlled 

environment and could not provide enough capacity to support a war-

 

12 The Army Test and Evaluation Command performed these tests and the office of the 

Director of Operational Test and Evaluation provided oversight of Preliminary Design 

Model testing and First Article Testing for the current solicitation to include determining 

the scope of testing required and approving the test plans. PEO Soldier provided subject 

matter experts to advise Army testers, developed the purchase descriptions, and approved 

test plans. Therefore, these entities are part of the program that needs to be reviewed and 

are not independent. Additionally, any other individuals and organizations associated with 

the Preliminary Design Model testing, First Article Testing, or lot acceptance testing should 

also be excluded. 

13 U.S. Army Audit Agency, Body Armor Testing PEO Soldier; Audit Report: A-2009-0086-

ALA, 30 March 2009—Just before the current solicitation, from January 2007 to June 2008, 

all 27 Army First Article Testing for new designs associated with ESAPIs (four vendors), 

their associated 1,024 lot acceptance quality assurance ballistic testing, and the long-term 

environmental conditions testing were all performed in an independent NIJ-certified testing 

facility. 

Page 93 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

related workload. The Army has spent about $10 million over the last few 

years upgrading the existing facilities with state-of-the-art capability to 

support research and development and production qualification testing for 

body armor, according to the Army Audit Agency. Army Test and 

Evaluation Command notes that there were several other tests between 

1997 and 2007, but according to Army officials these tests were customer 

tests not performed in accordance with a First Article Testing protocol. 

For example, the U.S. Special Operations Command test completed in May 

2007 and cited by DOD was a customer test not in accordance with First 

Article Testing protocol. The Aberdeen Test Center built new lanes and 

hired and trained contractors to perform the Preliminary Design Model 

testing and First Article Testing. 

19. DOD stated that, to date, it has obligated about $120 million for XSAPI 

and less than $2 million for ESAPI. However, the value of the 5-year 

indefinite delivery/indefinite quantity contracts we cited is based on the 

maximum amount of orders of ESAPI/XSAPI plates that can be purchased 

under these contracts. Given that the Army has fulfilled the minimum 

order requirements for this solicitation, the Army could decide to not 

purchase additional armor based on this solicitation and not incur almost 

$7.9 billion in costs. 

DOD stated in its response that there are only three contracts. However, 

the Army Contracting Office told us that there were four contracts 

awarded and provided those contracts to us for our review. Additionally, 

we witnessed four vendors participating in First Article Testing, all of 

which had to receive contracts to participate. It is unclear why the Army 

stated that there were only three contracts. 

20. DOD is correct that there is no limit or range specified for the second 

shot location for the impact subtest. However, this only reinforces that the 

shot should have been aimed at 1.5 inches, not at 1.0 inch or at various 

points between 1.0 inch and 1.5 inches. It also does not explain why the 

Army continued to mark plates as though there were a range for this shot. 

Army testers would draw lines at approximately 0.75 inches for the inner 

tolerance and 1.25 inches for the outer tolerance of ESAPI plates. They 

drew lines at approximately 1.0 inch for the inner tolerance and 1.5 inches 

for the outer tolerance of XSAPI plates. We measured these lines for every 

impact test plate and also had Army testers measure some of these lines to 

Page 94 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

confirm our measurements. We found that of 56 test items,14 17 were 

marked with shot ranges wholly inside of 1.5 inches. The ranges of 30 

other test items did include 1.5 inches somewhere in the range, but the 

center of the range (where Army testers aimed the shot) was still inside of 

1.5 inches. Only four test items were marked with ranges centered on 1.5 

inches. 

DOD may be incorrect in stating that shooting closer to the edge would 

have increased the risk of a failure for this subtest. For most subtests this 

may be the case, but according to Army subject matter experts the impact 

test is different. For the impact test, the plate is dropped onto a concrete 

surface, striking the crown (center) of the plate. The test is to determine if 

this weakens the structural integrity of the plate, which could involve 

various cracks spreading from the center of the plate outward. The reason 

the requirement for this shot on this subtest is written differently (i.e., to 

be shot at approximately 1.5 inches from the edge, as opposed to within a 

range between 0.75 inches and 1.25 inches or between 1.0 inches and 1.5 

inches on other subtests) is that it is meant to test the impact’s effect on 

the plate. For this subtest and this shot, there may actually be a higher risk 

of failure the closer to the center the shot occurs. PEO Soldier 

representatives acknowledged that the purchase descriptions should have 

been written more clearly and changed the requirement for this shot to a 

range of between 1.5 inches and 2.25 inches during First Article Testing. 

We confirmed that Army testers correctly followed shot location testing 

protocols during First Article Testing by double-checking the 

measurements on the firing lane prior to the shooting of the plate. We also 

note that, although DOD stated the Preliminary Design Model testing shot 

locations for the impact test complied with the language of the testing 

protocols, under the revised protocol used during First Article Testing 

several of these Preliminary Design Model testing impact test shot 

locations would not have been valid. DOD stated that there was no impact 

on the outcome of the test, but DOD cannot say that definitively. Because 

shooting closer to the edge may have favored the vendors in this case, the 

impact could have been that a solution or solutions may have passed that 

should not have. 

21. The Army stated that “V50 subtests for more robust threats...were 

executed to the standard protocols.” Our observations and analysis of the 

 

14 Nine test items were not able to be measured due either to the absence of lines or due to 

damage caused by the impact of the ballistic. 

Page 95 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

data show that this statement is incorrect. Sections 2.2.3.h(2) of the 

detailed test plans state: 

“If the first round fired yields a complete penetration, the 

propellant charge for the second round shall be equal to that of the 

actual velocity obtained on the first round minus a propellant 

decrement for 100 ft/s (30 m/s) velocity decrease in order to obtain 

a partial penetration. If the first round fired yields a partial 

penetration, the propellant charge for the second round shall be 

equal to that of the actual velocity obtained on the first round plus 

a propellant increment for a 50 ft/s (15 m/s) velocity increase in 

order to obtain a complete penetration. A propellant increment or 

decrement, as applicable, at 50 ft/s (15 m/s) from actual velocity of 

last shot shall be used until one partial and one complete 

penetration is obtained. After obtaining a partial and a complete 

penetration, the propellant increment or decrement for 50 ft/s (15 

m/s) shall be used from the actual velocity of the previous shot.” 

V50 testing is conducted to discern the velocity at which 50 percent of the 

shots of a particular threat would penetrate each of the body armor 

designs. The testing protocols require that, after every shot that is defeated 

by the body armor, the velocity of the next shot be increased. Whenever a 

shot penetrates the armor, the velocity should be decreased for the next 

shot. This increasing and decreasing of the velocities is supposed to be 

repeated until testers determine the velocity at which 50 percent of the 

shots will penetrate. In cases in which the armor far exceeds the V50 

requirement and is able to defeat the threat for the first six shots, the 

testing may be halted without discerning the V50 for the plate and the 

plate may be ruled as passing the requirements. During Preliminary Design 

Model V50 testing, Army testers would achieve three partial penetrations 

and then continue to shoot at approximately the same velocity, or lower, 

for shots 4, 5, and 6 in order to intentionally achieve six partial 

penetrations. Army testers told us that they did this to conserve plates. 

According to the testing protocols, Army testers should have continued to 

increase the charge weight in order to try to achieve a complete 

penetration and determine a V50 velocity. The effect of this methodology 

was that solutions were treated inconsistently. Army officials told us that 

this practice had no effect on which designs passed or failed, which we do 

not dispute in our report; however, this practice made it impossible to 

discern the true V50s for these designs based on the results of Preliminary 

Design Model testing. 

Page 96 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

22. DOD agreed that Army testers deviated from the testing protocols by 

measuring back-face deformation at the point of aim. DOD stated that this 

decision was made by Army leadership in consultation with the office of 

the Director of Operational Test and Evaluation, because this would not 

disadvantage any vendor. We agree with DOD that this decision was made 

by Army leadership in consultation with the office of the Director of 

Operational Test and Evaluation. We did not independently assess all 

factors being considered by Army leadership when they made the decision 

to overrule the Integrated Product Team and the Milestone Decision 

Authority’s initial decision to measure to the deepest point. 

DOD also stated that measuring back-face deformation at the point of aim 

is an accurate and repeatable process. As we pointed out in our previous 

responses, DOD’s own comments regarding DOD’s Assertion 3 contradict 

this statement where DOD writes that there were “potential variances 

between the actual aim point and impact point during testing.” 

Furthermore, we observed that the aim laser used by Army testers was 

routinely out of line with where the ballistic was penetrating the yaw 

card,15 despite continued adjustments to line up the aim laser with where 

the ballistic was actually traveling. 

DOD stated that it is not possible to know the reference point on a curved 

object when the deepest deformation point is laterally offset from the aim 

point. We disagree. DOD acknowledges in its response that PEO Soldier 

had an internally documented process to account for plate curvature when 

the deepest point of deformation was laterally offset from the point of aim. 

The use of correction factor tables is a well-known industry standard that 

has been in place for years, and this standard practice has been used by 

NIJ laboratories and is well-known by vendors. 

DOD and the Army presented several statistics on the difference between 

aim point back-face deformation and deepest point back-face deformation 

in testing and stated that the difference between the two is small. We do 

not agree with DOD’s assertion that a difference of 10.66 millimeters is 

small. In the case of Preliminary Design Model testing, the difference 

between measuring at the aim point and at the deepest point was that at 

least two solutions passed Preliminary Design Model testing that 

 

15The yaw card is a piece of paper placed in the intended path of the ballistic and is meant 

to measure the amount of yaw, or wobble, of the ballistic as it travels through the air. We 

observed that the hole made by the bullet in the yaw card was routinely not in line with 

where the aim laser was pointing. 

Page 97 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

otherwise would have failed. These designs passed subsequent First 

Article Testing but have gone on to fail lot acceptance testing, raising 

additional questions regarding the repeatability of the Aberdeen Test 

Center’s testing practices. 

DOD asserts that the adoption of the laser scanner measurement 

technique resolves the problems the Army experienced in measuring backface deformations completely. We would agree that the laser scanner has 

the potential to be a useful device but when used in the manner in which 

Aberdeen Test Center used it – without an adequate certification and 

without a thorough understanding of how the laser scanner might 

effectively change the standard for a solution to pass – we do not agree 

that it resolved back-face deformation measurement issues. Aberdeen Test 

Center officials told us that they did not know what the accuracy of the 

laser scanner was as it was used during First Article Testing. 

23. DOD acknowledged the shortcoming we identified. DOD then asserted 

that once the deviation of measuring back-face deformation at the point of 

aim, rather than at the deepest point of depression was identified, those 

involved acted decisively to resolve the issue. We disagree based on the 

timeline of events described in our response to DOD’s comments on 

Preliminary Design Model testing, as well as on the following facts. We 

were present and observed the Integrated Product Team meeting on 

March 25 and observed that all members of the Integrated Product Team 

agreed to start measuring immediately at the deepest point, to score 

solutions based on this deepest point data, to conserve plates, and then at 

the end of the testing to make up the tests incorrectly performed during 

the first third of testing, as needed. We observed Army testers implement 

this plan the following day. Then, on March 27, Army leadership halted 

testing for 2 weeks, considered the issue, and then reversed the 

unanimous decision by the Integrated Product Team and decided to score 

to the point of aim. 

The deviation of scoring solutions based on the back-face deformation at 

the point of aim created a situation in which the Army could not have 

confidence in any solution that passed the Preliminary Design Model 

testing. Because of this, the Army had to repeat testing, in the form of First 

Article Testing, to determine whether the solutions that had passed 

Preliminary Design Model testing actually met requirements. 

24. DOD did not concur with our finding that rain may have impacted the 

test results. DOD stated that such conditions had no impact upon First 

Article Testing results. Our statistical analysis of the test data shows 

Page 98 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

failure rates to be significantly higher on November 13 than during other 

days of testing, and our observations taken during that day of testing and 

our conversations with industry experts familiar with the clay, including 

the clay manufacturer, suggest the exposure of the clay to the cold, heavy 

rain on that day may have been the cause of the high failure rates. Our 

analysis examined the 83 plates tested against the most potent threat, 

Threat D. The testing protocols required that two shots for the record be 

taken on each plate. We performed a separate analysis for the 83 first 

shots taken on these plates from the 83 second shots taken on the plates. 

These confirmed statistically that the rate of failure on November 13 was 

significantly higher than the rate of failure on other days. Further, of the 5 

plates that experienced first-shot catastrophic failures during testing, 3 of 

them (60 percent) were tested on November 13 and all 3 of these were due 

to excessive back-face deformation. Given that only 9 plates were tested 

on November 13, while 74 were tested during all the other days of testing 

combined, it is remarkable that 60 percent of all catastrophic failures 

occurred on that one day of testing. 

DOD objected to our inclusion of no-test data in its calculation of first- and 

second-shot failure rates on November 13. We believe that the inclusion of 

no-test data is warranted because the Army’s exclusion of such plates was 

made on a post hoc basis after the shots were initially recorded as valid 

shots and because the rationale for determining the need for a re-test was 

not always clear. Additionally, we conducted an analysis excluding the notest plates identified by DOD and that analysis again showed that the 

failure rate on November 13 was statistically higher than during the other 

days of testing, even after the exclusions. Excluding the no-test plates, 38 

percent of first shots on November 13 (3 of 8) and 88 percent of second 

shots (7 of 8) failed. 

In its response, DOD reports that Aberdeen Test Center’s own statistical 

analysis of test data for Threat D reveals that the observed failure rate on 

November 13 is attributable to the “poor performance” of one design 

throughout testing. DOD asserts that its illustration indicates that “Design 

K was the weakest design on all days with no rain as well as days with 

rain.” DOD’s data do not support such a claim. As we have observed, 

excluding no-test plates, DOD’s data are based on 10 tests of two shots 

each for each of 8 designs (160 cases total). Each shot is treated as an 

independent trial, an assumption we find tenuous given that a plate’s 

structural integrity might be affected by the first shot. To account for date, 

DOD subdivides the data into cell sizes far too small to derive reliable 

statistical inferences about failure rates (between 2 and 6 shots per cell), 

Page 99 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

as evidenced by the wide confidence intervals illustrated in DOD’s visual 

representation of its analysis. 

Among evidence DOD presented to support its claim that Design K was the 

weakest performing design on both November 13 and other days is failure 

rate data for four designs that were not tested on the day in question. For 

two of the three designs tested on November 13 there were only one or 

two plates tested on November 13, far too few to conduct reliable 

statistical tests on differences in design performance. For the other type of 

plate tested on that day (Design L), the three plates tested had a markedly 

higher failure rate (3 of 6 shots, or 50 percent) on that day than on other 

days (when it had, in 14 shots, 5 failures, or a 36 percent failure rate). 

Design K had a failure rate of 6 of 6 shots (100 percent) on the day in 

question, compared with 8 of 14 shots (57 percent)16 on other days. 

Overall, it is impossible to determine from such a small set of tests 

whether the lack of statistical significance between different designs’ 

failure rates on November 13 and other days results from small sample 

size or a substantive difference in performance. 

Overall, the Army Test and Evaluation Command’s design-based analysis 

cannot distinguish between the potential effects of date and design on 

failure rates because sufficient comparison data do not exist to conduct 

the kind of multivariate analysis that might resolve this issue. Because the 

data alone are inadequate for distinguishing between the potential effects 

of date and design, we continue to recommend that independent experts 

evaluate the potential effects of variations in materials preparation and 

testing conditions, including those occurring on November 13, on overall 

First Article Testing results. 

Additionally, DOD stated that the clay is largely impervious to water. 

However, as stated in our report, body armor testers from NIJ-certified 

private laboratories, Army officials experienced in the testing of body 

armor, body armor manufacturers, and the manufacturer of the clay used 

told us that getting water on the clay backing material could cause a 

chemical bonding change on the clay’s surface. 

 

16 According to official test data, only 7 of these 14 shots were failures (50 percent). This is 

due to the Army’s practice of incorrectly rounding down back-face deformations during 

First Article Testing. One shot that resulted in a back-face deformation of 43.306 was 

officially rounded down to 43 and not penalized, but had Army testers followed the 

protocols and not rounded this result down, 8 of the 14 shots would have resulted in 

penalties. 

Page 100 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

DOD stated that one of its first actions when bringing in the clay is to 

scrape the top of the clay to level it. However, this only removes clay that 

is above the metal edge of the box. Clay that is already at or below the 

edge of the box is not removed by this scraping. We witnessed several 

instances in which the blade would remove clay at some points, but leave 

large portions of the clay surface untouched because the clay was below 

the edge of the box. 

25. See comment 11. 

26. The DOD is correct that the one particular example regarding deleting 

official test data only happened once. Fortunately, the results of the retest 

were the same as the initial test. After we noted this deficiency, Army 

officials told us that a new software program was being added that would 

prevent this from occurring again. DOD also stated that only two persons 

are authorized and able to modify the laser scanner software. We did not 

verify this statement; however, we assert that DOD needs to have an 

auditable trail when any such modifications are made and that it should 

require supervisory review and documentation or logging of these setting 

changes. 

27. DOD acknowledged that the Army did not formally document 

significant procedure changes that deviated from established testing 

protocols or assess the impact of these deviations. 

28. In our report we stated that the requirement to test at an NIJ-certified 

laboratory was withdrawn because the Aberdeen Test Center is not NIJcertified. DOD’s comments on this point do not dispute our statement. 

Instead, DOD discussed NIJ certification and stated that it does not believe 

that NIJ certification is appropriate for its test facilities. However, we did 

not recommend that any DOD test facilities be NIJ-certified or even that 

NIJ be the outside organization to provide an independent review of the 

testing practices at Aberdeen Test Center that we did recommend. 

However, we believe NIJ certification would meet our recommendation 

for an independent review. 

Regarding DOD’s comments regarding NIJ certification, DOD asserted that 

NIJ certification is not appropriate for its test facilities and asserted that 

there are significant differences between NIJ and U.S. Army body armor 

test requirements. NIJ certification of a test laboratory and NIJ protocol 

Page 101 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

for testing personal body armor primarily used by law enforcement 

officers are two distinct and different issues. Similar to a consumer United 

Laboratories laboratory certification, an NIJ laboratory certification17

includes an independent peer review of internal control procedures, 

management practices, and laboratory practices. This independent peer 

review is conducted to ensure that there are no conflicts of interest, and 

that the equipment utilized in the laboratory is safe and reliable. This peer 

review helps to ensure a reliable, repeatable, and accurate test, regardless 

of whether the test in question is following a U.S. Army testing protocol or 

a law enforcement testing protocol. NIJ-certified laboratories have 

consistently proven to be capable of following an Army testing protocol, 

which is demonstrated by the fact that NIJ-certified laboratories have 

conducted previous U.S. Army body armor source selection testing in 

accordance with First Article Testing protocol, as well as lot acceptance 

tests. The slide DOD included in its comments is not applicable here 

because it deals with the difference between testing protocols – the 

protocols for Army Interceptor Body Armor tests and the NIJ protocol for 

testing personal body armor primarily used by law enforcement officers. 

NIJ certification of a laboratory and NIJ certification of body armor for 

law enforcement purposes are two different things. 

29. DOD stated that we were incorrect in asserting that the Army decided 

to rebuild small arms ballistics testing facilities at Aberdeen Test Center 

after the 2007 House Armed Services Committee hearing. Instead, DOD 

stated that the contract to construct additional test ranges at the Aberdeen 

Test Center Light Armor Range was awarded in September 2006 and that 

construction was already underway at the time of June 2007 hearing. DOD 

also stated that this upgrade was not in response to any particular event 

but was undertaken to meet projected future Army ballistic test 

requirements. Army officials we spoke with before testing for this 

solicitation told us that this construction was being completed in order to 

perform the testing we observed. As of July 2007, the Light Armor Range 

included two pre-WWII era ballistic lanes and four modern lanes partially 

completed. However, we noted that, as of July 2007, the lanes we visited 

were empty and that none of the testing equipment was installed; only the 

buildings were completed. 

 

17 The U.S. Department of Justice offers this multi-departmental voluntary compliance 

program. 

Page 102 GAO-10-119 Warfighter Support
Appendix II: Comments from the Department 

of Defense 

In addition to the physical rebuilding of the test sites, the Amy also re-built 

its workforce to be able to conduct the testing. As stated on page 4 of 

DOD’s comments, PEO Soldier has instituted an effort to transfer testing 

expertise and experience from PEO Soldier to the Army Test and 

Evaluation Command. Prior to the start of testing we observed that 

Aberdeen Test Center hired, transferred in, and contracted for workers to 

conduct the testing. These workers were then trained by Aberdeen Test 

Center and conducted pilot tests in order to learn how to conduct body 

armor testing. We observed parts of this training, in person, and other 

parts via recorded video. In addition, we spoke with officials during this 

training and preparation process. From our observations and discussions 

with Army testers and PEO Soldier officials, we believe this process to 

have been a restarting of small arms ballistic testing capabilities at 

Aberdeen Test Center. Based on DOD’s comments, we clarified our report 

to reflect this information. 

Page 103 GAO-10-119 Warfighter Support
Appendix III: GAO

A

 Contact and Staff 

cknowledgments 

Page 104 GAO-10-119 

Appendix III: GAO Contact and Staff 

Acknowledgments 

William M. Solis, (202) 512-8365 

In addition to the contact named above, key contributors to this report 

were Cary Russell, Assistant Director; Michael Aiken; Gary Bianchi; 

Beverly Breen; Paul Desaulniers; Alfonso Garcia; William Graveline; Mae 

Jones; Christopher Miller; Anna Maria Ortiz; Danny Owens; Madhav 

Panwar; Terry Richardson; Michael Shaughnessy; Doug Sloane; Matthew 

Spiers; Karen Thornton; and John Van Schaik. 

Warfighter Support 

GAO Contact 

Acknowledgments 

(351282) 
GAO’s Mission The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help 

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost 

is through GAO’s Web site (www.gao.gov). Each weekday afternoon, GAO 

posts on its Web site newly released reports, testimony, and 

correspondence. To have GAO e-mail you a list of newly posted products, 

go to www.gao.gov and select “E-mail Updates.” 

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Phone The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s Web site, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information. 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Ralph Dawn, Managing Director, dawnr@gao.gov, (202) 512-4400 

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, DC 20548 

To Report Fraud, 

Waste, and Abuse in 

Federal Programs 

Congressional 

Relations 

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548 

Public Affairs 

Please Print on Recycled Paper