Warfighter Support: Independent Expert Assessment of Army Body Armor Test Results and Procedures Needed Before Fielding
Published: 2009-10-16T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Army has issued soldiers in Iraq and Afghanistan personal body armor, comprising an outer protective vest and ceramic plate inserts. GAO observed Preliminary Design Model testing of new plate designs, which resulted in the Army's awarding contracts in September 2008 valued at a total of over $8 billion to vendors of the designs that passed that testing. Between November and December 2008, the Army conducted further testing, called First Article Testing, on these designs. GAO is reporting on the degree to which the Army followed its established testing protocols during these two tests. GAO did not provide an expert ballistics evaluation of the results of testing. GAO, using a structured, GAO-developed data collection instrument, observed both tests at the Army's Aberdeen Test Center, analyzed data, and interviewed agency and industry officials to evaluate observed deviations from testing protocols. However, independent ballistics testing expertise is needed to determine the full effect of these deviations.
section_title: What GAO Found
paragraphs: During Preliminary Design Model testing the Army took significant steps to run a controlled test and maintain consistency throughout the process, but the Army did not always follow established testing protocols and, as a result, did not achieve its intended test objective of determining as a basis for awarding contracts which designs met performance requirements. In the most consequential of the Army's deviations from testing protocols, the Army testers incorrectly measured the amount of force absorbed by the plate designs by measuring back-face deformation in the clay backing at the point of aim rather than at the deepest point of depression. Army testers recognized the error after completing about a third of the test and then changed the test plan to call for measuring at the point of aim and likewise issued a modification to the contract solicitation. At least two of the eight designs that passed Preliminary Design Model testing and were awarded contracts would have failed if measurements had been made to the deepest point of depression. The deviations from the testing protocols were the result of Aberdeen Test Center's incorrectly interpreting the testing protocols. In all these cases of deviations from the testing protocols, the Aberdeen Test Center's implemented procedures were not reviewed or approved by the Army and Department of Defense officials responsible for approving the testing protocols. After concerns were raised regarding the Preliminary Design Model testing, the decision was made not to field any of the plate designs awarded contracts until after First Article Testing was conducted. During First Article Testing, the Army addressed some of the problems identified during Preliminary Design Model testing, but GAO observed instances in which Army testers did not follow the established testing protocols and did not maintain internal controls over the integrity and reliability of data, raising questions as to whether the Army met its First Article Test objective of determining whether each of the contracted designs met performance requirements. The following are examples of deviations from testing protocols and other issues that GAO observed: (1) The clay backing placed behind the plates during ballistics testing was not always calibrated in accordance with testing protocols and was exposed to rain on one day, potentially impacting test results. (2) Testers improperly rounded down back-face deformation measurements, which is not authorized in the established testing protocols and which resulted in two designs passing First Article Testing that otherwise would have failed. Army officials said rounding is a common practice; however, one private test facility that rounds told GAO that they round up, not down. (3) Testers used a new instrument to measure back-face deformation without adequately certifying that the instrument could function correctly and in conformance with established testing protocols. The impact of this issue on test results is uncertain, but it could call into question the reliability and accuracy of the measurements. (4) Testers deviated from the established testing protocols in one instance by improperly scoring a complete penetration as a partial penetration. As a result, one design passed First Article Testing that would have otherwise failed. With respect to internal control issues, the Army did not consistently maintain adequate internal controls to ensure the integrity and reliability of test data. In one example, during ballistic testing, data were lost, and testing had to be repeated because an official accidentally pressed the delete button and software controls were not in place to protect the integrity of test data. Army officials acknowledged that before GAO's review they were unaware of the specific internal control problems we identified.

section_title: Letter
paragraphs: Since combat operations began in Afghanistan after September 11, 2001,  and in Iraq in 2003, U.S. forces have been subjected to frequent and deadly  attacks from insurgents using improvised explosive devices, mortars,  rocket launchers, and increasingly lethal ballistic threats. To protect the  military and civilian personnel of the Department of Defense (DOD)  against these ballistic threats, since 2003 the U.S. Central Command has  required that DOD personnel in its area of operations be issued the  Interceptor Body Armor system, comprising ceramic plates that are  inserted into pockets of an outer protective vest. Over the past several  years, the media and Congress have raised concerns about whether the  Army has adequately evaluated and tested this body armor solution and  about the transparency of the Army’s body armor testing. Additionally,  several audits have found problems with the Army’s body armor testing  programs. For example, in 2009, both the DOD Inspector General and  Army Audit Agency reported that the Army had not followed established  test procedures during prior tests of body armor plates.
In 2007, we reported to the House and Senate Armed Services Committees  and testified to the House Armed Services Committee about the Army’s  and Marine Corps’s individual body armor systems. In that report we  found that the Army relied on several controls to ensure that body armor  met performance requirements, including testing at National Institute of  Justice (NIJ)-certified testing facilities. Later, under the Comptroller  General’s authority, we observed the testing of body armor solutions  submitted under a May 2007 Army contract solicitation for four categories  of body armor—specifically, the Enhanced Small Arms Protective Insert  (ESAPI), the Small Arms Protective Insert-X level (XSAPI), the Flexible  Small Arms Protective Vest-Enhanced (FSAPV-E), and the Flexible Small  Arms Protective Vest-X level (FSAPV-X). While present, we observed the  test procedures utilized by Army testers, spoke with Army officials, and  compared our observations with established testing protocols. The  purchase descriptions accompanying the contract solicitation  announcement identified the test procedures to be followed during the  first round of testing—called Preliminary Design Model testing.  Traditionally, Army body armor testing had been performed at an NIJ- certified facility. However, one manufacturer of flexible small arms  protective vests, which had failed previous testing conducted for the  Program Executive Office (PEO) Soldier at an NIJ-certified facility, made  allegations that the PEO Soldier and the facility had wrongly failed its  designs. As a result of these allegations, the Army decided instead to  conduct testing for this current solicitation at the Army’s Aberdeen Test  Center, which had not performed testing of Interceptor Body Armor for  PEO Soldier since the 1990s. Additionally, PEO Soldier decided not to  provide any on-site testing oversight to avoid any appearance of bias  against that manufacturer.
Preliminary Design Model testing was conducted by the Army’s Aberdeen  Test Center from February 2008 though June 2008. The objective of the  Preliminary Design Model testing was to determine whether candidate  designs submitted under the solicitation met required ballistics  performance specifications and would be awarded a production contract.  In October 2008, on the basis of the Preliminary Design Model testing  results, the Army awarded four 5-year indefinite delivery/indefinite  quantity contracts at a total of over $8 billion for the production of the  ESAPI and the XSAPI—two categories of ceramic plates. No FSAPV-E or  FSAPV-X solutions passed the testing. The Army decided to repeat testing,  through First Article Testing, of all of the ESAPI and XSAPI plates that  were awarded production contracts to determine whether these plate  designs indeed met the required ballistics performance specifications  before fielding the plates. The Aberdeen Test Center conducted First  Article Testing between November 2008 and December 2008.
In connection with the Army’s decision to conduct First Article Testing on  each of the designs that passed Preliminary Design Model testing and that  were awarded contracts, the House Armed Services Committee and its  Subcommittee on Air and Land Forces requested that we observe this  follow-on First Article Testing to assess the degree to which testing was  conducted according to the established testing protocols.  After  completing our analysis of both the Preliminary Design Model testing and  First Article Testing of body armor solutions, we are reporting on the  degree to which the Army followed its established testing protocols during  (1) Preliminary Design Model testing of the ESAPI, XSAPI, FSAPV-E and  FSAPV-X and (2) First Article Testing of the ESAPI and XSAPI models that  were awarded contracts after the Preliminary Design Model testing. We  did not provide an expert ballistics evaluation of the results of testing.
To conduct our review, we observed Preliminary Design Model testing and  First Article Testing at the Army’s Aberdeen Test Center in Aberdeen,  Maryland. We observed testing from inside the video viewing room and  firing lanes and also from the conditioning, X-ray, and physical  characterization rooms. We interviewed and collected information from  officials from the Aberdeen Test Center, the U.S. Army Evaluation Center,  PEO Soldier, DOD’s office of the Director of Operational Test and  Evaluation, and other Army components, as well as from body armor  manufacturers and private body armor testing facilities. We recorded  selected test data in a systematic and structured manner using a data  collection instrument we developed, analyzed selected test data, and  compared our observations of the way the Aberdeen Test Center  conducted Preliminary Design Model testing and First Article Testing with  the testing protocols that Army officials told us served as the testing  standards at the Aberdeen Test Center. Specifically, these testing  protocols were: (1) test procedures described in the contract solicitation  announcement’s purchase descriptions and (2) the Army’s detailed test  plans and test operations procedures that were to serve as guidance to  Aberdeen Test Center testers and that were developed by the Army Test  and Evaluation Command and approved by PEO Soldier, the office of the  Director of Operational Test and Evaluation, Army Research Labs, and  cognizant Army components. In this report, we refer to these testing  standards that were to be used at Aberdeen Test Center as testing  protocols. We also reviewed NIJ testing standards because Aberdeen Test  Center officials told us that, although Aberdeen Test Center is not an NIJ- certified testing facility, they have made adjustments to their procedures  based on those standards and consider them when evaluating Aberdeen  Test Center testing practices. Complete details on our scope and  methodology appear in appendix I.
We conducted this performance audit from July 2007 through October  2009 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Army Solicitation for Body Armor
paragraphs: In May 2007, the Army issued a solicitation for body armor designs to  replenish stocks and to protect against future threats by developing the  next generation (X level) of protection. According to Army officials, the  solicitation would result in contracts that the Army would use for  sustainment of protective plate stocks for troops in Iraq and Afghanistan.  The indefinite delivery/indefinite quantity contracts require the Army to  purchase a minimum of 500 sets per design and allow for a maximum  purchase of 1.2 million sets over the 5-year period.
The Army’s solicitation, which closed in February 2008, called for  preliminary design models in four categories of body armor protective  plates:    Enhanced Small Arms Protective Insert (ESAPI)—plates designed to  same protection specifications as those currently fielded and to fit into  currently fielded Outer Tactical Vests.
Flexible Small Arms Protective Vest-Enhanced (FSAPV-E)—flexible  armor system designed to same protection specifications as armor  currently fielded.
Small Arms Protective Insert-X level (XSAPI)—next-generation plates  designed to defeat higher level threat.
Flexible Small Arms Protective Vest-X level (FSAPV-X)—flexible  armor system designed to defeat higher level threat.
In figure 1, we show the ESAPI plates inside the Outer Tactical Vest.
Between May of 2007 and February of 2008 the Army established testing  protocols, closed the solicitation, and provided separate live-fire  demonstrations of the testing process to vendors who submitted items for  testing and to government officials overseeing the testing. Preliminary  Design Model testing was conducted at Aberdeen Test Center between  February 2008 and June 2008 at an estimated cost of $3 million.
Additionally, over $6 million was spent on infrastructure and equipment  improvements at Aberdeen Test Center to support future light armor test  range requirements, including body armor testing.
First Article Testing was then conducted at Aberdeen Test Center from  November 10, 2008, to December 17, 2008, on the three ESAPI and five  XSAPI designs that had passed Preliminary Design Model testing. First  Article Testing is performed in accordance with the Federal Acquisition  Regulation to ensure that the contractor can furnish a product that  conforms to all contract requirements for acceptance. First Article Testing  determines whether the proposed product design conforms to contract  requirements before or in the initial stage of production. During First  Article Testing, the proposed design is evaluated to determine the  probability of consistently demonstrating satisfactory performance and  the ability to meet or exceed evaluation criteria specified in the purchase  description. Successful First Article Testing certifies a specific design  configuration and the manufacturing process used to produce the test  articles. Failure of First Article Testing requires the contractor to examine  the specific design configuration to determine the improvements needed  to correct the performance of subsequent designs.
Testing of the body armor currently fielded by the Army was conducted by  private NIJ-certified testing facilities under the supervision of PEO Soldier.  According to Army officials, not a single death can be attributed to this  armor’s failing to provide the required level of protection for which it was  designed. However, according to Army officials, one of the body armor  manufacturers that had failed body armor testing in the past did not agree  with the results of the testing and alleged that the testers tested that armor  to higher–than–required standards. The manufacturer alleged a bias  against its design and argued that its design was superior to currently  fielded armor. As a result of these allegations and in response to  congressional interest, after the June 2007 House Armed Services  Committee hearing, the Army accelerated completion of the light armor  ranges to rebuild small arms ballistic testing capabilities at Aberdeen Test  Center and to conduct testing under the May 2007 body armor solicitation  there, without officials from PEO Soldier supervising the testing.  Furthermore, the decision was made to allow Aberdeen Test Center,  which is not an NIJ-certified facility, to be allowed to conduct the repeated  First Article Testing. In February 2009 the Army directed that all future  body armor testing be performed at Aberdeen Test Center. According to  Army officials, as of this date, none of the body armor procured under the  May 2007 solicitation had been fielded.
Given the significant congressional interest in the testing for this  solicitation and that these were the first small arms ballistic tests  conducted at Aberdeen Test Center in years, multiple defense  organizations were involved in the Preliminary Design Model testing.  These entities include the Aberdeen Test Center, which conducted the  testing; PEO Soldier, which provided the technical subject-matter experts;  and DOD’s office of the Director of Operational Test and Evaluation,  which combined to form the Integrated Product The Integrated Product  Team was responsible for developing and approving the test plans used for  the Preliminary Design Model testing and First Article Testing. Figure 2  shows a timeline of key Preliminary Design Model testing and First Article  Testing events.
section_title: Body Armor Test Procedures
paragraphs: The test procedures to be followed for Preliminary Design Model testing  were established and identified in the purchase descriptions  accompanying the solicitation announcement and in the Army’s detailed  test plans (for each of the four design categories), which served as  guidance to Army testers and were developed by the Army Test and  Evaluation Command and approved by PEO-Soldier, DOD’s office of the  Director of Operational Test and Evaluation, and others. Originally, PEO  Soldier required that testing be conducted at an NIJ-certified facility.  Subsequently, the decision was made to conduct testing at Aberdeen Test  Center, which is not NIJ-certified.
The test procedures for both Preliminary Design Model testing and First  Article Testing included both (1) physical characterization steps  performed on each armor design to ensure they met required  specifications, which included measuring weight, thickness, curvature,  and size and (2) ballistic testing performed on each design. Ballistics  testing for this solicitation included the following subtests: (1) ambient  testing to determine whether the designs can defeat the multiple threats  assigned in the respective solicitation’s purchase descriptions 100 percent  of the time; (2) environmental testing of the designs to determine whether  they can defeat each threat 100 percent of the time after being exposed to  nine different environmental conditions; and (3) testing, called V50 testing,  to determine whether designs can defeat each threat at velocities  significantly higher than those present or expected in Iraq or Afghanistan  at least 50 percent of the time.
Ambient and environmental testing seek to determine whether designs can  defeat each threat 100 percent of the time by both prohibiting the bullet  from penetrating through the plate and by prohibiting the bullet from  causing too deep of an indentation in the clay backing behind the plate.  Preventing a penetration is important because it prevents a bullet from  entering the body of the soldier. Preventing a deep indentation in the clay  (called “back-face deformation”) is important because the depth of the  indentation indicates the amount of blunt force trauma to the soldier.  Back-face deformation deeper than 43 millimeters puts the soldier at  higher risk of internal injury and death.
The major steps taken in conducting a ballistic subtest include:  1.  For environmental subtests, the plate is exposed to the environmental  condition tested (e.g., impact test, fluid soaks, temperature extremes,  etc.).  2.  The clay to be used to back the plate is formed into a mold and is  placed in a conditioning chamber for at least 3 hours.  3.  The test plate is placed inside of a shoot pack. 4.  The clay is taken out of the conditioning chamber. It is then tested to  determine if it is suitable for use and, if so, is placed behind the test  plate.  5.  The armor and clay are then mounted to a platform and shot. 6.  If the shot was fired within required specifications, the plate is  examined to determine if there is a complete or partial penetration,  and the back-face deformation is measured.  7.  The penetration result and back-face deformation are scored as a  pass, a limited failure, or a catastrophic failure. If the test is not  conducted according to the testing protocols, it is scored as a no-test.
section_title: Army Took Significant Steps during Preliminary Design Model Testing to Run a Controlled Test and Maintain Consistency but Did Not Consistently Follow Established Testing Protocols and, as a Result, Did Not Achieve the Intended Test Objective
subsections: section_title: Army Took Significant Steps to Run a Controlled Test and Maintain Consistency
paragraphs: Following are significant steps the Army took to run a controlled test and  maintain consistency throughout Preliminary Design Model testing:    The Army developed testing protocols for the hard-plate (ESAPI and  XSAPI) and flexible-armor (FSAPV-E and FSAPV-X) preliminary  design model categories in 2007. These testing protocols were  specified in newly created purchase descriptions, detailed test plans,  and other documents. For each of the four preliminary design model  categories, the Army developed new purchase descriptions to cover  both hard-plate and flexible designs. These purchase descriptions  listed the detailed requirements for each category of body armor in the  solicitation issued by the Army. Based on these purchase descriptions,  the Army developed detailed test plans for each of the four categories  of body armor. These detailed test plans provided additional details on  how to conduct testing and provided Army testers with the  requirements that each design needed to pass. After these testing  protocols were developed, Army testers then conducted a pilot test in  which they practiced test activities in preparation for Preliminary  Design Model testing, to help them better learn and understand the  testing protocols.
The Army consistently documented many testing activities by using  audio, video, and other electronic means. The use of cameras and  microphones to provide 24-hour video and audio surveillance of all of  the major Preliminary Design Model testing activities provided  additional transparency into many testing methods used and allowed  for enhanced oversight by Army management, who are unable to  directly observe the lanes on a regular basis but who wished to view  select portions of the testing. The Army utilized an electronic database  to maintain a comprehensive set of documentation for all testing  activities. This electronic database included a series of data reports  and pictures for each design including: physical characterization  records, X-ray pictures, pre- and post-shot pictures, ballistics testing  results, and details on the condition of the clay backing used for the  testing of those plates.
The Army took a number of additional actions to promote a consistent and  unbiased test. For example, the Army disguised vendor identity for each  type of solution by identifying vendors with random numbers to create a  blind test. The Army further reduced potential testing variance by shooting  subtests in the same shooting lane. The Army also made a good faith effort  to use consistent and controlled procedures to measure the weight,  thickness, and curvature of the plates. Additionally, the Army made  extensive efforts to consistently measure and maintain room temperature  and humidity within desired ranges. We also observed that projectile yaw  was consistently monitored and maintained. We also found no deviations  in the monitoring of velocities for each shot and the re-testing of plates in  cases where velocities were not within the required specifications. We  observed no instances of specific bias against any design, nor did we  observe any instances in which a particular vendor was singled out for  advantage or disadvantage.
section_title: Army Did Not Consistently Follow All Testing Protocols
paragraphs: We identified several instances in which the Aberdeen Test Center did not  follow established testing protocols. For example, during V50 testing,  testers failed to properly adjust shot velocities. V50 testing is conducted to  discern the velocity at which 50 percent of the shots of a particular threat  would penetrate each of the body armor designs. The testing protocols  require that after every shot that is defeated by the body armor the  velocity of the next shot be increased. Whenever a shot penetrates the  armor, the velocity should be decreased for the next shot. This increasing  and decreasing of the velocities is supposed to be repeated until testers  determine the velocity at which 50 percent of the shots will penetrate. In  cases in which the armor far exceeds the V50 requirements and is able to  defeat the threat for the first six shots, the testing may be halted without  discerning the V50 for the plate, and the plate is ruled as passing the  requirements. During Preliminary Design Model testing, in cases in which  plates defeated the first three shots, Army testers failed to increase shot  velocities, but rather continued to shoot at approximately the same  velocity or lower for shots four, five, and six in order to obtain six partial  penetrations and conclude the test early. Army officials told us that this  deviation was implemented by Aberdeen Test Center to conserve plates  for other tests that needed repeating as a result of no-test events,  according to Aberdeen Test Center officials—but was a practice not  described in the protocols. Army officials told us that this practice had no  effect on which designs passed or failed; however, this practice made it  impossible to discern the true V50s for these designs and was a deviation  from the testing protocols that require testers to increase velocities for  shots after the armor defeats the threat.
In another example, Aberdeen Test Center testers did not consistently  follow testing protocols in the ease-of-insertion test. According to the  testing protocols, one barehanded person shall demonstrate insertion and  removal of the ESAPI/XSAPI plates in the Outer Tactical Vest pockets  without tools or special aids. Rather than testing the insertion of both the  front and the rear pockets as required, testers only tested the ability to  insert into the front pocket. Testing officials told us that they did not test  the ability to insert the plates into the rear pocket because they were  unable to reach the rear pocket while wearing the Outer Tactical Vest. The  cause for this deviation is that the testers misinterpreted the testing  protocols, as there is no requirement in the established testing protocols to  wear the Outer Tactical Vest when testing the ability to insert the plates in  the rear pocket of the Outer Tactical Vest. Officials from PEO Soldier told  us that, had they been present to observe this deviation during testing,  they would have informed testers that the insertion test does not require  that the Outer Tactical Vest be worn, which would have resulted in testers  conducting the insertion test as required. According to Aberdeen Test  Center officials, this violation of the testing protocols had no impact on  test results. While we did not independently verify this assertion,  Aberdeen Test Center officials told us that the precise physical  characterization measurements of the plate’s width and dimensions are,  alone, sufficient to ensure the plate will fit.
In addition, testers deviated from the testing protocols by placing shots at  the wrong location on the plate. The testing protocols require that the  second shot for one of the environmental sub-tests, called the impact test,  be taken approximately 1.5 inches from the edge of the armor. However,  testers mistakenly aimed closer to the edge of the armor for some of the  designs tested. Army officials said that the testing protocols were unclear  for this test because they did not prescribe a specific hit zone (e.g., 1.25 –  1.75 inches), but rather relied upon testers’ judgment to discern the  meaning of the word “approximately.” One of the PEO Soldier technical  advisors on the Integrated Product Team told us he was contacted by the  Test Director after the plates had been shot and asked about the shot  location. He told us that he informed the Test Director that the plates had  been shot in the wrong location. The PEO Soldier Technical advisor told  us that, had he been asked about the shot location before the testing was  conducted, he could have instructed testers on the correct location at  which to shoot. For 17 of the 47 total designs that we observed and  measured, testers marked target zones that were less than the required  1.5 inches from the plate’s edge, ranging from .75 inches to 1.25 inches  from the edge. Because 1.5 inches was outside of the marked aim area for  these plates, we concluded that testers were not aiming for 1.5 inches. For  the remaining 30 designs tested that we observed and measured, testers  used a range that included 1.5 inches from the edge (for example, aiming  for 1 to 1.5 inches). It is not clear what, if any, effect this deviation had on  the overall test results. While no design failed Preliminary Design Model  testing due to the results of this subtest, there is no way to determine if a  passing design would have instead failed if the testing protocol had been  correctly followed. However, all designs that passed this testing were later  subject to First Article Testing, where these tests were repeated in full  using the correct shot locations.
Of potentially greater consequence to the final test results is our  observation of deviations from testing protocols regarding the clay  calibration tests. According to testing protocols, the calibration of the clay  backing material was supposed to be accomplished through a series of  pre-test drops. The depths of the pre-test drops should have been  between 22 and 28 millimeters. Aberdeen Test Center officials told us that  during Preliminary Design Model testing they did not follow a consistent  system to determine if the clay was conditioned correctly. According to  Aberdeen Test Center officials, in cases in which pre-test drops were  outside the 22- to 28-millimeter range, testers would sometimes repeat one  or all of the drops until the results were within range—thus resulting in the  use of clay backing materials that should have been deemed unacceptable  for use. These inconsistencies occurred because Army testers in each test  lane made their own, sometimes incorrect, interpretation of the testing  protocols. Members of the Integrated Product Team expressed concerns  about these inconsistencies after they found out how calibrations were  being conducted. In our conversations with Army and private body armor  testing officials, consistent treatment and testing of clay was identified as  critical to ensure consistent, accurate testing. According to those officials  if the clay is not conditioned correctly it will impact the test results. Given  that clay was used during Preliminary Design Model testing that failed the  clay calibration tests, it is possible that some shots may have been taken  under test conditions different than those stated in the testing protocols,  potentially impacting test results. Figure 3 shows an Army tester  calibrating the clay with pre-test drops.
The most consequential of the deviations from testing protocols we  observed involved the measurement of back-face deformation, which did  affect final test results. According to testing protocol, back-face  deformation is to be measured at the deepest point of the depression in  the clay backing. This measure indicates the most force that the armor will  allow to be exerted on an individual struck by a bullet. According to A rmy  officials, the deeper the back-face deformation measured in the cla backing, the higher the risk of internal injury or death. During  approximately the first one-third of testing, however, Army testers  incorrectly measured deformation at the point of aim, rather than at the  deepest point of depression. This is significant because, in many instances,  measuring back-face deformation at the point of aim results in measuri ng  at a point upon which less ballistic force is exerted, resulting in lower  back-face deformation measurements and overestimating the effectiveness  of the armor. The Army’s subject matter experts on the Integrated Product  Team were not on the test lanes during testing and thus not made aware of  the error until approximately one-third of the testing had been completed.
When members of the Integrated Product Team overseeing the testing  were made aware of this error, the Integrated Product Team decided to  begin measuring at the deepest point of depression. When senior Army  leadership was made aware of this error, testing was halted for 2 weeks  while Army leadership considered the situation. Army leadership  developed many courses of action, including restarting the entire  Preliminary Design Model testing with new armor plate submissions, but  ultimately decided to continue measuring and scoring officially at the  point of aim, since this would not disadvantage any vendors. The Army  then changed the test plans and modified the contract solicitation to call  for measuring at the point of aim. The Army also decided to collect  deepest point of depression measurements for all shots from that point  forward, but only as a government reference. During the second two-thirds  of testing, we observed significant differences between the measurements  taken at the point of aim and those taken at the deepest point, as much as  a 10-millimeter difference between measurements. As a result, at least two  of the eight designs that passed Preliminary Design Model testing and  were awarded contracts would have failed if the deepest point of  depression measurement had been used. Figures 4 and 5 illustrate the  difference between the point of aim and the deepest point.
section_title: Army Decided to Repeat Testing in First Article Testing in an Attempt to Address Back-Face Deformation Measurement Problem Identified during Preliminary Design Model Testing
paragraphs: Before Preliminary Design Model testing began at Aberdeen Test Center,  officials told us that Preliminary Design Model testing was specifically  designed to meet all the requirements of First Article Testing. However,  Preliminary Design Model testing failed to meet its goal of determining  which designs met requirements, because of the deviations from  established testing protocols described earlier in this report. Those  deviations were not reviewed or approved by officials from PEO Soldier,  the office of the Director of Operational Test and Evaluation, or by the  Integrated Product Team charged with overseeing the test. PEO Soldier  officials told us that the reason for a lack of PEO Soldier on-site presence  during this testing was because of a deliberate decision made by PEO  Soldier management to be as removed from the testing process as possible  in order to maximize the independence of the Aberdeen Test Center. PEO  Soldier officials told us that it was important to demonstrate the  independence of the Aberdeen Test Center to quash allegations of bias  made by a vendor whose design had failed prior testing and that this  choice may have contributed to some of the deviations not being identified  by the Army earlier during testing. After the conclusion of Preliminary  Design Model testing, PEO Soldier officials told us that they should have  been more involved in the testing and that they would be more involved in  future testing.
After the completion of Preliminary Design Model testing, the  Commanding General of PEO Soldier said that, as the Milestone Decision  Authority for the program, he elected to repeat the testing conducted  during Preliminary Design Model testing through First Article Testing  before any body armor was fielded based on the solicitation. According to  PEO Soldier officials, at the beginning of Preliminary Design Model  testing, there was no intention or plan to conduct First Article Testing  following contract awards given that the Preliminary Design Model testing  was to follow the First Article Testing protocol. However, because of the  fact that back-face deformation was not measured to the deepest point,  PEO-Soldier and Army Test and Evaluation and Command acknowledged  that there was no longer an option of forgoing First Article Testing. PEO  Soldier also expressed concerns that Aberdeen Test Center test facilities  have not yet demonstrated that they are able to test to the same level as  NIJ-certified facilities. However, officials from Army Test and Evaluation  Command and DOD’s office of the Director of Operational Test and  Evaluation asserted that Aberdeen Test Center was just as capable as NIJ- certified laboratories, and Army leadership eventually decided that First  Article Testing would be performed at Aberdeen.
section_title: During First Article Testing the Army Addressed Some of the Problems Identified in Preliminary Design Model Testing, but Army Testers Did Not Always Follow Established Testing Protocols and Did Not Maintain Some Internal Controls
subsections: section_title: During First Article Testing, the Army Addressed Some of the Problems Identified during Preliminary Design Model Testing
paragraphs: PEO Soldier maintained an on-site presence in the test lanes and the Army  technical experts on the Integrated Product Team charged with testing  oversight resolved the following problems during First Article Testing:    The Army adjusted its testing protocols to clarify the required shot  location for the impact test, and Army testers correctly placed these  shots as required by the protocols.
After the first few days of First Article Testing, in accordance with  testing protocols, Army testers began to increase the velocity after  every shot defeated by the armor required during V50 testing.
As required by the testing protocols, Army testers conducted the ease- of-insertion tests for both the front and rear pockets of the outer  protective vest, ensuring that the protective plates would properly fit  in both pockets.
The Army began to address the problems identified during Preliminary  Design Model testing with the clay calibration tests and back-face  deformation measurements. Army testers said they developed an informal  set of procedures to determine when to repeat failed clay calibration tests.  The procedures, which were not documented, called for repeating the  entire series of clay calibration drops if one of the calibration drops  showed a failure. If the clay passes either the first or second test, the clay  is to be used in testing. If the clay fails both the first and the second series  of drops, the clay is to then be placed back in conditioning and testers get  a new block of clay. With respect to back-face deformation measurements,  Army testers measured back-face deformation at the deepest point, rather  than at the point of aim.
section_title: Army Did Not Follow All Established Testing Protocols during First Article Testing
paragraphs: Although the Army began to address problems relating to the clay  calibration tests and back-face deformation measurements, Army testers  still did not follow all established testing protocols in these areas. As a  result, the Army may not have achieved the objective of First Article  Testing—to determine if the designs tested met the minimum  requirements for ballistic protection. First, the orally agreed-upon  procedures used by Army testers to conduct the clay calibration tests were  inconsistent with the established testing protocols. Second, with respect  to back-face deformation measurements, Army testers rounded back-face  deformation measurements to the nearest millimeter, a practice that was  neither articulated in the testing protocols nor consistent with Preliminary  Design Model testing. Third, also with respect to back-face deformation  measurements, Army testers introduced a new, unproven measuring  device.
Although Army testers told us that they had orally agreed upon an  informal set of procedures to determine when to repeat failed clay  calibration tests, those procedures are inconsistent with the established  testing protocols. The Army deviated from established testing protocols by  using clay that had failed the calibration test as prescribed by the testing  protocols. The testing protocols specify that a series of three pre-test  drops of a weight on the clay must be within specified tolerances before  the clay is used. However, in several instances, the Army repeated the  calibration test on the same block of clay after it had initially failed until  the results of a subsequent series of three drops were within the required  specifications. Army officials told us that the testing protocols do not  specify what procedures should be performed when the clay does not pass  the first series of calibration drops, so Army officials stated they  developed the procedure they followed internally prior to First Article  Testing and provided oral guidance on those procedures to all test  operators to ensure a consistent process. Officials we spoke with from the  Army, private NIJ-certified laboratories, and industry had mixed opinions  regarding the practice of re-testing failed clay, with some expressing  concerns that performing a second series of calibration drops on clay that  had failed might introduce risk that the clay may not be at the proper  consistency for testing because as the clay rests it cools unevenly, which  could affect the calibration. Aberdeen Test Center’s Test Operating  Procedure states that clay should be conditioned so that the clay passes  the clay calibration test, and Army officials, body armor testers from  private laboratories, and body armor manufacturers we spoke to agreed  that when clay fails the calibration test, this requires re-evaluation and  sometimes adjustment of the clay calibration procedures used. After  several clay blocks failed the clay calibration test on November 13, 2008,  Army testers recognized that the clay conditioning process used was  yielding clay that was not ideal and, as a result, Army testers adjusted their  clay conditioning process by lowering the temperature at which the clay  was stored.
On that same day of testing, November 13, 2008, we observed heavy, cold  rain falling on the clay blocks that were being transported to test lanes.  These clay blocks had been conditioned that day in ovens located outside  of the test ranges at temperatures above 100 degrees Fahrenheit to  prepare them  for testing, and then were transported outside uncovered on  a cold November day through heavy rain on the way to the temperature-  and humidity-controlled test lane. We observed an abnormally high level of  clay blocks failing the clay calibration test and a significantly higher-than- normal level of failure rates for the plates tested on that day. The only  significant variation in the test environment we observed that day was  constant heavy rain throughout the day. Our analysis of test data also  showed that 44 percent (4 of 9) of the first shots and 89 percent (8 of 9) of  the second shots taken on November 13, 2008, resulted in failure  penalties. On all of the other days of testing only 14 percent (10 of 74) of  the first shots and 42 percent (31 of 74) of the second shots resulted in  failure penalties. Both of these differences are statistically significant, and  we believe the differences in the results may be attributable to the  different test condition on that day. The established testing protocols  require the use of a specific type of non-hardening oil-based clay. Body  armor testers from NIJ-certified private laboratories, Army officials  experienced in the testing of body armor, body armor manufacturers, and  the clay manufacturer we spoke with said that the clay used for testing is a  type of sculpting clay that naturally softens when heat is added and that  getting water on the clay backing material could cause a chemical bonding  change on the clay surface. Those we spoke with further stated that the  cold water could additionally cause the outside of the clay to cool  significantly more rapidly than the inside causing the top layer of clay to  be harder than the middle. They suggested that clay be conditioned inside  the test lanes and said that clay exposed to water or extreme temperature  changes should not be used. Army Test and Evaluation Command officials  we spoke with said that there is no prohibition in the testing protocols on  allowing rain to fall onto the clay backing material and that its exposure to  water would not impact testing. However, these officials were unable to  provide data to validate their assertion that exposure to water would not  affect the clay used during testing or the testing results. Army test officials  also said that, since the conclusion of First Article Testing, Aberdeen Test  Center has procured ovens to allow clay to be stored inside test lanes,  rather than requiring that the clay be transported from another room  where it would be exposed to environmental conditions, such as rain.
With respect to the issue of the rounding of back-face deformation  measurements, during First Article Testing Army testers did not award  penalty points for shots with back-face deformations between 43.0 and  43.5 millimeters. This was because the Army decided to round back-face  deformation measurements to the nearest millimeter—a practice that is  inconsistent with the Army’s established testing protocols, which require  that back-face deformation measurements in the clay backing not exceed  43 millimeters and that is inconsistent with procedures followed during  Preliminary Design Model testing. Army officials said that a decision to  round the measurements for First Article Testing was made to reflect  testing for past Army contract solicitations and common industry  practices of recording measurements to the nearest millimeter. While we  did not validate this assertion that rounding was a common industry  practice, one private industry ballistics testing facility said that its practice  was to always round results up, not down, which has the same effect as  not rounding at all. Army officials further stated that they should have also  rounded Preliminary Design Model results but did not realize this until  March 2008—several weeks into Preliminary Design Model testing—and  wanted to maintain consistency throughout Preliminary Design Model  testing. The Army’s decision to round measurement results had a  significant outcome on testing because two designs that passed First  Article Testing would have instead failed if the measurements had not  been rounded.
With respect to the introduction of a new device to measure back-face  deformation, the Army began to use a laser scanner to measure back-face  deformation without adequately certifying that the scanner could measure  against the standard established when the digital caliper was used as the  measuring instrument. Although Army Test and Evaluation Command  certified the laser scanner as accurate for measuring back-face  deformation, we observed the following certification issues:    The laser was certified based on testing done in a controlled laboratory  environment that is not similar to the actual conditions on the test  lanes. For example, according to the manufacturer of the laser  scanner, the scanner is operable in areas of vibration provided the area  scanned and the scanning-arm are on the same plane or surface. This  was not the case during testing, and thus it is possible the impact of  the bullets fired may have thrown the scanner out of alignment or  calibration.
The certification is to a lower level of accuracy than required by the  testing protocols. The certification study says that the laser is accurate  to 0.2 millimeters; however, the testing protocols require an accuracy  of 0.1 millimeters or better. Furthermore, the official letter from the  Army Test and Evaluation Command certifying the laser for use  incorrectly stated the laser meets an accuracy requirement of 1.0  millimeter rather than 0.1 millimeters as required by the protocols.  Officials confirmed that this was not a typographical error.
The laser certification was conducted before at least three major  software upgrades were made to the laser, which according to Army  officials may have significantly changed the accuracy of the laser.  Because of the incorporation of the software upgrades, Army testers  told us that they do not know the accuracy level of the laser as it was  actually used in First Article Testing.
In evaluating the use of the laser scanner, the Army did not compare  the actual back-face deformation measurements taken by the laser  with those taken by digital caliper, previously used during Preliminary  Design Model testing and by NIJ-certified laboratories. According to  vendor officials and Army subject matter experts, the limited data they  had previously collected have shown that back-face deformation  measurements taken by laser have generally been deeper by about 2  millimeters than those taken by digital caliper. Given those preliminary  findings, there is a significant risk that measurements taken by the  laser may represent a significant change in test requirements.
Although Army testing officials acknowledged that they were unable to  estimate the exact accuracy of the laser scanner as it was actually used  during testing, they believed that based on the results of the certification  study, it was suitable for measuring back-face deformation. These test  officials further stated that they initially decided to use the laser because  they did not believe it was possible to measure back-face deformations to  the required level of accuracy using the digital caliper. However, officials  from PEO Soldier and private NIJ-certified laboratories have told us that  they believe the digital caliper method is capable of making these  measurements with the required level of accuracy and have been using  this technique successfully for several years. PEO Soldier officials also  noted that the back-face deformation measurements in the testing  protocols were developed using this digital caliper method. Army testing  officials noted that the laser certification study confirmed their views that  the laser method was more accurate than the digital caliper. However,  because of the problems with the study that we have noted in this report,  it is still unclear whether the laser is the most appropriate and accurate  technique for measuring back-face deformation.
Although we did not observe problems in the Army’s determination of  penetration results during Preliminary Design Model testing, during First  Article Testing we observed that the Army did not consistently follow its  testing protocols in determining whether a shot was a partial or a  complete penetration. Army testing protocols require that penalty points  be awarded when any fragment of the armor material is imbedded or  passes into the soft under garment used behind the plate; however, the  Army did not score the penetration of small debris through a plate as a  complete penetration of the plate in at least one case that we observed. In  this instance, we observed small fragments from the armor three layers  deep inside the Kevlar backing behind the plate. This shot should have  resulted in the armor’s receiving 1.5 penalty points, which would have  caused the design to fail First Article Testing. Army officials said that  testers counted the shot as only a partial penetration of the plate because  it was determined that fibers of the Kevlar backing placed behind the plate  were not broken, which they stated was a requirement for the shot to be  counted as a complete penetration of the plate. This determination was  made with the agreement of an Army subject-matter expert from PEO- Soldier present on the lane. However, the requirement for broken fibers is  inconsistent with the written testing protocols. Army officials  acknowledged that the requirement for broken fibers was not described in  the testing protocols or otherwise documented but said that Army testers  discussed this before First Article Testing began. Figure 6 shows the tear  in the fibers of the rear of the plate in question.
section_title: Army Did Not Maintain Internal Controls over the Integrity and Reliability of Test Data at All Times
paragraphs: Federal internal control standards require that federal agencies maintain  effective controls over information processing to help ensure  completeness, accuracy, authorization, and validity of all transactions.  However, the Army did not consistently maintain adequate internal  controls to ensure the integrity and reliability of its test data. For example,  in one case bullet velocity data were lost because the lane Test Director  accidentally pressed the delete button on the keyboard, requiring a test to  be repeated. Additionally, we noticed that the software being used with  the laser scanner to calculate back-face deformation measurements lacked  effective edit controls, which could potentially allow critical variables to  be inappropriately modified during testing. We further observed a few  cases in which testers attempted to memorize test data for periods of time,  rather than writing that data down immediately. In at least one case, this  practice resulted in the wrong data being reported and entered into the  test records.
section_title: Army Did Not Formally Document Significant Procedures That Deviated from Established Testing Protocols or Assess the Impact of These Deviations
paragraphs: According to Army officials, decisions to implement those procedures that  deviated from testing protocols were reviewed and approved by  appropriate officials. However, these decisions were not formally  documented, the testing protocols were not modified to reflect the  changes, and vendors were not informed of the procedures. At the  beginning of testing, the Director of Testing said that any change to the  testing protocols has to be approved by several Army components;  however, the Army was unable to produce any written documentation  indicating approval of the deviations we observed by those components.  With respect to internal control issues, Army officials acknowledged that  before our review they were unaware of the specific internal control  problems we identified. We noted during our review that in industry, as  part of the NIJ certification process, an external peer review process is  used to evaluate testing processes and procedures of ballistics testing  facilities to ensure that effective internal controls are in place. However,  we found that the Aberdeen Test Center has conducted no such reviews, a  contributing factor to  the Army’s lack of  unawareness of the control  problems we noted.
As a result of the deviations from testing protocols that we observed, three  of the five designs that passed First Article Testing would not have passed  under the existing testing protocols. Furthermore, one of the remaining  two designs that passed First Article Testing was a design that would have  failed Preliminary Design Model testing if back-face deformation was  measured in accordance with the established protocols for that test. Thus,  four of the five designs that passed First Article Testing and were certified  by the Army as ready for full production would have instead failed testing  at some point during the process, either during the initial Preliminary  Design Model testing or the subsequent First Article testing, if all the  established testing protocols had been followed. As a result, the overall  reliability and repeatability of the test results are uncertain. However,  because ballistics experts from the Army or elsewhere have not assessed  the impact of the deviations from the testing protocols we observed during  First Article Testing, it is not certain whether the effect of these deviations  is sufficient to call into question the ability of the armor to meet mission  requirements. Although it is certain that some armor passed testing that  would not have if specific testing protocols had been followed, it is  unclear if there are additional factors that would mean the armor still  meets the required performance specifications. For example, the fact that  the laser scanner used to measure back-face deformation may not be as  accurate as what the protocol requires may offset the effects of rounding  down back-face deformations. Likewise, it is possible that some of the  deviations that did not on their own  have a visible effect on testing results  could, when taken together with other deviations, have a combined effect  that is greater. In our opinion, given the significant deviations in the  testing protocols, independent ballistics testing expertise would be  required to determine whether or not the body armor designs procured  under this solicitation provide the required level of protection. The Army  has ordered 2,500 sets of plates (at two plates per set) from those vendors  whose designs passed First Article Testing to be used for additional  ballistics testing and 120,000 sets of plates to be put into inventory to  address future requirements. However, to date, none of these designs have  been fielded because, according to Army officials, there are adequate  quantities of armor plates produced under prior contracts already in the  inventory to meet current requirements.
section_title: Conclusions
paragraphs: Body armor plays a critical role in protecting our troops, and the testing  inconsistencies we identified call into question the quality and  effectiveness of testing performed at Aberdeen Test Center. Because we  observed several instances in which actual test practices deviated from the  established testing protocols, it is questionable whether the Army met its  First Article Testing objectives of ensuring that armor designs fully met  Army’s requirements before the armor is purchased and used in the field.  While it is possible that the testing protocol deviations had no significant  net effect or may have even resulted in armor being tested to a more  rigorous standard, it is also possible that some deviations may have  resulted in armor being evaluated against a less stringent standard than  required. We were unable to determine the full effects of these deviations  as they relate to the quality of the armor designs and believe such a  determination should only be made based on a thorough assessment of the  testing data by independent ballistics testing experts. In light of such  uncertainty and the critical need for confidence in the equipment by the  soldiers, the Army would take an unacceptable risk if it were to field these  designs without taking additional steps to gain the needed confidence that  the armor will perform as required.
The Army is now moving forward with plans to conduct all future body  armor testing at Aberdeen Test Center. Therefore, it is essential that the  transparency and consistency of its program be improved by ensuring that  all test practices fully align with established testing protocols and that any  modifications in test procedures be fully reviewed and approved by the  appropriate officials, with supporting documentation, and that the testing  protocols be formally changed to reflect the revised or actual procedures.  Additionally, it is imperative that all instrumentation, such as the laser  scanner, used for testing be fully evaluated and certified to ensure its  accuracy and applicability to body armor testing. Furthermore, it is  essential that effective internal controls over data and testing processes be  in place. The body armor industry has adopted the practice, through the  NIJ certification program, of using external peer reviews to evaluate and  improve private laboratories’ test procedures and controls. This type of  independent peer review could be equally beneficial to the Aberdeen Test  Center. Without all of these steps, there will continue to be uncertainty  with regard to whether future testing data are repeatable and reliable and  can be used to accurately evaluate body armor designs. Until Aberdeen  Test Center has effectively honed its testing practices to eliminate the  types of inconsistencies we observed, concerns will remain regarding the  rigor of testing conducted at that facility.
section_title: Recommendations for Executive Action
paragraphs: To determine what effect, if any, the problems we observed had on the test  data and on the outcomes of First Article Testing, we recommend the  Secretary of Defense direct the Secretary of the Army to provide for an  independent evaluation of the First Article Testing results by ballistics and  statistical experts external to DOD before any armor is fielded to soldiers  under this contract solicitation and that the Army report the results of that  assessment to the office of the Director of Operational Test and  Evaluation and the Congress. In performing this evaluation, the  independent experts should specifically evaluate the effects of the  following practices observed during First Article Testing:  the rounding of back-face deformation measurements;    not scoring penetrations of material through the plate as a complete  penetration unless broken fibers are observed in the Kevlar backing  behind each plate;  the use of the laser scanner to measure back-face deformations  without a full evaluation of its accuracy as it was actually used during  testing, to include the use of the software modifications and operation  under actual test conditions;  the exposure of the clay backing material to rain and other outside  environmental conditions as well as the effect of high oven  temperatures during storage and conditioning; and  the use of an additional series of clay calibration drops when the first  series of clay calibration drops does not pass required specifications.
To better align actual test practices with established testing protocols  during future body armor testing, we recommend that the Secretary of the  Defense direct the Secretary of the Army to document all key decisions  made to clarify or change the testing protocols. With respect to the  specific inconsistencies we identified between the test practices and  testing protocols, we recommend that the Secretary of the Army, based on  the results of the independent expert review of the First Article Test  results, take the following actions:    Determine whether those practices that deviated from established  testing protocols during First Article Testing will be continued during  future testing and change the established testing protocols to reflect  those revised practices.
Evaluate and re-certify the accuracy of the laser scanner to the correct  standard with all software modifications incorporated and include in  this analysis a side-by-side comparison of the laser measurements of  the actual back-face deformations with those taken by digital caliper to  determine whether laser measurements can meet the standard of the  testing protocols.
To improve internal controls over the integrity and reliability of test data  for future testing as well as provide for consistent test conditions and  comparable data between tests, we recommend that the Secretary of  Defense direct the Secretary of the Army to provide for an independent  peer review of Aberdeen Test Center’s body armor testing protocols,  facilities, and instrumentation to ensure that proper internal controls and  sound management practices are in place. This peer review should be  performed by testing experts external to the Army and DOD.
section_title: Matter for Congressional Consideration
paragraphs: DOD did not concur with our recommendation for an independent  evaluation of First Article Testing results and accordingly plans to take no  action to provide such an assessment. DOD asserted that the issues we  identified do not alter the effects of testing. However, based on our  analysis and findings there is sufficient evidence to raise questions as to  whether the issues we identified had an impact on testing results. As a  result, we continue to believe it is necessary to have an independent  external expert review these test results and the overall effect of the  testing deviations we observed on those results before any armor is fielded  to military personnel. Without such an independent review, the First  Article Test results remain questionable, undermining the confidence of  the public and those who might rely on the armor for protection.  Consequently, Congress should consider directing the Office of the  Secretary of Defense to either require that an independent external review  of these body armor test results be conducted or that DOD officially  amend its testing protocols to reflect any revised test procedures and  repeat First Article Testing to ensure that only properly tested designs are  fielded.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, DOD takes the position that  our findings had no significant impact on the test results and on the  subsequent contracting actions taken by the Army. DOD also does not  concur with what it perceives as our two overarching conclusions: (1) that  Preliminary Design Model testing did not achieve its intended objective of  determining, as a basis for contract awards, which designs met  performance requirements and (2) that First Article Testing may not have  met its objective of determining whether each of the contracted plate  designs met performance requirements. DOD commented that it  recognizes the importance of personal protection equipment such as body  armor and provided several examples of actions DOD and the Army have  taken to improve body armor testing. DOD generally concurred with our  findings that there were deviations from the testing protocols during  Preliminary Design Model testing and First Article Testing. We agree that  DOD has taken positive steps to improve its body armor testing program  and to address concerns raised by Congress and others. DOD also  concurred with our second recommendation to document all key  decisions made to clarify or change the testing protocols. DOD did not  concur with our first recommendation that an independent evaluation of  First Article Testing results be performed by independent ballistics and  statistical experts before any of the armor is fielded to soldiers under  contracts awarded under this solicitation. Similarly, DOD did not agree  with our conclusions that Preliminary Design Model testing did not meet  its intended objectives and that First Article Testing may not have met its  intended objectives. In supporting its position, DOD cited, for example,  that rounding back-face deformation measurements during First Article  Testing was an acceptable test practice because rounding is a practice that  has been used historically. It was the intent of PEO Soldier to round back- face deformations for all testing associated with this solicitation, and the  Integrated Product Team decided collectively to round back-face  deformations during First Article Testing. However, as stated in our report  and acknowledged by DOD, the rounding down of back-face deformations  was not spelled out or provided for by any of the testing protocol  documents. Additionally, it created an inconsistency between Preliminary  Design Model testing, where back-face deformations were not rounded  down and in First Article Testing, where back-face deformations were  rounded down. Of greatest consequence, rounding down back-face  deformations lowered the requirements that solutions had to meet to pass  testing. Two solutions passed First Article Testing because back-face  deformations were rounded down, meaning that the Army may be taking  unacceptable risk if plates are fielded without an additional, independent  assessment by experts.
DOD also did not agree with our finding that a penetration of a plate was  improperly scored. DOD did agree that figure 6, which shows the tear in  the Kevlar fibers of the rear of the plate in question, appears to show  evidence of a perforation and that an Aberdeen Test Center ballistics  subject matter expert found particles in the soft backing material behind  the plate. Nevertheless, DOD did not concur with our finding because it  asserted that no threads were broken on the first layer of Kevlar. However,  as we stated in the report, the protocols define a complete penetration as  having occurred when the projectile, fragment of the projectile, or  fragment of the armor material is imbedded or passes into the soft under  garment used behind the protective inserts plates, not when threads of the  Kevlar are broken. The fragments found by the Aberdeen Test Center  subject matter expert, as well as the three frayed, tattered, and separated  Kevlar layers that we and Army testers observed, confirm our observations  during testing. DOD also stated that the first layer of soft armor behind the  plate under test serves as a witness plate during testing and if that first  layer of soft armor is not penetrated, as determined by the breaking of  threads on that first layer of soft armor, the test shot is not scored as a  complete penetration in accordance with the PEO Soldier’s scoring  criteria. We disagree with DOD’s position because the protocols do not  require the use of a “witness plate” during testing to determine if a  penetration occurred. If this shot would have been ruled a complete  penetration rather than a partial penetration, this design would have  accrued additional point deductions causing it to fail First Article Testing.
DOD did not agree that the certification of the laser scanner was  inadequate and made several statements in defense of both the laser and  its certification. Among these is the fact that the laser removes the human  factor of subjectively trying to find the deepest point, potentially pushing  the caliper into the clay, and removing the need to use correction factors,  all of which we agree may be positive things. However, we maintain that  the certification of the laser was not adequately performed. As indicated in  the certification letter, the laser was certified to a standard that did not  meet the requirement of the testing protocols. Additionally, DOD stated  that software modifications added to the laser after certification did not  affect measurements; however, Army testers told us on multiple occasions  that the modifications were designed to change the measurements  reported by the laser. DOD added that the scanner does not artificially  overstate back-face deformations and relies on the verified accuracy of the  scanner and the study involving the scanning of clay replicas to support its  claim. Based on our observations, the scanner was certified to the wrong  standard and the certification study was not performed in the actual test  environment using actual shots. DOD asserts that the scanner does not  overstate back-face deformations and that it does not establish a new  requirement.  However, DOD cannot properly validate these assertions  without a side-by-side comparison of the laser scanner and the digital  caliper in their operational environment. Given the numerous issues  regarding the laser and its certification, we maintain that its effect on First  Article Testing should be examined by an external ballistics expert.
DOD also stated that it did not agree with our finding that exposure of the  clay backing to heavy rain on one day may have affected test results. DOD  challenged our statistical analysis and offered its own statistical analysis  as evidence that it was the poor designs themselves that caused unusual  test results that day. We stand by our analysis, in combination with  statements made by DOD and non-DOD officials with testing expertise and   by the clay manufacturer, that exposure of the clay to constant, heavy cold  rain may have had an effect on test results. Further, in analyzing the  Army’s statistical analysis presented in DOD’s comments, we did not find  this information to demonstrate that the designs were the factor in  unusual test results that day or that the rain exposure could not have had  an effect on the results. More detailed discussions of the Army’s analysis  and our conclusions are provided in comments 13 and 24 of appendix II.
DOD partially disagreed that the use of an additional series of clay  calibration drops when the first series of drops were outside specifications  did not meet First Article Test requirements and added that all clay used in  testing passed the clay calibration in effect at the time. However, we  witnessed several clay calibration drops that were not within  specifications. These failed clay boxes were repaired, re-dropped, and  either used if they passed the subsequent drop calibration series or  discarded if they failed. The protocols only allow for one series of drops  per clay box, which is the methodology that Army testers should have  followed. DOD stated that NIJ standards do permit the repeating of failed  calibration drops. However, our review of the NIJ standards reveals that  there is no provision that allows repeat calibration drops. DOD states in its  comments that NIJ standards are inappropriate for its test facilities,  stating that these standards are insufficient for the U.S. Army given the  expanded testing required to ensure body armor meets U.S. Army  requirements. NIJ standards were not the subject of our review, but rather  Aberdeen Test Center’s application of the Army’s current solicitation’s  protocols during testing. Further, DOD acknowledged in its comments that  National Institute of Standards and Technology officials recommended  only one series of drops for clay calibration. However, DOD stated that it  will partner with the National Institute of Standards and Technology to  study procedures for clay calibration, to include repeated calibration  attempts, and document any appropriate procedural changes, which we  agree is a good step.
Based on our analyses as described in our report and in our above  responses to DOD’s comments, we believe there is sufficient evidence to  raise questions as to whether the issues we identified had an impact on  testing results. As a result, we continue to believe that it is necessary that  DOD allow an independent external expert to review these test results and  the overall effect of DOD’s deviations on those results before any armor is  fielded to military personnel. Without such an independent review, it is our  opinion that the First Article Testing results will remain questionable.  Consequently, we have added a matter for congressional consideration to  our report suggesting that Congress consider either directing DOD to  require that an independent external review of these body armor test  results be conducted or require that DOD officially amend its testing  protocols to reflect any revised test procedures and repeat First Article  Testing to ensure properly tested designs.
DOD partially concurred with our third recommendation to determine  whether those procedures that deviated from established testing protocols  during First Article Testing should be continued during future testing and  to change the established testing protocols to reflect those revised  procedures. DOD recognized the need to update testing protocols and  added that when the office of the Director of Operational Test and  Evaluation promulgates standard testing protocols across DOD, these  standards will address issues that we identified. As long as DOD  specifically addresses all the inconsistencies and deviations that we  observed prior to any future body armor testing, this would satisfy our  recommendation.
DOD stated that it partially concurs with our fourth recommendation to  evaluate and recertify the accuracy of the laser scanner to the correct  standard with all software modifications incorporated, based on the  results of the independent expert review of the First Article Testing  results. We also recommended that this process include a side-by-side  comparison of the laser’s measurement of back-face deformations and  those taken by digital caliper. DOD concurred with the concept of an  independent evaluation, but it did not concur that one is needed in this  situation because according to DOD its laser certification was sufficient.
We disagree that the laser certification was performed correctly. As  discussed in the body of our report and further in appendix II,  recertification of the laser is critical because (1) the laser was certified to  the wrong standard, (2) software modifications were added after the  certification of the laser, and (3) these modifications did change the way  the laser scanner measured back-face deformations. DOD did not  explicitly state whether it concurred with our recommendation for a side- by-side comparison of the laser scanner and the digital caliper in their  operational environment. We assert that such a study is important because  without it the Army and DOD do not know the effect the laser scanner may  have on the back-face deformation standard that has been used for many  years and was established with the intention of being measured with a  digital caliper. If the comparison reveals a significant difference between  the laser scanner and the digital caliper, DOD and the Army may need to  revisit the back-face deformation standard of its requirements with the  input of industry experts and the medical community.
DOD generally concurred with our fifth recommendation to conduct an  independent evaluation of the Aberdeen Test Center’s testing protocols,  facilities, and instrumentation and stated that such an evaluation would be  performed by a team of subject matter experts that included both DOD  and non-DOD members. We agree that in principal this approach meets the  intent of our recommendation as long as the DOD members of the  evaluation team are independent and not made up of personnel from those  organizations involved in the body armor testing such as office of the  Director of Operational Test and Evaluation, the Army Test and Evaluation  Command, or PEO Soldier. DOD’s comments and our specific responses to  them are provided in appendix II.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Defense, and the Secretary of the Army. In  addition, the report will be available at no charge on GAO’s Web site at  http://www.gao.gov. If you or your staff has any questions about this  report, please contact me at (202) 512-8365 or solisw@gao.gov. Contact   points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix III.
section_title: Appendix I: Scope and Methodology
paragraphs: Our review of body armor testing focused on testing conducted by the  Army in response to specific concerns raised by the House and Senate  Armed Services Committees and multiple members of Congress. During  our review, we were present during two rounds of testing of body armor  designs that were submitted in response to a May 2007-February 2008  Army contract solicitation. The first round of testing, called Preliminary  Design Model testing, was conducted from February 2008 through June  2008 with the objective of determining whether designs submitted under  the contract solicitation met the required ballistic performance  specifications and were eligible for contract award. The second round of  testing, called First Article Testing, was conducted between November  2008 and December 2008 on the body armor designs that passed the  Preliminary Design Model testing. Both tests were conducted at Aberdeen  Proving Grounds in Aberdeen, Md., and were performed by Aberdeen Test  Center. During the course of our review, we observed how the Army  conducted its body armor testing and compared our observations with the  established body armor testing protocols. We did not verify the accuracy  of the Army’s test data and did not provide an expert evaluation of the  results of testing. To understand the practices the Army used and the  established testing protocols we were comparing the practices with, we  met with and/or obtained data from officials from the Department of  Defense (DOD) organizations and the industry experts listed in table 1:  To determine the degree to which the Army followed established testing  protocols during the Preliminary Design Model testing of body armor  designs, we were present and made observations during the entire period  of testing, compared our observations with established testing protocols,  and interviewed numerous DOD and other experts about body armor  testing. We observed Army testers as they determined whether designs  met the physical and ballistics specifications described in the contract  solicitation, and as encouraged by Aberdeen Test Center officials, we  observed the ballistics testing from inside a viewing room equipped with  video and audio connections to the firing lanes. We also were present and  observed the physical characterization of the test items and visited the  environmental conditioning chambers, the weathering chamber, and the X- ray facility. We were at Aberdeen Test Center when the designs were  delivered for testing on February 7, 2008, and were on-site every day of  physical characterization, which comprises the steps performed to  determine whether each design meets the required weight and  measurement specifications. We systematically recorded our observations  of physical characterization on a structured, paper data-collection  instrument that we developed after consulting with technical experts from  Program Executive Office (PEO) Soldier before testing started. We were  also present for every day except one of the ballistics testing, observing  and collecting data on approximately 80 percent of the tests from a video  viewing room that was equipped with an audio connection to each of the  three firing lanes. To gather data from the day that we were not present to  observe ballistic testing, we viewed that day’s testing on video playback.  We systematically recorded our observations of ballistics testing using a  structured, electronic data-collection instrument that we developed to  record relevant ballistic test data—such as the shot velocity, penetration  results, and the amount of force absorbed (called “back-face  deformation”) by the design tested. Following testing, we supplemented  the information we recorded on our data collection instrument with some  of the Army’s official test data and photos from its Vision Digital Library  System. We developed the data collection instrument used to collect  ballistics testing data by consulting with technical experts from Program  Executive Office Soldier and attending a testing demonstration at  Aberdeen Test Center before Preliminary Design Model testing began.  After capturing the Preliminary Design Model testing data in our data  collection instruments, we compared our observations of the way the  Aberdeen Test Center conducted testing with the testing protocols that  Army officials told us served as the testing standards at the Aberdeen Test  Center. According to these officials, these testing protocols comprised the  (1) test procedures described in the contract solicitation announcement’s  purchase descriptions and (2) Army’s detailed test plans and Test  Operating Procedure that serve as guidance to the Aberdeen Test Center  testers and that were developed by the Army Test and Evaluation  Command and approved by Program Executive Office Soldier, the office  of the Director of Operational Test and Evaluation, the Army Research  Labs, and cognizant Army components. We also reviewed National  Institute of Justice testing standards because Aberdeen Test Center  officials told us that, although Aberdeen Test Center is not a National  Institute of Justice-certified testing facility, they have made adjustments to  their procedures based on those standards and consider them when  evaluating Aberdeen Test Center’s test practices. Regarding the edge shot  locations for the impact test samples, we first measured the area of  intended impact on an undisturbed portion of the test item on all 56 test  samples after the samples had already been shot. The next day we had  Aberdeen Test Center testers measure the area of intended impact on a  random sample of the impact test samples to confirm our measurements.  Throughout testing we maintained a written observation log and compiled  all of our ballistic test data into a master spreadsheet. Before, during, and  after testing, we interviewed representatives from numerous Army  entities—including the Assistant Secretary of the Army for Acquisition,  Technology and Logistics; Aberdeen Test Center; Developmental Test  Command; Army Research Laboratories; and Program Executive Office  Soldier—and also attended Integrated Product Team meetings.
To determine the degree to which the Army followed established testing  protocols during First Article Testing of the body armor designs that  passed Preliminary Design Model testing, we were present and made  observations during the entire period of testing, compared our  observations with established testing protocols, and interviewed  numerous DOD and industry experts about body armor testing. As during  Preliminary Design Model testing, we observed Army testers as they  determined whether designs met the physical and ballistics specifications  described in the contract solicitation. However, different from our review  of Preliminary Design Model testing, we had access to the firing lanes  during ballistic testing. We also still had access to the video viewing room  used during Preliminary Design Model testing, so we used a bifurcated  approach of observing testing from both the firing lanes and the video  viewing room. We were present for every day except one of First Article  Testing—from the first day of ballistics testing on November 11, 2008, until  the final shot was fired on December 17, 2008. We noted the weights and  measures of plates during physical characterization on the same data  collection instrument that we used during Preliminary Design Model  testing. For the ballistics tests, we revised our Preliminary Design Model  testing data collection instrument so that we could capture data while in  the firing lane—data that we were unable to confirm first hand during  Preliminary Design Model testing. For example, we observed the pre-shot  measurements of shot locations on the plates and the Aberdeen Test  Center’s method for recording data and tracking the chain of custody of  the plates; we also recorded the depth of the clay calibration drops (the  series of pre-test drops of a weight on clay that is to be placed behind the  plates during the shots), the temperature of the clay, the temperature and  humidity of the firing lane, the temperatures in the fluid soak conditioning  trailer, and the time it took to perform tests. We continued to record all of  the relevant data that we had recorded during Preliminary Design Model  testing, such as the plate number, type of ballistic subtest, the charge  weight of the shot, the shot velocity, the penetration results, and the back- face deformation. Regarding the new laser arm that Aberdeen Test Center  acquired to measure back-face deformation during First Article Testing,  we attended a demonstration of the arm’s functionality performed by  Aberdeen Test Center and also acquired documents related to the laser  arm’s certification by Army Test, Measurement, and Diagnostic Equipment  activity. With a GAO senior methodologist and a senior technologist, we  made observations related to Aberdeen Test Center’s methods of handling  and repairing clay, calibrating the laser guide used to ensure accurate  shots, and measuring back-face deformation. Throughout testing we  maintained a written observation log and compiled all of our ballistic test  data into a master spreadsheet. Following testing, we supplemented the  information we recorded on our data collection instrument with some of  the Army’s official test data and photos from its Vision Digital Library  System to complete our records of the testing. After capturing the testing  data in our data collection instruments, we compared our observations of  the way Aberdeen Test Center conducted testing with the testing protocols  that Army officials told us served as the testing standards at the Aberdeen  Test Center. In analyzing the potential impact of independent variables on  testing, such as the potential impact of the November 13th rain on the clay,  we conducted statistical tests including chi-square and Fisher’s Exact Test  methods to accommodate small sample sizes. Before, during, and after  testing, we interviewed representatives from numerous Army agencies,  including Aberdeen Test Center, Developmental Test Command, Army  Research Laboratories, and Program Executive Office Soldier. We also  spoke with vendor representatives who were present and observing the  First Article Testing, as well as with Army and industry subject matter  experts.
We conducted this performance audit from July 2007 through October  2009 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Defense
subsections: section_title: GAO Comments
paragraphs: 1. The Department of Defense (DOD) stated that undertakings of this  magnitude are not without flaws and that what was most important was  fielding body armor plates that defeated the threat. While DOD may have  identified some flaws that may not be serious enough to call the testing  results into question, several of the deviations to the testing protocols that  we observed do call the testing results into question for the reasons stated  in our report. An independent expert has not evaluated the impact of these  deviations on the test results and, until such a study is conducted, DOD  cannot be assured that the plates that passed testing can defeat the threat.  DOD also noted several actions DOD and the Army have taken to improve  procedures associated with body armor testing. Our responses to these  actions are included in comments 2 through 6.  2. The office of the Director of Operational Test and Evaluation’s efforts to  respond to members of the Armed Services Committees and to address  issues raised by the Department of Defense Inspector General were  outside the scope of our audit. Therefore, we did not validate the  implementation of the actions DOD cited or evaluate their effectiveness in  improving test procedures. With regard to the office of the Director of  Operational Test and Evaluation’s establishing a policy to conduct First  Article Testing at government facilities, using a government facility to  conduct testing may not necessarily produce improved test results.  3. Regarding the office of the Director of Operational Test and Evaluation’s  oversight of testing, the office of the Director of Operational Test and  Evaluation led the Integrated Product Team and approved the test plans.  However, while we were present at the Aberdeen Test Center during  Preliminary Design Model testing and First Article Testing, we did not  observe on-site monitoring of the testing by the office of the Director of  Operational Test and Evaluation staff beyond incidental visits during VIP  events and other demonstrations.  4. Regarding the procedures and policies DOD stated were implemented  by the Army Test and Evaluation Command to improve testing:    Only two of the test ranges were completed prior to Preliminary  Design Model testing. Two additional test ranges were completed after  Preliminary Design Model testing.
Regarding the certification of the laser scanner measurement device,  as noted in our report, the Army had not adequately certified that it  was an appropriate tool for body armor testing (see our comment 12).
The Army’s Test Operating Procedure was not completed or  implemented until after Preliminary Design Model testing.
New clay conditioning chambers inside each test range were not  constructed until after all testing was completed (see our comment  13).
The improved velocity measurement accuracy study was not  conducted until after all testing was completed.
Regarding the implementation of electronic data collection and  processing for body armor testing, as stated in our report, we observed  that not all data are electronically collected. Many types of data are  manually collected and are later converted to electronic data storage.  5. Regarding Program Executive Office (PEO) Soldier’s efforts to improve  the acquisition of personal protection equipment:    The contract solicitation allowed all prospective body armor  manufacturers to compete for new contracts.
We observed that PEO Soldier did transfer expertise and experience to  support Army Acquisition Executive direction that all First Article  Testing and lot-acceptance testing be conducted by the Army Test and  Evaluation Command.
The task force that focused on soldier protection was not initiated  until February 2009, after all Preliminary Design Model testing and  First Article Testing was completed.
According to Army officials, PEO Soldier instituted a non-destructive  test capability that became operational after Preliminary Design Model  testing, but prior to First Article Testing.
PEO Soldier’s personal protection evaluation process was described in  our previous report—GAO-07-662R. Although we recognized the  strength of PEO Soldier’s personal protection evaluation process in  our earlier report, not all the protections that were in place at that time  remain in place. For example, the requirement that testing be  conducted at a National Institute of Justice (NIJ)-certified facility was  waived.  6. DOD stated that many of the actions by Army Test and Evaluation  Command and PEO Soldier were initiated and improved upon during the  course of our review. However, as discussed above, several of these  actions were initiated before and during testing, but many of them were  not completed until after testing was completed.  7. DOD and the Army stated that Preliminary Design Model testing had  achieved its objective to identify those vendor designs that met the  performance objectives stated in PEO Soldier’s purchase description and  that “it is incorrect to state that ‘at least two’ of the preliminary design  models should have failed as they passed in accordance with the modified  solicitation.” We disagree with these statements. As stated in our report,  the most consequential of the deviations from testing protocols we  observed involved the measurement of back-face deformation, which did  affect final test results. According to original testing protocols, back-face  deformation was to be measured at the deepest point of the depression in  the clay backing. This measure indicates the most force that the armor  will allow to be exerted on an individual struck by a bullet. According to  Army officials, the deeper the back-face deformation measured in the clay  backing, the higher the risk of internal injury or death. DOD and the Army  now claim that these solutions passed in accordance with the modified  solicitation, which overlooks the fact that the reason the solicitation had  to be modified was that Army testers deviated from the testing protocols  laid out in the purchase descriptions and did not measure back-face  deformation at the deepest point. DOD and the Army also stated in their  response that they decided to use the point of aim because they  determined it was an accurate and repeatable process. Yet in DOD’s  detailed comments regarding edge shot locations, DOD acknowledged that  there were “potential variances between the actual aim point and impact  point during testing.” Army Research Laboratory and NIJ-certified  laboratories use the benchmark process of measuring back-face  deformation at the deepest point, not at the point of aim.
As set forth in our report, at least two solutions passed Preliminary Design  Model testing that would have failed if back-face deformation had been  measured to the deepest point. This statement came directly from  Aberdeen Test Center officials during a meeting in July 2008, where they  specifically told us which two solutions would have failed. We said “at  least” two because Army testers did not record deepest point back-face  deformation data for the first 30 percent of testing, and therefore there  could be more solutions that would have failed had the deepest point been  measured during this first portion of the test. Because the Army did not  measure back-face deformation to the deepest point, it could not identify  whether these two solutions in particular and all the solutions in general  met performance requirements. As a result, Army could not waive First  Article Testing for successful candidates and was forced to repeat the test  to ensure that all solutions did indeed meet requirements. By repeating  testing, the Army incurred additional expense and further delays in  fielding armor from this solicitation to the soldiers. During the course of  our audit, the Army also acknowledged that the Preliminary Design Model  testing did not meet its objective because First Article Testing could not be  waived without incurring risk to the soldiers.
DOD and the Army stated that, upon discovery of the back-face  deformation deviation from the testing protocols described in the  purchase descriptions, the Army stopped testing. The Army’s Contracting  Office was informed of this deviation through a series of questions posed  by a vendor who was present at the Vendor Demonstration Day on  February 20, 2008. This vendor sent questions to the Contracting Office on  February 27 asking whether testers were measuring at the aim point or at  the deepest point. This vendor also raised questions about how damage to  the soft pack would be recorded and about the location of edge shots.  Based on our observations, all of these questions involved issues where  Army testers deviated from testing protocols and are discussed in our  responses to subsequent comments. The Army did not respond until  March 19 and replied that its test procedures complied with solicitation  requirements. It was not until Army leadership learned of the vendor’s  questions and of the deviation in measuring back-face deformation that  testing was finally halted on March 27, a full month after the issue came to  the Army Test and Evaluation Command’s attention.  8. DOD stated that in 2007, prior to the initiation of Preliminary Design  Model testing, the Army Test and Evaluation Command, the office of the  Director of Operational Test and Evaluation, and Army leadership all  agreed that First Article Testing would be conducted as part of the Army’s  body armor testing. However, DOD did not provide any documentation  dated prior to April 2008—that is, prior to the discovery of the back-face  deformation deviation—that suggested that DOD intended to conduct First  Article Testing following Preliminary Design Model testing. In July 2008,  the Army Test and Evaluation Command and PEO Soldier stated in official  written responses to our questions regarding Preliminary Design Model  testing that the conduct of First Article Testing became essential following  Preliminary Design Model testing because of the Army’s measuring back- face deformation at the point of aim as opposed to at the deepest point of  deformation. In fact, because of this deviation, DOD could not waive First  Article Testing as originally planned and was forced to conduct  subsequent tests to verify that the designs that had passed Preliminary  Design Model testing met testing requirements. DOD asserted that a multi- phase concept including Preliminary Design Model testing, First Article  Testing, and extended ballistic testing to support the development of an  improved test standard was briefed to a congressional member and  professional staff on November 14, 2007. We were present at this  November 14 test overview and strategy/schedule briefing and noted that  it did not include plans for First Article Testing to be performed in  addition to Preliminary Design Model testing. Excerpts from the slides  briefed that day showed Preliminary Design Model (Phase 1) testing and a  subsequent ballistic and suitability testing (Phase 2). As indicated in the  slides (see fig. 7 and fig. 8) from that November 14 briefing, the Phase 2  test was designed to test the form, fit, and function of those solutions that  had passed Preliminary Design Model testing as well as the ballistic  statistical confidence tests. According to information we obtained, Phase  2 was never intended to be First Article Testing and was to have no impact  on whether or not a solution received a contract.
It was not until after the back-face deformation deviation was discovered  that briefing slides and other documentation on test plans and schedules  started describing First Article Testing as following Preliminary Design  Model testing. For example, as stated by DOD in its comments, the  October 2008 briefing to a congressional member and professional staff  clearly showed First Article Testing as following Preliminary Design Model  testing (Phase 1) and preceding Phase 2. Therefore, it is not clear why  DOD’s test plan briefings would make no mention of a First Article Testing  prior to the back-face deformation measurement deviation while including  First Article Testing in subsequent briefings if the plan had always been to  conduct both Preliminary Design Model testing and First Article Testing.  Furthermore, it is not clear why DOD would intentionally plan at the start  of testing to repeat Preliminary Design Model testing (which was supposed  to be performed in accordance with the First Article Testing protocol)  with an identical test (First Article Testing) given that it has been the  Army’s practice to use such Preliminary Design Model testing to meet First  Article Testing requirements – a practice that was also supported by the  DOD Inspector General and the Army Acquisition Executive after an audit  of the Army’s body armor testing program. DOD also stated that First  Article Testing waivers were not permitted under the body armor  solicitation. However, the solicitation and its amendments are unclear as  to whether waivers of First Article Testing would be permitted.  Nonetheless, in written answers to questions we posed to the Army in July  2008, the Army Test and Evaluation Command and PEO Soldier in a  combined response stated that due to the fact that back-face deformation  was not measured to the deepest point of penetration during Phase I tests,  there would be no waivers of First Article Testing after the contract  award.
DOD also stated that it and the Army concluded that First Article Testing  had achieved its objective of verifying that contracted vendors could  produce, in a full-rate capacity, plates that had passed Preliminary Design  Model testing. DOD further stated that it is incorrect to say that First  Article Testing did not meet its objective and it is incorrect to assert that  three of five vendor designs should have failed First Article Testing.  However, our analysis showed that two solutions that passed First Article  Testing would have failed if back-face deformations had not been rounded  and had been scored as they were during Preliminary Design Model  testing. The third solution that passed would have failed if Army testers  had correctly scored a shot result as a complete penetration in accordance  with the definition of a complete penetration in the purchase description,  rather than as a partial penetration. Because questions surround these  scoring methods and because DOD and the Army cannot confidently  identify whether these vendors can mass produce acceptable plates, we  restate that First Article Testing may not have achieved its objective. See  comments 12, 10, and 11 regarding DOD’s statements about the  certification of the laser scanning equipment, the rounding of back-face  deformations, and the Aberdeen Test Center’s scoring procedures,  respectively.
We agree with DOD that an open dialog with the DOD Inspector General,  external test and technology experts, and us will improve the current body  armor testing. However, we disagree with DOD’s statement that NIJ- certified laboratories lack expertise to provide reliable information on  body armor testing issues. Before the current solicitation, the Army relied  on these NIJ-certified laboratories for all body armor source selection and  lot acceptance tests. The Marine Corps also conducts source selection  tests at these facilities. As these independent laboratories have performed  numerous tests for the Army conducted in accordance with First Article  Testing protocol, we assert that the credentials of these laboratories  warrant consideration of their opinions on body armor testing matters.  9. DOD did not concur with our recommendation for an independent  evaluation of First Article Testing results before any armor is fielded to  soldiers because the First Article Testing achieved its objectives. We  disagree with DOD’s position that First Article Testing and Preliminary  Design Model testing achieved their objectives because we found  numerous deviations from testing protocols that allowed solutions to pass  testing that otherwise would have failed. Due to these deviations, the  majority of which seem to make the testing easier to pass and favor the  vendors, we continue to believe that it is necessary to have an independent  external expert review the results of First Article Testing and the overall  effect of DOD’s deviations on those results before the plates are fielded.  An independent observer, external to DOD, is best suited to determine the  overall impact of DOD’s many deviations during the testing associated  with this solicitation. Consequently, we have added a matter for Congress  to consider directing DOD to either conduct this external review or direct  that DOD officially amend its testing protocols to reflect any revised test  procedures and repeat First Article Testing.  10. DOD did not concur with our recommendation that the practice of  rounding down back-face deformations should be reviewed by external  experts because the practice has been used historically by NIJ-certified  laboratories. Although DOD acknowledged that the practice of rounding is  not adequately described in the testing protocols, it stated that rounding is  permitted under American Society for Testing and Materials (ASTM) E-29.  The purchase descriptions (attachments 01 and 02 of the solicitation)  referenced five ASTM documents, but ASTM E-29 is not referenced and  therefore is not part of the protocol. The detailed test plans state that  solutions shall incur a penalty on deformations greater than 43  millimeters, and the Army is correct that neither the purchase description  nor the detailed test plans provide for rounding. During Preliminary  Design Model testing, Army testers measured back-face deformations to  the hundredths place and did not round. Any deformation between 43.00  and 43.50 received a penalty. During First Article Testing, deformations in  this range were rounded down and did not incur a penalty, so the decision  to round effectively changed the standard in favor of the vendors. Two  solutions passed First Article Testing that would have failed if back-face  deformations had been scored without rounding as they were during  Preliminary Design Model testing. We recognize that there are other  factors, such as the fact that the new laser scanner may overstate back- face deformations that might justify the decision to round down back-face  deformations. However, as a stand-alone event, rounding down  deformations did change the standard in the middle of the solicitation  between Preliminary Design Model testing and First Article Testing. That  is why it is important for an independent external expert to review the  totality of the test and the Army’s deviations from testing protocols to  determine the actual effect of this and other deviations.  11. Regarding the incorrect scoring of a complete penetration as a partial  penetration, DOD stated that the first layer of soft armor behind the plate  serves as a witness plate during testing. If that first layer of soft armor is  not penetrated, as determined by the breaking of threads on that first layer  of soft armor, the test shot is not scored as a complete penetration in  accordance with the PEO Soldier’s scoring criteria. However, DOD’s  position is not consistent with the established testing protocols as  evidenced by the following:  (1) We did not observe the use of and the testing protocols do not  require the use of a witness plate during testing to determine if  a penetration occurred; and   (2) The testing protocols do not state that “the breaking of  threads” is the criterion for determining a penetration.
The language of the testing protocols, not undocumented criteria, should  be used in scoring and determining penetration results. The criteria for  scoring a penetration are found in the current solicitation’s protocols.  Paragraph 6.6, of each of the purchase descriptions state, under  “Definitions: Complete Penetration (CP) for Acceptance Testing-- Complete penetrations have occurred when the projectile, fragment of the  projectile, or fragment of the armor material is imbedded or passes into  the soft under garment used behind the protective inserts plates” (ESAPIs  or XSAPIs).
Our multiple observations and thorough inspection of the soft armor in  question revealed that black-grayish particles had penetrated at least three  Kevlar layers as evidenced by their frayed, fuzz-like and separated  appearance to the naked eye. The black-grayish particles were stopped by  the fourth Kevlar layer. DOD acknowledged that figure 6 of our report  appears to show evidence of a perforation on the rear of the test plate in  question and that the Aberdeen Test Center’s subject matter expert found  dust particles. These particles are fragments of the projectile or fragments  of the armor material that were imbedded and indeed passed into the soft  undergarment used behind the protective insert; therefore, the shot should  have been ruled a complete penetration according to the testing protocols,  increasing the point penalties and causing the design to fail First Article  Testing.
DOD’s comments stated that we acknowledged there were no broken  threads on the first layer of the soft armor. We made no such comment  and this consideration is not relevant as the requirement for broken fibers  is not consistent with the written testing protocols as we have stated. Of  consequence, DOD and Army officials acknowledged that the requirement  for broken fibers was not described in the testing protocols or otherwise  documented.
In addition to the DOD acknowledgement that an Aberdeen Test Center  subject matter expert found particles on the soft body armor, more  convincing evidence is the picture of the subject plate. Figure 6 of our  report clearly shows the tear in the fibers that were placed behind the  plate in question allowing the penetration of the particles found by the  Aberdeen Test Center subject matter expert. These particles can only be  fragments of the projectile or fragments of the armor material that passed  into the soft under garment used behind the protective inserts (plates),  confirming our observations of the event and the subsequent incorrect  scoring. The shot should have been scored a complete penetration, and the  penalty incurred would have caused the design in question to fail First  Article Testing.  12. DOD did not concur with our recommendation that the use of the laser  scanner needs to be reviewed by experts external to DOD due to the lack  of a full evaluation of the scanner’s accuracy to measure back-face  deformations, to include an evaluation of the software modifications and  operation under actual test conditions. DOD asserted that the laser  scanner measurement device provides a superior tool for providing  accurate, repeatable, defensible back-face deformation measurements to  the deepest point of depression in the clay. We agree that once it is  properly certified, tested, and evaluated, the laser may eliminate human  errors such as incorrectly selecting the location of the deepest point or  piercing the clay with the sharp edge of the caliper and making the  depression deeper. However, as we stated, the Army used the laser  scanner as a new method to measure back-face deformation without  adequately certifying that the scanner could function: (1) in its operational  environment, (2) at the required accuracy, (3) in conjunction with its  software upgrades, and (4) without overstating deformation  measurements.
DOD asserted that the software upgrades did not affect the measurement  system of the laser scanner and that these software changes had no effect  on the physical measurement process of the back-face deformation  measurement that was validated through the certification process. The  software upgrades were added after the certification and do include  functions to purposely remove spikes and other small crevices on the clay  and a smoothing algorithm that changed back-face deformation  measurements. We have reviewed these software functions and they do in  fact include calculations that change the back-face deformation  measurement taken. Furthermore, Army officials told us that additional  upgrades to the laser scanner were made after First Article Testing by  Aberdeen Test Center to correct a software laser malfunction identified  during the subsequent lot acceptance testing of its plates. According to  these officials, this previously undetected error caused an overstatement  of the back-face deformation measurement taken by several millimeters,  calling into question all the measurements taken during First Article  Testing. Also, vendors have told us that they have conducted several  studies that show that the laser scanner overestimates back-face  deformation measurements by about 2 millimeters as compared with  measurements taken by digital caliper, thereby over-penalizing vendors’  designs and causing them to fail lot acceptance testing. Furthermore, the  laser scanner was certified to an accuracy of 1.0 millimeters, but section  4.9.9.3 of the purchase descriptions requires a device capable of measuring  to an accuracy of ±0.1 millimeters. Therefore, the laser does not meet this  requirement making the certification invalid. The laser scanner is an  unproven measuring device that may reflect a new requirement because  the back-face deformation standards are based on measurements obtained  with a digital caliper. This raises concerns that results obtained using the  laser scanner may be more inconsistent than those obtained using the  digital caliper. As we stated in the report, the Aberdeen Test Center has  not conducted a side-by-side test of the new laser scanner used during  First Article Testing and the digital caliper previously used during  Preliminary Design Model testing. Given the discrepancies on back-face  deformation measurements we observed and the overstating of the back- face deformation alleged by the vendors, the use of the laser is still called  into question. Thus, we continue to support our recommendation that  experts independent of DOD review the use of the laser during First  Article Testing and that a full evaluation of the laser scanner is imperative  to ensure that the tests are repeatable and can be relied upon to certify  procurement of armor plates for our military personnel based on results of  body armor testing at the Aberdeen Test Center using the laser scanner.
Lastly, DOD stated that the laser scanner is used by the aeronautical  industry; however, the Army Test and Evaluation Command officials told  us that the scanner had to be customized for testing through various  software additions and mounting customizations to mitigate vibrations and  other environmental factors. These software additions and customizations  change the operation of the scanner.  13. DOD does not concur with our recommendation that experts examine,  among other items, “the exposure of clay backing material to rain and  other outside environmental conditions as well as the effect of high oven  temperatures during storage and conditioning,” because it believes that  such conditions had no impact upon First Article Testing results. As  detailed in the report, we observed these conditions at different points  throughout the testing period. Major variations in materials preparation  and testing conditions such as exposure to rain and/or violations of testing  protocols merit consideration when analyzing the effectiveness and  reliability of First Article Testing.
As one specific example, we described in this report statistically  significant differences between the rates of failure in response to one  threat on November 13 and the failure rates on all other days of testing but  do not use the statistical analysis as the definitive causal explanation for  such failure. We observed one major environmental difference in testing  conditions that day, the exposure of temperature-conditioned clay to  heavy, cold rain in transit to the testing site. After experts confirmed that  such variation might be one potential factor relating to overall failure rates  on that day, we conducted statistical tests to assess whether failures rates  were different on November 13 compared to other dates..
Our assertion that the exposure of the clay to rain may have had an impact  on test results is based not solely on our statistical analysis of test results  that day; rather, it is also based on our conversations with industry  experts, including the clay manufacturer, and on the fact that we  witnessed an unusually high number of clay calibration failures during  testing that comprised plate designs of multiple vendors, not just the one  design that DOD points to as the source for the high failure rate. We  observed that the clay conditioning trailer was located approximately 25  feet away from the entrance to the firing lane. The clay blocks, weighing in  excess of 200 lbs., were loaded face up onto a cart and then a single  individual pulled the cart over approximately 25 feet of gravel to the firing  lane entrance. Once there, entry was delayed because the cart had to be  positioned just right to get through the firing lane door. Army testers  performed all of this without covering the clay to protect it from the rain  and the cold, and once inside the clay had significant amounts of water  collected on it.
With respect to the unusually high number of clay calibration failures on  November 13, there were seven clay calibration drops that were not within  specifications. Some of these failed clay boxes were discarded in  accordance with the testing protocols; however, others were repaired, re- dropped, and used if they had passed the second drop series. These  included one plate that was later ruled a no-test and three plates for which  the first shot yielded a catastrophic back-face deformation. These were the  only three first-shot catastrophic back-face deformations during the whole  test, and they all occurred on the same rainy day and involved two  different solutions, not just the one that DOD claims performed poorly.
The failure rates of plates as a whole, across all plate designs, were very  high this day, and the failures were of both the complete penetration and  the back-face deformation variety. Water conducts heat approximately 25  times faster than air, which means the water on the surface cooled the clay  considerably faster than the clay would have cooled by air exposure alone.  Moreover, Army testers lowered the temperature of the clay conditioning  trailers during testing on November 13 and told us that the reason was that  the ovens and clay were too hot.  This is consistent with what Army  subject matter experts and other industry experts told us— that the  theoretical effect of having cold rain collecting on hot clay may create a  situation where the clay is more susceptible to both complete penetrations  because of the colder, harder top layer and to excessive back-face  deformations because of the overheated, softer clay beneath the top layer.  Finally, the clay manufacturer told us that, although this is an oil-based  clay, water can affect the bonding properties of the clay, making it more  difficult for wet clay to stick together. This is consistent with what we  observed on November 13. After the first shot on one plate, as Army  testers were removing the plate from the clay in order to determine the  shot result, we observed a large chunk of clay fall to the floor. This clay  was simply swept off to the side by the testers. In another instance, as  testers were repairing the clay after the calibration drop, one of the testers  pulled a long blade over the surface of the clay to smooth it. When he hit  the spot where one of the calibration drops had occurred and the clay had  been repaired, the blade pulled up the entire divot and the testers had to  repair the clay further.
Regarding our use of no-test data, we were strict in the instances where  we used this data, see our comment 24.
DOD stated that it was the poor performance of one solution in particular  that skewed the results for this day and that this solution failed 70 percent  of its shots against Threat D during First Article Testing. DOD’s statistic is  misleading. This solution failed 100 percent of its shots (6 of 6) on  November 13, but only 50 percent for all other test days (7 of 14). Also, the  fact that this solution managed to pass the Preliminary Design Model  testing but performed so poorly during First Article Testing raises  questions about the repeatability of DOD’s and the Army’s test practices.  Finally, DOD’s own analysis confirms that two of the four solutions tested  on November 13 performed at their worst level in the test on that day. If  the one solution whose plate was questionably ruled a no-test on this day  is included in the data, then three of the four solutions performed at their  worst level in the test on this day.
DOD said that after testing Aberdeen Test Center completed the planned  installation of new clay conditioning chambers inside the test ranges  precluding any external environmental conditioning interacting with the  clay. We believe it is a step in the right direction that the Aberdeen Test  Center has corrected this problem for future testing, but we continue to  believe that an external entity needs to evaluate the impact of introducing  this new independent variable on this day of First Article Testing.  14. DOD concurred that it should establish a written standard for  conducting clay calibration drops but non-concurred that failed blocks  were used during testing. DOD asserted that all clay backing material used  during testing passed the calibration drop test prior to use. We disagree  with this position because the calibration of the clay required by the  testing protocols calls for “a series of drops,” meaning one series of three  drops, not multiple series of three drops as we observed on various  occasions. DOD stated that, as a result of our review and the concerns  cited in our report, the Aberdeen Test Center established and documented  a revised procedure stating that only one repeat of calibration attempt can  be made and, if the clay does not pass calibration upon the second  attempt, it is reconditioned for later use and a new block of clay is  substituted for calibration. Based on the testing protocols, this is still an  incorrect procedure to ensure the proper calibration of the clay prior to  shooting. The testing protocols do not allow for a repeat series of  calibration drops. DOD also says that, upon completion of testing under  the current Army solicitation and in coordination with the National  Institute of Standards and Technology, the office of the Director of  Operational Test and Evaluation and the Army will review the procedures  for clay calibration to include repeated calibration attempts and will  document any appropriate procedural changes. DOD goes on to say that  the NIJ standard as verified by personnel at the National Institute of  Standards and Technology does not address specifically the issue of  repeating clay calibration tests. However, the Aberdeen Test Center’s  application of the Army’s current solicitation’s protocols during testing,  and not the NIJ standards, was the subject of our review.
In its comments, DOD acknowledged that the National Institute of  Standards and Technology officials recommend only one series of drops  for clay calibration, but the Aberdeen Test Center did multiple drops  during testing. We are pleased that DOD has agreed to partner with the  National Institute of Standards and Technology to conduct experiments to  improve the testing community’s understanding of clay performance in  ballistic testing, but these conversations and studies in our opinion should  have occurred prior to testing, not after, as this deviation from testing  protocols calls the tests results into question. We reassert that an external  entity needs to evaluate the impact of this practice on First Article Testing  results.  15. DOD partially concurred with our recommendation and agreed that  inconsistencies were identified during testing; however, DOD asserted that  the identified inconsistencies did not alter the test results. As stated in our  response to DOD’s comments on our first recommendation, we do not  agree. Our observations clearly show that (1) had the deepest point been  used during Preliminary Design Model testing, two designs that passed  would have failed and (2) had the Army not rounded First Article Testing  results down, two designs that passed would have failed. Further, if the  Army had scored the particles (which in their comments to this report  DOD acknowledges were imbedded in the shoot pack behind the body  armor) according to the testing protocols, a third design that passed First  Article Testing would have failed. In all, four out of the five designs that  passed Preliminary Design Model testing and First Article Testing would  have failed if testing protocols had been followed.  16. DOD partially concurred with our recommendation that, based on the  results of the independent expert review of the First Article Testing  results, it should evaluate and recertify the accuracy of the laser scanner  to the correct standard with all software modifications incorporated and  include in this analysis a side-by-side comparison of the laser  measurements of the actual back-face deformations with those taken by  digital caliper to determine whether laser measurements can meet the  standard of the testing protocols. DOD maintains that it performed an  independent certification of the laser measurement system and process  and that the software changes that occurred did not affect the  measurement system in the laser scanner. However, as discussed in  comment 12, we do not agree that an adequate, independent certification  of the laser measurement system and process was conducted. Based on  our observations, we continue to assert that the software changes added  after certification did affect the measurement system in the laser.  17. DOD partially concurred with our recommendation for the Secretary of  the Army to provide for an independent peer review of the Aberdeen Test  Center’s body armor testing protocols, facilities, and instrumentation. We  agree that a review conducted by a panel of external experts that also  includes DOD members could satisfy our recommendation. However, to  maintain the independence of this panel, the DOD members should not be  composed of personnel from those organizations involved in the body  armor testing (such as the office of the Director of Operational Test and  Evaluation, the Army Test and Evaluation Command, or PEO Soldier.  18. DOD stated that Aberdeen Test Center had been extensively involved  in body armor testing since the 1990s and has performed several tests of  body armor plates. We acknowledge that Aberdeen Test Center had  conducted limited body armor testing for the initial testing on the  Interceptor Body Armor system in the 1990s and have clarified the report  to reflect that. However, as acknowledged by DOD, Aberdeen Test Center  did not perform any additional testing on that system for PEO Soldier  since the 1990s and this lack of experience in conducting source selection  testing for that system may have led to the misinterpretations of testing  protocols and deviations noted on our report. According to a recent Army  Audit Agency report, NIJ testing facilities conducted First Article Testing  and lot acceptance testing for the Interceptor Body Armor system prior to  this current solicitation. Another reason Aberdeen Test Center could not  conduct source selection testing was that in the past Aberdeen Test Center  lacked a capability for the production testing of personnel armor systems  in a cost-effective manner; the test facilities were old and could not  support test requirements for a temperature- and humidity-controlled  environment and could not provide enough capacity to support a war- related workload. The Army has spent about $10 million over the last few  years upgrading the existing facilities with state-of-the-art capability to  support research and development and production qualification testing for  body armor, according to the Army Audit Agency. Army Test and  Evaluation Command notes that there were several other tests between  1997 and 2007, but according to Army officials these tests were customer  tests not performed in accordance with a First Article Testing protocol.  For example, the U.S. Special Operations Command test completed in May  2007 and cited by DOD was a customer test not in accordance with First  Article Testing protocol. The Aberdeen Test Center built new lanes and  hired and trained contractors to perform the Preliminary Design Model  testing and First Article Testing.  19. DOD stated that, to date, it has obligated about $120 million for XSAPI  and less than $2 million for ESAPI. However, the value of the 5-year  indefinite delivery/indefinite quantity contracts we cited is based on the  maximum amount of orders of ESAPI/XSAPI plates that can be purchased  under these contracts. Given that the Army has fulfilled the minimum  order requirements for this solicitation, the Army could decide to not  purchase additional armor based on this solicitation and not incur almost  $7.9 billion in costs.
DOD stated in its response that there are only three contracts. However,  the Army Contracting Office told us that there were four contracts  awarded and provided those contracts to us for our review. Additionally,  we witnessed four vendors participating in First Article Testing, all of  which had to receive contracts to participate. It is unclear why the Army  stated that there were only three contracts.  20. DOD is correct that there is no limit or range specified for the second  shot location for the impact subtest. However, this only reinforces that the  shot should have been aimed at 1.5 inches, not at 1.0 inch or at various  points between 1.0 inch and 1.5 inches. It also does not explain why the  Army continued to mark plates as though there were a range for this shot.  Army testers would draw lines at approximately 0.75 inches for the inner  tolerance and 1.25 inches for the outer tolerance of ESAPI plates. They  drew lines at approximately 1.0 inch for the inner tolerance and 1.5 inches  for the outer tolerance of XSAPI plates. We measured these lines for every  impact test plate and also had Army testers measure some of these lines to  confirm our measurements. We found that of 56 test items, 17 were  marked with shot ranges wholly inside of 1.5 inches. The ranges of 30  other test items did include 1.5 inches somewhere in the range, but the  center of the range (where Army testers aimed the shot) was still inside of  1.5 inches. Only four test items were marked with ranges centered on 1.5  inches.
DOD may be incorrect in stating that shooting closer to the edge would  have increased the risk of a failure for this subtest. For most subtests this  may be the case, but according to Army subject matter experts the impact  test is different. For the impact test, the plate is dropped onto a concrete  surface, striking the crown (center) of the plate. The test is to determine if  this weakens the structural integrity of the plate, which could involve  various cracks spreading from the center of the plate outward. The reason  the requirement for this shot on this subtest is written differently (i.e., to  be shot at approximately 1.5 inches from the edge, as opposed to within a  range between 0.75 inches and 1.25 inches or between 1.0 inches and 1.5  inches on other subtests) is that it is meant to test the impact’s effect on  the plate. For this subtest and this shot, there may actually be a higher risk  of failure the closer to the center the shot occurs. PEO Soldier  representatives acknowledged that the purchase descriptions should have  been written more clearly and changed the requirement for this shot to a  range of between 1.5 inches and 2.25 inches during First Article Testing.  We confirmed that Army testers correctly followed shot location testing  protocols during First Article Testing by double-checking the  measurements on the firing lane prior to the shooting of the plate. We also  note that, although DOD stated the Preliminary Design Model testing shot  locations for the impact test complied with the language of the testing  protocols, under the revised protocol used during First Article Testing  several of these Preliminary Design Model testing impact test shot  locations would not have been valid. DOD stated that there was no impact  on the outcome of the test, but DOD cannot say that definitively. Because  shooting closer to the edge may have favored the vendors in this case, the  impact could have been that a solution or solutions may have passed that  should not have.  21. The Army stated that “V50 subtests for more robust threats…were  executed to the standard protocols.” Our observations and analysis of the  data show that this statement is incorrect. Sections 2.2.3.h(2) of the  detailed test plans state:  “If the first round fired yields a complete penetration, the  propellant charge for the second round shall be equal to that of the  actual velocity obtained on the first round minus a propellant  decrement for 100 ft/s (30 m/s) velocity decrease in order to obtain  a partial penetration. If the first round fired yields a partial  penetration, the propellant charge for the second round shall be  equal to that of the actual velocity obtained on the first round plus  a propellant increment for a 50 ft/s (15 m/s) velocity increase in  order to obtain a complete penetration. A propellant increment or  decrement, as applicable, at 50 ft/s (15 m/s) from actual velocity of  last shot shall be used until one partial and one complete  penetration is obtained. After obtaining a partial and a complete  penetration, the propellant increment or decrement for 50 ft/s (15  m/s) shall be used from the actual velocity of the previous shot.”
V50 testing is conducted to discern the velocity at which 50 percent of the  shots of a particular threat would penetrate each of the body armor  designs. The testing protocols require that, after every shot that is defeated  by the body armor, the velocity of the next shot be increased. Whenever a  shot penetrates the armor, the velocity should be decreased for the next  shot. This increasing and decreasing of the velocities is supposed to be  repeated until testers determine the velocity at which 50 percent of the  shots will penetrate. In cases in which the armor far exceeds the V50  requirement and is able to defeat the threat for the first six shots, the  testing may be halted without discerning the V50 for the plate and the  plate may be ruled as passing the requirements. During Preliminary Design  Model V50 testing, Army testers would achieve three partial penetrations  and then continue to shoot at approximately the same velocity, or lower,  for shots 4, 5, and 6 in order to intentionally achieve six partial  penetrations. Army testers told us that they did this to conserve plates.  According to the testing protocols, Army testers should have continued to  increase the charge weight in order to try to achieve a complete  penetration and determine a V50 velocity. The effect of this methodology  was that solutions were treated inconsistently. Army officials told us that  this practice had no effect on which designs passed or failed, which we do  not dispute in our report; however, this practice made it impossible to  discern the true V50s for these designs based on the results of Preliminary  Design Model testing.  22. DOD agreed that Army testers deviated from the testing protocols by  measuring back-face deformation at the point of aim. DOD stated that this  decision was made by Army leadership in consultation with the office of  the Director of Operational Test and Evaluation, because this would not  disadvantage any vendor. We agree with DOD that this decision was made  by Army leadership in consultation with the office of the Director of  Operational Test and Evaluation. We did not independently assess all  factors being considered by Army leadership when they made the decision  to overrule the Integrated Product Team and the Milestone Decision  Authority’s initial decision to measure to the deepest point.
DOD also stated that measuring back-face deformation at the point of aim  is an accurate and repeatable process. As we pointed out in our previous  responses, DOD’s own comments regarding DOD’s Assertion 3 contradict  this statement where DOD writes that there were “potential variances  between the actual aim point and impact point during testing.”  Furthermore, we observed that the aim laser used by Army testers was  routinely out of line with where the ballistic was penetrating the yaw  card, despite continued adjustments to line up the aim laser with where  the ballistic was actually traveling.
DOD stated that it is not possible to know the reference point on a curved  object when the deepest deformation point is laterally offset from the aim  point. We disagree. DOD acknowledges in its response that PEO Soldier  had an internally documented process to account for plate curvature when  the deepest point of deformation was laterally offset from the point of aim.  The use of correction factor tables is a well-known industry standard that  has been in place for years, and this standard practice has been used by  NIJ laboratories and is well-known by vendors.
DOD and the Army presented several statistics on the difference between  aim point back-face deformation and deepest point back-face deformation  in testing and stated that the difference between the two is small. We do  not agree with DOD’s assertion that a difference of 10.66 millimeters is  small. In the case of Preliminary Design Model testing, the difference  between measuring at the aim point and at the deepest point was that at  least two solutions passed Preliminary Design Model testing that  otherwise would have failed. These designs passed subsequent First  Article Testing but have gone on to fail lot acceptance testing, raising  additional questions regarding the repeatability of the Aberdeen Test  Center’s testing practices.
DOD asserts that the adoption of the laser scanner measurement  technique resolves the problems the Army experienced in measuring back- face deformations completely. We would agree that the laser scanner has  the potential to be a useful device but when used in the manner in which  Aberdeen Test Center used it – without an adequate certification and  without a thorough understanding of how the laser scanner might  effectively change the standard for a solution to pass – we do not agree  that it resolved back-face deformation measurement issues. Aberdeen Test  Center officials told us that they did not know what the accuracy of the  laser scanner was as it was used during First Article Testing.  23. DOD acknowledged the shortcoming we identified. DOD then asserted  that once the deviation of measuring back-face deformation at the point of  aim, rather than at the deepest point of depression was identified, those  involved acted decisively to resolve the issue. We disagree based on the  timeline of events described in our response to DOD’s comments on  Preliminary Design Model testing, as well as on the following facts. We  were present and observed the Integrated Product Team meeting on  March 25 and observed that all members of the Integrated Product Team  agreed to start measuring immediately at the deepest point, to score  solutions based on this deepest point data, to conserve plates, and then at  the end of the testing to make up the tests incorrectly performed during  the first third of testing, as needed. We observed Army testers implement  this plan the following day. Then, on March 27, Army leadership halted  testing for 2 weeks, considered the issue, and then reversed the  unanimous decision by the Integrated Product Team and decided to score  to the point of aim.
The deviation of scoring solutions based on the back-face deformation at  the point of aim created a situation in which the Army could not have  confidence in any solution that passed the Preliminary Design Model  testing. Because of this, the Army had to repeat testing, in the form of First  Article Testing, to determine whether the solutions that had passed  Preliminary Design Model testing actually met requirements.  24. DOD did not concur with our finding that rain may have impacted the  test results. DOD stated that such conditions had no impact upon First  Article Testing results. Our statistical analysis of the test data shows  failure rates to be significantly higher on November 13 than during other  days of testing, and our observations taken during that day of testing and  our conversations with industry experts familiar with the clay, including  the clay manufacturer, suggest the exposure of the clay to the cold, heavy  rain on that day may have been the cause of the high failure rates. Our  analysis examined the 83 plates tested against the most potent threat,  Threat D. The testing protocols required that two shots for the record be  taken on each plate. We performed a separate analysis for the 83 first  shots taken on these plates from the 83 second shots taken on the plates.  These confirmed statistically that the rate of failure on November 13 was  significantly higher than the rate of failure on other days. Further, of the 5  plates that experienced first-shot catastrophic failures during testing, 3 of  them (60 percent) were tested on November 13 and all 3 of these were due  to excessive back-face deformation. Given that only 9 plates were tested  on November 13, while 74 were tested during all the other days of testing  combined, it is remarkable that 60 percent of all catastrophic failures  occurred on that one day of testing.
DOD objected to our inclusion of no-test data in its calculation of first- and  second-shot failure rates on November 13. We believe that the inclusion of  no-test data is warranted because the Army’s exclusion of such plates was  made on a post hoc basis after the shots were initially recorded as valid  shots and because the rationale for determining the need for a re-test was  not always clear. Additionally, we conducted an analysis excluding the no- test plates identified by DOD and that analysis again showed that the  failure rate on November 13 was statistically higher than during the other  days of testing, even after the exclusions. Excluding the no-test plates, 38  percent of first shots on November 13 (3 of 8) and 88 percent of second  shots (7 of 8) failed.
In its response, DOD reports that Aberdeen Test Center’s own statistical  analysis of test data for Threat D reveals that the observed failure rate on  November 13 is attributable to the “poor performance” of one design  throughout testing. DOD asserts that its illustration indicates that “Design  K was the weakest design on all days with no rain as well as days with  rain.” DOD’s data do not support such a claim. As we have observed,  excluding no-test plates, DOD’s data are based on 10 tests of two shots  each for each of 8 designs (160 cases total). Each shot is treated as an  independent trial, an assumption we find tenuous given that a plate’s  structural integrity might be affected by the first shot. To account for date,  DOD subdivides the data into cell sizes far too small to derive reliable  statistical inferences about failure rates (between 2 and 6 shots per cell),  as evidenced by the wide confidence intervals illustrated in DOD’s visual  representation of its analysis.
Among evidence DOD presented to support its claim that Design K was the  weakest performing design on both November 13 and other days is failure  rate data for four designs that were not tested on the day in question. For  two of the three designs tested on November 13 there were only one or  two plates tested on November 13, far too few to conduct reliable  statistical tests on differences in design performance. For the other type of  plate tested on that day (Design L), the three plates tested had a markedly  higher failure rate (3 of 6 shots, or 50 percent) on that day than on other  days (when it had, in 14 shots, 5 failures, or a 36 percent failure rate).  Design K had a failure rate of 6 of 6 shots (100 percent) on the day in  question, compared with 8 of 14 shots (57 percent) on other days.  Overall, it is impossible to determine from such a small set of tests  whether the lack of statistical significance between different designs’  failure rates on November 13 and other days results from small sample  size or a substantive difference in performance.
Overall, the Army Test and Evaluation Command’s design-based analysis  cannot distinguish between the potential effects of date and design on  failure rates because sufficient comparison data do not exist to conduct  the kind of multivariate analysis that might resolve this issue. Because the  data alone are inadequate for distinguishing between the potential effects  of date and design, we continue to recommend that independent experts  evaluate the potential effects of variations in materials preparation and  testing conditions, including those occurring on November 13, on overall  First Article Testing results.
Additionally, DOD stated that the clay is largely impervious to water.  However, as stated in our report, body armor testers from NIJ-certified  private laboratories, Army officials experienced in the testing of body  armor, body armor manufacturers, and the manufacturer of the clay used  told us that getting water on the clay backing material could cause a  chemical bonding change on the clay’s surface.
DOD stated that one of its first actions when bringing in the clay is to  scrape the top of the clay to level it. However, this only removes clay that  is above the metal edge of the box. Clay that is already at or below the  edge of the box is not removed by this scraping. We witnessed several  instances in which the blade would remove clay at some points, but leave  large portions of the clay surface untouched because the clay was below  the edge of the box.  25. See comment 11.  26. The DOD is correct that the one particular example regarding deleting  official test data only happened once. Fortunately, the results of the retest  were the same as the initial test. After we noted this deficiency, Army  officials told us that a new software program was being added that would  prevent this from occurring again. DOD also stated that only two persons  are authorized and able to modify the laser scanner software. We did not  verify this statement; however, we assert that DOD needs to have an  auditable trail when any such modifications are made and that it should  require supervisory review and documentation or logging of these setting  changes.  27. DOD acknowledged that the Army did not formally document  significant procedure changes that deviated from established testing  protocols or assess the impact of these deviations.  28. In our report we stated that the requirement to test at an NIJ-certified  laboratory was withdrawn because the Aberdeen Test Center is not NIJ- certified. DOD’s comments on this point do not dispute our statement.  Instead, DOD discussed NIJ certification and stated that it does not believe  that NIJ certification is appropriate for its test facilities. However, we did  not recommend that any DOD test facilities be NIJ-certified or even that  NIJ be the outside organization to provide an independent review of the  testing practices at Aberdeen Test Center that we did recommend.  However, we believe NIJ certification would meet our recommendation  for an independent review.
Regarding DOD’s comments regarding NIJ certification, DOD asserted that  NIJ certification is not appropriate for its test facilities and asserted that  there are significant differences between NIJ and U.S. Army body armor  test requirements. NIJ certification of a test laboratory and NIJ protocol  for testing personal body armor primarily used by law enforcement  officers are two distinct and different issues. Similar to a consumer United  Laboratories laboratory certification, an NIJ laboratory certification  includes an independent peer review of internal control procedures,  management practices, and laboratory practices. This independent peer  review is conducted to ensure that there are no conflicts of interest, and  that the equipment utilized in the laboratory is safe and reliable. This peer  review helps to ensure a reliable, repeatable, and accurate test, regardless  of whether the test in question is following a U.S. Army testing protocol or  a law enforcement testing protocol. NIJ-certified laboratories have  consistently proven to be capable of following an Army testing protocol,  which is demonstrated by the fact that NIJ-certified laboratories have  conducted previous U.S. Army body armor source selection testing in  accordance with First Article Testing protocol, as well as lot acceptance  tests. The slide DOD included in its comments is not applicable here  because it deals with the difference between testing protocols – the  protocols for Army Interceptor Body Armor tests and the NIJ protocol for  testing personal body armor primarily used by law enforcement officers.  NIJ certification of a laboratory and NIJ certification of body armor for  law enforcement purposes are two different things.  29. DOD stated that we were incorrect in asserting that the Army decided  to rebuild small arms ballistics testing facilities at Aberdeen Test Center  after the 2007 House Armed Services Committee hearing. Instead, DOD  stated that the contract to construct additional test ranges at the Aberdeen  Test Center Light Armor Range was awarded in September 2006 and that  construction was already underway at the time of June 2007 hearing. DOD  also stated that this upgrade was not in response to any particular event  but was undertaken to meet projected future Army ballistic test  requirements. Army officials we spoke with before testing for this  solicitation told us that this construction was being completed in order to  perform the testing we observed. As of July 2007, the Light Armor Range  included two pre-WWII era ballistic lanes and four modern lanes partially  completed. However, we noted that, as of July 2007, the lanes we visited  were empty and that none of the testing equipment was installed; only the  buildings were completed.
In addition to the physical rebuilding of the test sites, the Amy also re-built  its workforce to be able to conduct the testing. As stated on page 4 of  DOD’s comments, PEO Soldier has instituted an effort to transfer testing  expertise and experience from PEO Soldier to the Army Test and  Evaluation Command. Prior to the start of testing we observed that  Aberdeen Test Center hired, transferred in, and contracted for workers to  conduct the testing. These workers were then trained by Aberdeen Test  Center and conducted pilot tests in order to learn how to conduct body  armor testing. We observed parts of this training, in person, and other  parts via recorded video. In addition, we spoke with officials during this  training and preparation process. From our observations and discussions  with Army testers and PEO Soldier officials, we believe this process to  have been a restarting of small arms ballistic testing capabilities at  Aberdeen Test Center. Based on DOD’s comments, we clarified our report  to reflect this information.
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, key contributors to this report  were Cary Russell, Assistant Director; Michael Aiken; Gary Bianchi;  Beverly Breen; Paul Desaulniers; Alfonso Garcia; William Graveline; Mae  Jones; Christopher Miller; Anna Maria Ortiz; Danny Owens; Madhav  Panwar; Terry Richardson; Michael Shaughnessy; Doug Sloane; Matthew  Spiers; Karen Thornton; and John Van Schaik.