Defense Acquisitions: DOD Needs to Develop Performance Criteria to Gauge Impact of Reform Act Changes and Address Workforce Issues
Published: 2010-07-29T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In May 2009, Congress passed the Weapon Systems Acquisition Reform Act of 2009 (Reform Act). The Reform Act contains a number of systems engineering and developmental testing requirements that are aimed at helping weapon programs establish a solid foundation from the start of development. GAO was asked to examine (1) DOD's progress in implementing the systems engineering and developmental testing requirements, (2) views on the alignment of the offices of the Directors of Systems Engineering and Developmental Test and Evaluation, and (3) challenges in strengthening systems engineering and developmental testing activities. In conducting this work, GAO analyzed implementation status documentation and obtained opinions from current and former DOD systems engineering and testing officials on the placement of the two offices as well as improvement challenges.
section_title: What GAO Found
paragraphs: DOD has implemented or is implementing the Reform Act requirements related to systems engineering and developmental testing. Several foundational steps have been completed. For example, new offices have been established, directors have been appointed for both offices, and the directors have issued a joint report that assesses their respective workforce capabilities and 42 major defense acquisition programs. Many other requirements that have been implemented will require sustained efforts by the directors' offices, such as approving systems engineering and developmental testing plans, as well as reviewing these efforts on specific weapon programs. DOD is studying the option of allowing the Director, Developmental Test and Evaluation, to serve concurrently as the Director of the Test Resource Management Center. The directors have not yet developed joint guidance for assessing and tracking acquisition program performance of systems engineering and developmental testing activities. It is unclear whether the guidance will include specific performance criteria that address long-standing problems and program risks, such as those related to concurrency of development and production activities and adequacy of program resources. Current and former systems engineering and developmental testing officials offered varying opinions on whether the new directors' offices should have been placed under the Director of Defense Research and Engineering organization--an organization that focuses primarily on developing and transitioning technologies to acquisition programs. The Director of Defense Research and Engineering believes aligning the offices under his organization helps address congressional and DOD desires to increase emphasis on and strengthen activities prior to the start of a new acquisition program. Most of the officials GAO spoke with believe the two offices should report directly to the Under Secretary for Acquisition, Technology and Logistics or otherwise be more closely aligned with acquisition programs because most of their activities are related to weapon programs. They also believe cultural barriers and staffing issues may limit the effectiveness of the two offices under the current organizational structure. Currently, DOD is not reporting to Congress on how successfully the directors are effecting program changes, making it difficult to determine if the current placement of the offices makes sense or if the Reform Act is having an impact. The military services face a number of challenges as they try to strengthen systems engineering and developmental testing activities on acquisition programs. Although the services believe they have enough staff to perform both of these activities, they have not been able to clearly identify the number of staff that are actually involved. The Director of Developmental Test and Evaluation does not believe the military services have enough testing personnel and is concerned that DOD does not have the capacity to train the large influx of contractors that are expected to be converted to government employees.

section_title: Letter
paragraphs: For years, GAO has reported on significant cost overruns on the  Department of Defense’s (DOD) major weapon system acquisition  programs.  Even though DOD has incorporated previous legislative  provisions into its acquisition policies, such as requiring weapon programs  to use mature technologies from the start of development, programs are  still experiencing cost and schedule problems. The Senate Armed Services  Committee reported that since the beginning of 2006, nearly half of DOD’s  largest acquisition programs have exceeded Nunn-McCurdy cost-growth  standards established by Congress. DOD is now faced with making tough  decisions about the viability of some of its weapon system programs. In  2009, for example, the Secretary of Defense proposed canceling or  significantly curtailing weapon programs with a projected cost of at least  $126 billion.
Cost and schedule overruns can be attributed to a number of factors that  occur early in an acquisition, including poorly analyzed requirements,  design instability, and inadequate systems engineering and testing. In May  2009, Congress passed the Weapon Systems Acquisition Reform Act of  2009 (Reform Act), aimed at improving DOD’s organization and  procedures for the acquisition of major weapon systems. This legislation  places more emphasis on activities that should occur early in weapon  systems development, including those related to systems engineering and  developmental testing, in order to help establish a solid program  foundation from the start of development. The Senate Armed Services  Committee asked us to examine (1) DOD’s progress in implementing  systems engineering and developmental testing requirements called for in  the Reform Act, (2) views on the alignment of the offices of the Director of  Systems Engineering and the Director of Developmental Test and  Evaluation within the Office of the Secretary of Defense, and                     (3) challenges in strengthening systems engineering and developmental  testing activities.
In conducting our work, we interviewed officials and collected documents  from the offices of the Director of Systems Engineering and the Director of  Developmental Test and Evaluation in order to learn the status of their  efforts to implement the Reform Act legislation and challenges they are  addressing. We also interviewed officials from various offices within the  Office of the Under Secretary of Defense for Acquisition, Technology and  Logistics (AT&L); the office of the Director, Operational Test and  Evaluation; each of the military services; the Defense Science Board; as  well as former DOD systems engineering and developmental testing  executives to obtain their opinions on the alignment of the two offices  within the Office of the Secretary of Defense and potential challenges. We  conducted this performance audit from December 2009 to July 2010 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
subsections: section_title: Background
paragraphs: Systems engineering and test and evaluation are critical parts of the  weapon system acquisition process and how well these activities are  conducted early in the acquisition cycle can greatly affect program  outcomes. Systems engineering translates customer needs into specific  product requirements for which requisite technological, software,  engineering, and production capabilities can be identified through  requirements analysis, design, and testing. Early systems engineering  provides the knowledge that weapon system requirements are achievable  with available resources such as technologies, time, people, and money. It  allows a product developer to identify and resolve performance and  resource gaps before product development begins by reducing  requirements, deferring them to the future, or increasing the estimated  cost for the weapon system’s development. Systems engineering plays a  fundamental role in the establishment of the business case for a weapon  acquisition program by providing information to DOD officials to make  tradeoffs between requirements and resources. Systems engineering is  then applied throughout the acquisition process to manage the engineering  and technical risk in designing, developing, and producing a weapon  system. The systems engineering processes should be applied prior to the  start of a new weapon acquisition program and then continuously  throughout the life-cycle.
Test and evaluation provides information about the capabilities of a  weapon system and can assist in managing program risk. There are  generally two broad categories of testing: developmental and operational.  Developmental testing is used to verify the status of technical progress,  substantiate achievement of contract technical performance, and certify  readiness for initial operational testing. Early developmental testing  reduces program risks by evaluating performance at progressively higher  component and subsystem levels, thus allowing program officials to  identify problems early in the acquisition process. Developmental testing  officials in the Office of the Secretary of Defense and the military services  provide guidance and assistance to program managers on how to develop  sound test plans. The amount of developmental testing actually conducted  however, is controlled by the program manager and the testing  requirements explicitly specified in the development contract. In contrast,  operational testing determines if a weapon system provides operationally  useful capability to the warfighter. It involves field testing a weapon  system, under realistic conditions, to determine the effectiveness and  suitability of the weapon for use in combat by military users, and the  evaluation of the results of such tests. DOD’s Director of Operational Test  and Evaluation conducts independent assessments of programs and  reports the results to the Secretary of Defense and Congress.
In 2008, the Defense Science Board reported that operational testing over  the previous 10 years showed that there had been a dramatic increase in  the number of weapon systems that did not meet their suitability  requirements. The board found that failure rates were caused by several  factors, notably the lack of a disciplined systems engineering process early  in development and a robust reliability growth program. The board also  found that weaknesses in developmental testing, acquisition workforce  reductions and retirements, limited government oversight, increased  complexity of emerging weapon systems, and increased reliance on  commercial standards (in lieu of military specifications and standards) all  contributed to these failure rates. For example, over the last 15 years, all  service acquisition and test organizations experienced significant  personnel cuts, including the loss of a large number of the most  experienced technical and management personnel, including subject  matter experts, without an adequate replacement pipeline. The services  now rely heavily on contractors to help support these activities.
Over the past two decades, the prominence of the developmental testing  and systems engineering communities within the Office of the Secretary of  Defense has continuously evolved, as the following examples illustrate.
In 1992, a systems engineering directorate did not exist and the  developmental test function was part of the Office of the Director of  Test and Evaluation, which reported directly to the Under Secretary of  Defense for Acquisition. At that time, the director had direct access to  the Under Secretary on an array of issues related to test policy, test  assets, and the workforce.
In 1994, the Development Test, Systems Engineering and Evaluation  office was formed. This organization effectively expanded the  responsibilities of the former testing organization to formally include  systems engineering. The organization had two deputy directors: the  Deputy Director, Development Test and Evaluation, and the Deputy  Director, Systems Engineering. This organization was dissolved in  1999.
From 1999 to 2006, systems engineering and developmental testing  responsibilities were aligned under a variety of offices. The  responsibility for managing test ranges and resources, for example,  was transferred to the Director of Operational Test and Evaluation.  This function was later moved to the Test Resource Management  Center, which reports directly to AT&L, where it remains today. In  2004, a Director of Systems Engineering was re-established and then in  2006 this became the System and Software Engineering Directorate.  Developmental testing activities were part of this directorate’s  responsibilities. As a result, systems engineering and developmental  testing issues were reported indirectly to AT&L through the Deputy  Under Secretary for Acquisition and Technology.
Congress passed the Weapon Systems Acquisition Reform Act of 2009  (Reform Act)—the latest in a series of congressional actions taken to  strengthen the defense acquisition system. The Reform Act establishes a  Director of Systems Engineering and a Director of Developmental Test and  Evaluation within the Office of the Secretary of Defense and defines the  responsibilities of both offices. The Reform Act requires the services to  develop, implement, and report on their plans for ensuring that systems  engineering and developmental testing functions are adequately staffed to  meet the Reform Act requirements. In addition, it requires the directors to  report to Congress on March 31 of each year on military service and major  defense acquisition program systems engineering and developmental  testing activities from the previous year. For example, the report is to  include a discussion of the extent to which major defense acquisition  programs are fulfilling the objectives of their systems engineering and  developmental test and evaluation master plans, as well as provide an  assessment of the department’s organization and capabilities to perform  these activities. Figure 1 shows some of the major reorganizations over the  past two decades, including the most recent change where DOD decided  to place the two new directors’ offices under the Director of Defense  Research and Engineering.
section_title: DOD Has Made Progress in Implementing Reform Act Requirements, but Has Not Developed Performance Criteria to Track Success
paragraphs: DOD has made progress in implementing the systems engineering and  developmental test and evaluation provisions of the Reform Act, but has  not yet developed performance criteria that would help assess the  effectiveness of the changes. Some requirements, such as the  establishment of the two new offices, have been fully implemented. The  implementation of other requirements, such as the review and approval of  systems engineering and developmental test and evaluation plans, has  begun but requires sustained efforts. The department has not fully  implemented other requirements. For example, DOD has begun  development of joint guidance that will identify measurable performance  criteria to be included in the systems engineering and developmental  testing plans. DOD initially decided that one discretionary provision of the  act—naming the Director of Developmental Test and Evaluation also as  the Director of the Test Resource Management Center—would not be  implemented. However, the Director of Defense Research and Engineering  is currently examining the implications of this organizational change. It  will be several years before the full impact of the Reform Act provisions is  known.
The offices of the Director of Systems Engineering and Developmental  Test and Evaluation were officially established by the Under Secretary of  Defense for AT&L in June 2009 to be his principal advisors on systems  engineering and developmental testing matters. The directors took office   3 months and 9 months later, respectively, and are working on obtaining  the funding, workforce, and office space needed to accomplish their  responsibilities. The directors have also completed evaluations of the  military services’ organizations and capabilities for conducting systems  engineering and developmental testing, and identified areas for  improvement. These evaluations were based on reports provided by the  services that were also required by the Reform Act.
As shown in table 1, many of the requirements that have been  implemented will require ongoing efforts.
The directors have the responsibility for reviewing and approving systems  engineering and developmental test and evaluation plans as well as the  ongoing responsibility to monitor the systems engineering and  developmental test and evaluation activities of major defense acquisition  programs. During fiscal year 2009, the Director of Systems Engineering  reviewed 22 systems engineering plans and approved 16, while the  Director of Developmental Test and Evaluation reviewed and approved 25  developmental test and evaluation plans within the test and evaluation  master plans. Both offices are monitoring and reviewing activities on a  number of major acquisition programs, including the Virginia Class  Submarine, the Stryker Family of Vehicles, and the C-130 Avionics  Modernization Program. Once their offices are fully staffed, the directors  plan to increase efforts in reviewing and approving applicable planning  documents and monitoring the activities of about 200 major defense  acquisition and information system programs.
Evaluations of 42 weapon systems were included in the directors’ first  annual joint report to Congress. The individual systems engineering  program assessments were consistent in that they typically included  information on 10 areas, including requirements, critical technologies,  technical risks, reliability, integration, and manufacturing. In some cases,  the assessments also included an overall evaluation of whether the  program was low, medium, or high risk; the reasons why; and a general  discussion of recommendations or efforts the director has made to help  program officials reduce any identified risk. Examples include the  following.
In an operational test readiness assessment of the EA-18G aircraft, the  Director of Systems Engineering found multiple moderate-level risks  related to software, communications, and mission planning and made  recommendations to reduce the risks. The program acted on the risks  and recommendations identified in the assessment and delayed the  start of initial operational testing by 6 weeks to implement the fixes. It  has completed initial operational testing and was found to be effective  and suitable by Navy testers. The Director of Operational Test and  Evaluation rated the system effective but not suitable, and stated that  follow-on testing has been scheduled to verify correction of noted  deficiencies. The program received approval to enter full rate  production and is rated as a low risk in the joint annual report.
The systems engineering assessment of the Global Hawk program was  high risk pending the determination of actual system capability; it also  stated that there is a high probability that the system will fail  operational testing. The assessment cited numerous issues, including  questions regarding the system’s ability to meet mission reliability  requirements, poor system availability, and the impact of simultaneous  weapon system block builds (concurrency). Despite the director’s  concerns and efforts to help the program office develop a reliability  growth plan for Global Hawk, no program funding has been allocated  to support reliability improvements.
The Expeditionary Fighting Vehicle assessment did not include an  overall evaluation of risk. The assessment noted that the program was  on track to meet the reliability key performance parameter of            43.5 hours mean time between operational mission failure. Problems  related to meeting this and other reliability requirements were a  primary reason why the program was restructured in 2007. However,  the assessment did not address the high degree of concurrency  between development and production, which will result in a  commitment to fund 96 low-rate initial procurement vehicles prior to  demonstrating that the vehicle can meet the reliability threshold value  at initial operational test and evaluation, currently scheduled for  completion by September 2016.
Developmental testing assessments covered fewer programs and were not  as structured as those provided by the systems engineering office in that  there were no standard categories of information that were included in  each assessment. Part of the reason is that the Director of the  Developmental Test and Evaluation office was just developing the  necessary expertise to review and provide formal assessments of  programs. For the programs that were reviewed, the assessments included  a status of developmental testing activities on programs and in some cases  an assessment of whether the program was low, medium, or high risk. For  example, the Director of Developmental Test and Evaluation supported an  assessment of operational test readiness for the C-5 Reliability  Enhancement and Reengining Program. The assessment stated that due to  incomplete testing and technical issues found in developmental testing,  there is a high risk of failure in operational testing. The assessment  recommended that the program resolve these issues before beginning  operational testing.
The Reform Act also requires that the Director of Systems Engineering  develop policies and guidance on, among other things, the use of systems  engineering principles and best practices and the Director of  Developmental Test and Evaluation develop policies and guidance on,  among other things, the conduct of developmental testing within DOD.  The directors have issued some additional policies to date, such as  expanded guidance on addressing reliability and availability on weapon  programs and on incorporating test requirements in acquisition contracts.  The directors plan to update current guidance and issue additional  guidance in the future. According to DOD officials, there are over 25  existing documents that provide policy and guidance for systems  engineering and developmental testing. The directors also have an ongoing  responsibility to advocate for and support their respective DOD  acquisition workforce career fields, and have begun examining the training  and education needs of these workforces.
Two provisions, one of which is discretionary, have not been completed.  The Reform Act requires that the directors, in coordination with the newly  established office of the Director for Program Assessments and Root  Cause Analysis, issue joint guidance on the development of detailed,  measurable performance criteria that major acquisition programs should  include in their systems engineering and testing plans. The performance  criteria would be used to track and measure the achievement of specific  performance objectives for these programs, giving decision makers a  clearer understanding each program’s performance and progress. The  offices have begun efforts to develop these policies and guidance, but  specific completion dates have not been identified. At this time, it is  unclear whether the guidance will include specific performance criteria  that should be consistently tracked on programs and any risks associated  with these programs, such as ones related to technology maturity, design  stability, manufacturing readiness, concurrency of development and  production activities, prototyping, and adequacy of program resources.  Finally, the Reform Act gives DOD the option of permitting the Director of  Developmental Test and Evaluation to serve as the Director of the Test  Resource Management Center. DOD initially decided not to exercise this  option. However, the Director of Defense Research and Engineering  recently stated that his organization is examining the possibility of  consolidating the offices. The director stated that it makes sense to  combine the two offices because it would merge test oversight and test  resource responsibilities under one organization, but the ultimate decision  will be based on whether there are any legal obstacles to combining the  two offices.
While most of the Reform Act’s requirements focus on activities within the  Office of the Secretary of Defense, the military services are ultimately  responsible for ensuring that their weapon systems start off with strong  foundations. To that end, in November 2009, the services, in reports to the  Directors of Systems Engineering and Developmental Test and Evaluation,  identified plans for ensuring that appropriate resources are available for  conducting systems engineering and developmental testing activities. The  individual reports also highlighted management initiatives undertaken to  strengthen early weapon acquisition activities. For example, the Army is  establishing a center at Aberdeen Proving Ground that will focus on  improving reliability growth guidance, standards, methods, and training  for Army acquisition programs. The Navy has developed criteria, including  major milestone reviews and other gate reviews, to assess the “health” of  testing and evaluation at various points in the acquisition process. The Air  Force has undertaken an initiative to strengthen requirements setting,  systems engineering, and developmental testing activities prior to the start  of a new acquisition program. Air Force officials believe this particular  initiative will meet the development planning requirements of the Reform  Act.
section_title: Experts Offer Varying Opinions on the Placement of the Systems Engineering and Developmental Test and Evaluation Offices
paragraphs: Experts provided different viewpoints on the proper placement of the new  systems engineering and developmental test and evaluation offices, with  some expressing concern that as currently placed, the offices will wield  little more power or influence than they had prior to the passage of the  Reform Act. According to the Director of Defense Research and  Engineering, the Under Secretary of Defense for AT&L placed the new  offices under his organization because the department wanted to put  additional emphasis on systems engineering and developmental testing  prior to the start of a weapons acquisition program. The director believes  this is already occurring and that both offices will continue to have a  strong relationship with acquisition programs even though they do not  report directly to an organization with significant involvement with major  defense acquisition programs. However, many current and former DOD  systems engineering and developmental testing officials we spoke with  believe the offices should be closely linked to weapon acquisition  programs because most of their activities are related to those programs.  Similarly, the Defense Science Board recommended that a developmental  testing office be established and report directly to an organization that has  significant involvement with major defense acquisition programs. In  addition, officials we spoke with believe several other significant  challenges, including those related to staffing and the culture of the  Defense Research and Engineering organization, are already negatively  affecting the offices’ effectiveness. DOD has not established any  performance criteria that would help gauge the success of the new  directors’ offices, making it difficult to determine if the offices are  properly aligned within the department or if the Reform Act is having an  impact on program outcomes.
subsections: section_title: DOD Aligned New Systems Engineering and Developmental Test and Evaluation Offices with the Research and Engineering Organization
paragraphs: After the passage of the Reform Act, DOD considered several options on  where to place the new offices of the Director of Systems Engineering and  Director of Developmental Test and Evaluation. According to an official  who helped evaluate potential alternatives, DOD could have aligned the  offices under AT&L in several different ways (see fig. 2). For example, the  offices could have reported directly to the Under Secretary of AT&L or  indirectly to the Under Secretary of AT&L either through the Assistant  Secretary of Defense (Acquisition) or the Director of Defense Research  and Engineering. DOD decided to place the offices under the Director of  Defense Research and Engineering, an organization that previously  primarily focused on science and technology issues.
Under Secretary of Defense for  Acquisition, Technology & Logistics  (USD AT&L)
section_title: Director of Defense Research and Engineering Believes Offices Are Properly Aligned
paragraphs: The Director of Defense Research and Engineering is aware of the  challenges of placing the offices under an organization whose primary  mission is to develop and transition technologies to acquisition programs,  but believes that the current placement makes sense given congressional  and DOD desires to place more emphasis on activities prior to the start of  a new acquisition program. He stated that the addition of systems  engineering and developmental testing not only stretches the role and  mission of his organization, but also strengthens the organization’s role in  acquisitions because it helps give the organization’s research staff another  point of view in thinking about future technologies and systems.
He plans for the offices to perform both assessment and advisory  activities, including:    providing risk assessments of acquisition programs for the Defense  Acquisition Board,  continuing to help programs succeed by providing technical insight  and assisting the programs in the development of the systems  engineering plan and the test and evaluation master plan, and  educating and assisting researchers to think through new concepts or  technologies using systems engineering to inform fielding and  transition strategies.
According to the Director of Defense Research and Engineering, the  offices are already performing some of these functions. For example, the  new directors have provided technical input to the Defense Acquisition  Board on various weapons programs. The director stated the systems  engineering organization is reviewing manufacturing processes and  contractor manufacturing readiness for weapons programs such as the  Joint Strike Fighter. In addition, a developmental testing official stated  they are assisting the Director of Defense Research and Engineering  Research Directorate in conducting technology readiness assessments and  helping programs identify the trade spaces for testing requirements while  reviewing the test and evaluation master plan. The director believes the  value of having the offices perform both assessment and advisory  activities is that they can look across the acquisition organization and  identify programs that are succeeding from a cost, schedule, and  performance perspective and identify common threads or trends that  enable a program to succeed. Conversely, they could identify common  factors that make programs fail.
The Director of Defense Research and Engineering identified three  challenges that he is trying to address in order for systems engineering and  developmental testing to have a more positive influence on weapon system  outcomes. First, the director would like to improve the technical depth of  the systems engineering and developmental testing offices. Both functions  have atrophied over the years and need to be revitalized. This will require  the offices to find highly qualified people to fill the positions, which will  not be easy. Second, the director wants to improve the way the Defense  Research and Engineering organization engages with other DOD  organizations that are involved in weapon system acquisition. The director  noted that there are a lot of players and processes involved in weapon  acquisition and that the systems engineering office can play a large role in  facilitating greater interaction. Third, the director would like the Defense  Research and Engineering organization to find better ways to shape,  engage with, contract with, and get information from the defense  industrial base.
In addition to the three challenges, it will also be difficult to determine  whether the two new offices are having a positive impact on weapon  system outcomes. The Directors of Systems Engineering and  Developmental Test and Evaluation are not reporting the number of  recommendations implemented by program managers or the impact the  recommendations have had on weapon programs, which would allow  senior leaders to gauge the success of the two offices. This type of  information could help the Under Secretary of AT&L determine if the  offices need to be placed under a different organization, if the offices need  to place more emphasis on advisory or assessment activities, and if the  Reform Act is having an impact on program outcomes.
section_title: Most Experts Believe Offices Would Be Better Aligned under an Acquisition Organization
paragraphs: The vast majority of current and former DOD systems engineering and test  officials we spoke with were opposed to the placement of the offices  under the Director of Defense Research and Engineering. Their chief  concern is that the mission of the Director of Defense Research and  Engineering organization is primarily focused on developing new  technologies and transitioning those technologies to acquisition programs.  While they recognize that the systems engineering and developmental  testing offices need to be involved in activities prior to the official start of  a new weapons program, they believe the offices’ expertise should be  focused on helping DOD acquisition programs establish doable  requirements given the current state of technologies, not on the  technologies themselves. Therefore, they believe the offices would be  more appropriately placed under the newly established offices of the  Principal Deputy Under Secretary of Defense for AT&L or the Assistant  Secretary of Defense for Acquisition, whose missions are more closely  aligned with acquisition programs.
Some officials we spoke with believe that a cultural change involving the  focus and emphasis of the office of the Director of Defense Research and  Engineering will have to take place in order for that organization to fully  support its role in overseeing acquisition programs and improving the  prominence of the two new offices within the department. However, these  same officials believe that this cultural change is not likely to occur and  that the Director of Defense Research and Engineering will continue to  focus primarily on developing and transitioning new technologies to  weapon programs. Therefore, the offices may not get sufficient support  and resources or have the clout within DOD to effect change. One former  systems engineering official pointed out that the historic association of  systems engineering with the Director of Defense Research and  Engineering does not bode well for the systems engineering office. Based  upon his experience, the Director of Defense Research and Engineering’s  focus and priorities resulted in a fundamental change in philosophy for the  systems engineering mission, the virtual elimination of a comprehensive  focus on program oversight or independent identification of technical risk,  and a reduction in systems engineering resources. In short, he found that  the Director of Defense Research and Engineering consistently focused on  science and technology, in accordance with the organization’s charter,  with systems engineering being an afterthought.
Likewise, current and former developmental testing officials are  concerned about the Director of Defense Research and Engineering’s  support for developmental testing activities. They identified several  staffing issues that they believe are key indicators of a lack of support.
First, they pointed out that it took almost 9 months from the time the  Director of Developmental Test and Evaluation office was established  before a new director was in place compared to 3 months to place the  Director of Systems Engineering. If developmental testing was a  priority, officials believe that the Director of Defense Research and  Engineering should have filled the position earlier.
Second, test officials believe the Director of Developmental Test and  Evaluation office needs to have about the same number of staff as the  offices of the Director of Systems Engineering and the Director of  Operational Test and Evaluation. According to officials, DOD currently  plans to have about 70 people involved with developmental testing  activities, 180 people for systems engineering, and 250 for operational  testing. However, testing officials believe the offices should be roughly  the same size given the fact that developmental testing will cover the  same number of programs as systems engineering and operational  testing and that roughly 80 percent of all testing activities are related  to developmental tests, with the remaining 20 percent being for  operational tests.
Third, even though the Director of Developmental Test and Evaluation  expects the office to grow to about 70 people by the end of fiscal year  2011, currently there are 30 people on board. The director believes  there are a sufficient number of qualified people seeking positions and  therefore the office could be ramped up more quickly.
Finally, the Director of Developmental Test and Evaluation stated that  his office has only one senior-level executive currently on staff who  reports to him and that there are no plans to hire more for the             70-person organization. The director believes it is crucial that the  organization have more senior-level officials because of the clout they  carry in the department. The director believes that the lack of an  adequate number of senior executives in the office weakens its ability  to work effectively with or influence decisions made by other DOD  organizations. Further, officials from other testing organizations, as  well as the systems engineering office, indicated they have two or  more senior executive-level employees.
A May 2008 Defense Science Board report, which was focused on how  DOD could rebuild its developmental testing activities, recommended that  developmental testing be an independent office that reports directly to the  Deputy Under Secretary of Defense (Acquisition and Technology). At that  time, according to the report, there was no office within the Office of the  Secretary of Defense with comprehensive developmental testing oversight  responsibility, authority, or staff to coordinate with operational testing. In  addition, the existing residual organizations lacked the clout to provide  development test guidance and developmental testing was not considered  to be a key element in AT&L system acquisition oversight. According to  the study director, placing the developmental testing office under the  Director of Defense Research and Engineering does not adequately  position the new office to perform the oversight of acquisition programs.
section_title: Military Services Face Workforce and Resource Challenges as They Strive to Strengthen Their Systems Engineering and Developmental Testing Efforts
paragraphs: The military services, the Directors of Systems Engineering and  Developmental Test and Evaluation, and we have identified a number of  workforce and resource challenges that the military services will need to  address to strengthen their systems engineering and developmental testing  activities. For example, it is unclear whether the services have enough  people to perform both systems engineering and developmental testing  activities. Even though the services reported to the directors that they  have enough people, they do not have accurate information on the number  of people performing these activities. The Director of Developmental Test  and Evaluation disagreed with the services’ assertions, but did not know  how many additional people are needed. Service officials have also  expressed concern about the department’s ability to train individuals        who do not meet requisite certification requirements on a timely basis  and being able to obtain additional resources to improve test facilities.
The military services were required by the Reform Act to report on their  plans to ensure that they have an adequate number of trained systems  engineering and developmental testing personnel and to identify additional  authorities or resources needed to attract, develop, train, and reward their  staff. In November 2009, the military services submitted their reports to  the respective directors within the Office of the Secretary of Defense on  their findings. In general, the services concluded that even with some  recruiting and retention challenges, they have an adequate number of  personnel to conduct both systems engineering and developmental testing  activities (see table 2 below). According to service officials, this  determination was based on the fact that no program offices identified a  need for additional staffing to complete these activities. The reports also  stated the services generally have sufficient authorities to attract and  retain their workforce. In DOD’s first annual joint report to Congress, the  Director of Developmental Test and Evaluation did not agree with the  military services’ assertion that they have enough staff to perform the full  range of developmental testing activities. The director does not know how  many more personnel are needed, but indicated that the office plans to  work with the services to identify additional workforce needs. The  Director of Systems Engineering agreed with the services’ reports that  they have adequate staffing to support systems engineering activities  required by current policy. According to the director, this was based on  the 35,000 current personnel identified in the System Planning, Research  Development, and Engineering workforce—a generic workforce category  that includes systems engineering activities—as well as the services’ plans  to hire over 2,500 additional personnel into this same workforce category  over the next several years.
Although not clearly articulated in the services’ reports, military service  officials acknowledged that the personnel data in their reports may not be  entirely accurate. For example, officials believe the systems engineering  numbers identified in table 2 overstate the number of people actually  performing systems engineering activities because that particular career  field classification is a generic category that includes all types of  engineers. The developmental test workforce shown in the table does not  completely reflect the number of people who actually perform  developmental testing activities because the information provided by the  military services only identifies the personnel identified in the test and  evaluation career field. Service officials told us that there are many other  people performing these activities who are identified in other career fields.  The Director of Developmental Test and Evaluation believes these other  people may not be properly certified and that in the case of contractors,  they do not possess certifications which are equivalent to the certification  requirements of government personnel. This director plans to request  another report from the services in fiscal year 2010. This report will  address the overall workforce data; it will cover current staffing assigned  to early test and evaluation activities, training, and certification concerns  they have related to in-sourcing staff, rapid acquisition resource plans, and  infrastructure needs for emerging technologies. The Director of Systems  Engineering does not intend to request another report from the services.  Nevertheless, each of the military services plans to increase its systems  engineering workforce over the next several years. The exact number of  personnel is uncertain because the services’ hiring projections relate to a  general engineering personnel classification, not a specific systems  engineering career field.
The directors also identified challenges they believe the services will face  in strengthening systems engineering and developmental testing activities.  The Director of Systems Engineering pointed out that the services need to  put greater emphasis on development planning activities, as called for by  the Reform Act. The services are currently conducting these activities to  some extent, but the director believes a more robust and consistent  approach is needed. The Director of Developmental Test and Evaluation  highlighted two other challenges facing the military services. First, the  director would like to increase the number of government employees  performing test and evaluation activities. The services experienced  significant personnel cuts in these areas in the mid-1990s and has to rely  on contractors to perform the work. DOD’s joint report to Congress noted  that the Air Force in particular relies heavily on prime contractor  evaluations and that this approach could lead to test results that are  inaccurate, misleading, or not qualified, resulting in turn, in premature  fielding decisions since prime contractors would not be giving impartial  evaluations of results. The director believes there are a number of  inherently governmental test and evaluation functions that produce a more  impartial evaluation of results and that a desired end state would be one  where there is an appropriate amount of government and contractor  testing. Second, the director is concerned that DOD does not have the  capacity to train and certify an estimated 800 individuals expected to be  converted from contractor to government employees within the required  time frame. While most of the contractors are expected to have some level  of training and experience performing test activities, they probably will  not meet certifications required of government employees because they  have not had the same access to DOD training.
In addition to those challenges recognized by the directors, we have  identified other challenges we believe the services may face in  implementing more robust systems engineering and developmental testing,  including the following.
According to the military services, they plan to meet hiring targets  primarily through the conversion of contractors who are already  performing those activities, but do not have plans in place to ensure  that they have the right mixture of staff and expertise both now and in  the future. DOD officials acknowledge that they do not know the  demographics of the contractor workforce. However, they believe  many contractors are often retired military with prior systems  engineering experience. Therefore, while they may be able to meet  short-term needs, there could be a challenge in meeting long-term  workforce needs.
Army test officials indicated that they have experienced a significant  increase in their developmental testing workload since the terrorist  attacks of September 2001, with no corresponding increase in staffing.  As a result, personnel at their test ranges are working longer hours and  extra shifts, which testing officials are concerned may affect their  retention rates. Army officials also indicated that test ranges are  deteriorating more quickly than expected and they may not have the  appropriate funding to upgrade and repair the facilities and  instrumentation. Test personnel are often operating in obsolete and  outdated facilities that cannot meet test requirements, resulting in  safety issues, potential damage to equipment, and degraded quality of  life.
DOD’s increased emphasis on fielding rapid acquisition systems may  require the services to tailor their approach to systems engineering.  According to an Air Force official, efforts that normally take months to  complete for a more traditional acquisition program, have to be  completed in a matter of weeks for rapid acquisition programs.
section_title: Conclusions
paragraphs: DOD efforts to implement Reform Act requirements are progressing, but it  will take some time before the results of these efforts can be evaluated.  Current and former systems engineering and developmental testing  officials offer compelling insights concerning the placement of the new  directors’ offices under the Office of the Director of Defense Research and  Engineering, but it is still too soon to judge how effective the offices will  be at influencing outcomes on acquisition programs. The current  placement of the offices may present several challenges that could hinder  their ability to effectively oversee weapon system acquisition programs  and ensure that risks are identified, discussed, and addressed prior to the  start of a new program or the start of operational testing. Foremost among  these potential challenges is the ability of the Director of Defense  Research and Engineering to change the focus of the organization to  effectively assimilate the roles and missions of the two new offices and  then ensure that the offices are properly staffed and have the appropriate  number of senior leaders. The mission of the office of the Director of  Defense Research and Engineering has been to develop technology for  weapon programs; its focus has not been to manage the technical aspects  of weapon system acquisition programs.
Ultimately, the real proof of whether an organization outside of the major  defense acquisition program arena can influence acquisition program  decisions and outcomes should be based on results. The directors’ offices  have started to assess and report on the systems engineering and  developmental testing activities on some of the major defense acquisition  programs. They have also made recommendations and worked with  program officials to help reduce risks on programs such as the EA-18G,  Global Hawk, and the C-5 Reliability Enhancement and Reengining  programs. However, guidance on the development and tracking of  performance criteria that would provide an indication of how much risk is  associated with a particular weapon system—such as those related to  technology maturity, design stability, manufacturing readiness,  concurrency of development and production activities, prototyping, and  adequacy of program resources—has yet to be developed. Further, the  directors are not reporting to Congress on the extent to which programs  are implementing recommendations and the impact recommendations are  having on weapon programs, which would provide some insight as to the  impact the two offices are having on acquisition programs. Although not  required by the Reform Act, this type of information could be useful for  Congress to gauge the effectiveness of the directors’ offices.
The military services, which face increasing demands to develop and field  more reliable weapon systems in shorter time frames, may need additional  resources and training to ensure that adequate developmental testing and  systems engineering activities are taking place. However, DOD’s first joint  annual report to Congress, which was supposed to assess the department’s  organization and capabilities for performing systems engineering and  developmental testing activities, did not clearly identify the workforce  performing these activities, future workforce needs, or specific hiring  plans. In addition, DOD’s strategy to provide the necessary training within  the required time period to the large number of staff it plans to hire is  unclear. Therefore, workforce and training gaps are unknown.
section_title: Recommendations for Executive Action
paragraphs: In order to determine the effectiveness of the newly established offices,  we recommend that the Secretary of Defense direct the Directors of  Systems Engineering and Developmental Test and Evaluation to take the  following five actions:    Ensure development and implementation of performance criteria for  systems engineering plans and developmental test and evaluation  master plans, such as those related to technology maturity, design  stability, manufacturing readiness, concurrency of development and  production activities, prototyping, and the adequacy of program  resources.
Track the extent to which program offices are adopting systems  engineering and developmental testing recommendations.
Work with the services to determine the appropriate number of  government personnel needed to perform the scope of systems  engineering and developmental testing activities.
Develop plans for addressing the training needs of the new hires and  contractors who are expected to be converted to government  personnel.
Report to Congress on the status of these efforts in future joint annual  reports required by the Reform Act.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided us with written comments on a draft of this report.  DOD  concurred with each of the recommendations, as revised in response to  agency comments.  DOD’s comments appear in appendix I.
Based upon a discussion with DOD officials during the agency comment  period, we revised the first recommendation.  Specifically, instead of  recommending that the Directors of Systems Engineering and  Developmental Test and Evaluation develop a comprehensive set of  performance criteria that would help assess program risk, as stated in the  draft report, we now recommend that the directors ensure the  development and implementation of performance criteria for systems  engineering plans and developmental test and evaluation master plans.   The wording change clarifies the nature and scope of performance criteria  covered by our recommendation and is consistent with Reform Act  language that requires the directors to develop guidance on the  development of detailed, measurable performance criteria that major  acquisition programs should include in their systems engineering and  developmental testing plans.  According to DOD officials, the military  services are then responsible for developing the specific criteria that  would be used on their respective programs.
DOD also provided technical comments, which we incorporated as  appropriate.
We are sending copies of this report to the Secretary of Defense, the  Director of the Office of Management and Budget, and interested  congressional committees. We will also make copies available at no charge  on the GAO Web site at http://www.gao.gov.
If you have any questions about this report or need additional information,  please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. Key contributors to this report were Bruce  Thomas, Assistant Director; Cheryl Andrew; Rae Ann Sapp; Megan Hill;  and Kristine Hassinger.
section_title: Appendix I: Comments from the Department of Defense