Weapons Acquisition Reform: Actions Needed to Address Systems Engineering and Developmental Testing Challenges
Published: 2011-09-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: For the past 2 years, the Department of Defense (DOD) has been implementing the Weapon Systems Acquisition Reform Act (Reform Act) requirements for systems engineering and developmental testing. These activities are important to DOD's ability to control acquisition costs, which increased by $135 billion over the past 2 years for 98 major defense acquisition programs. GAO was asked to determine (1) DOD's progress in implementing the Reform Act's requirements and (2) whether there are challenges at the military service level that could affect their systems engineering and developmental testing activities. To do this, GAO analyzed implementation status documents, discussed developmental testing office concerns with current and former DOD officials, and analyzed military service workforce growth plans and test range funding data..
section_title: What GAO Found
paragraphs: The new offices for systems engineering and developmental test and evaluation are continuing to make progress implementing Reform Act requirements. Since GAO's 2010 report on this topic, the Deputy Assistant Secretaries for Systems Engineering and Developmental Test and Evaluation have issued additional policies and guidance, assisted more weapons acquisition programs in the development of acquisition plans, and provided input to senior leaders at Defense Acquisition Board meetings. DOD also designated the Deputy Assistant Secretary for Developmental Test and Evaluation for concurrent service as the Director of the Test Resource Management Center. This was an optional Reform Act provision, which places oversight of testing resources and acquisition program developmental testing activities under one official. Despite these steps, the developmental test and evaluation office reports having difficulty covering its portfolio of about 250 defense acquisition programs with its current authorized staff of 63 people. Current and former testing officials believe the office needs more influence and resources to be effective, but they said thorough analysis has not been done to determine the appropriate office size. Further, according to the Deputy Assistant Secretary for Developmental Test and Evaluation, a statutory provision that designates the Test Resource Management Center as a field activity may limit his ability to achieve management and reporting efficiencies that could be obtained by combining or shifting resources between the two organizations. GAO has a matter for Congressional consideration to allow shifting resources between the Test Resource Management Center and the developmental test and evaluation office. The military services are facing workforce challenges that could curb systems engineering and developmental testing efforts, if not properly addressed. The services planned to increase their systems engineering and test and evaluation career fields by about 5,000 people (14 percent) and about 300 people (4 percent), respectively, between fiscal years 2009 and 2015 through hiring actions and converting contractor positions to government positions. The services have increased the systems engineering career field by about half of its projections and exceeded its planned growth for the test and evaluation career field through the end of fiscal year 2010. However, future growth may be difficult because of budget cuts and a clarification in DOD's insourcing approach, which may make civilian hiring more difficult. For example, the services now plan to hire about 800 fewer systems engineers by 2015 than they originally projected. Further, cuts to development test ranges' fiscal year 2012 budgets of nearly $1.2 billion (17 percent) over the next 5 years could offset some of the workforce gains already achieved. Currently, the services lack metrics that could be used to justify funding levels, effectively allocate funding cuts, make workforce decisions, or make difficult decisions related to mothballing, closing, or consolidating test capabilities, if future budget cuts are necessary. To the extent DOD cannot provide adequate systems engineering and developmental testing support to its weapon systems portfolio, the risks of executing the portfolio within cost and schedule are increased.
section_title: What GAO Recommends
paragraphs: GAO recommends that DOD assess the resources needed by the developmental test and evaluation office, develop a plan to implement the assessment, develop metrics to aid funding decisions, and report the effect budget cuts are having on the services' ability to meet program office needs. GAO also has a matter for congressional consideration. DOD concurred with two recommendations, and offered clarifying language, which GAO incorporated, on the other two recommendations for which DOD partially concurred.

section_title: Letter
paragraphs: For the past 2 years, the Department of Defense (DOD) has been  implementing provisions of the Weapon Systems Acquisition Reform Act  of 2009 (Reform Act) related to systems engineering and developmental  testing. Greater attention to these activities provides the Office of the  Secretary of Defense an opportunity to affect weapon acquisition plans  early and assess technical risks throughout weapon system development.  In addition, it could help control total acquisition costs, which we recently  reported had increased by $135 billion over the past 2 years for DOD’s  current portfolio of 98 major defense acquisition programs.
Last year, we reported on the status of DOD’s initial efforts to implement  Reform Act requirements related to systems engineering and  developmental testing. We found that DOD had taken steps to implement  the Reform Act requirements, including establishing new offices for the  Director of Systems Engineering and the Director of Developmental Test  and Evaluation within the Office of the Secretary of Defense. The offices  have since been renamed as the offices of the Deputy Assistant  Secretary of Defense for Systems Engineering and the Deputy Assistant  Secretary of Defense for Developmental Test and Evaluation. We found  that there were concerns about the amount of influence the Deputy  Assistant Secretary for Developmental Test and Evaluation could have on  weapon acquisition programs based upon where the office is placed  organizationally within the department. However, we could not determine  whether the office had the appropriate amount of influence because it  was not tracking the extent to which its recommendations were being  adopted or impacting weapon programs. We also identified issues the  military services face as they enhance systems engineering and  developmental testing activities on their weapon acquisition programs,  including determining whether they have enough people to perform these  activities, training the influx of new hires they expect, and addressing test  range resource needs.
Based on our initial work, the Senate Armed Services Committee asked  us to continue to monitor DOD’s efforts to implement the Reform Act  provisions, as well as look at the military services’ systems engineering  and developmental test and evaluation capabilities. Our specific  objectives were to determine (1) DOD’s progress in implementing the  Reform Act’s systems engineering and developmental testing  requirements within the Office of the Secretary of Defense and               (2) whether there are challenges at the military service level that could  affect their systems engineering and developmental testing activities.
In conducting our work, we analyzed information obtained from the offices  of the Deputy Assistant Secretaries of Defense for Systems Engineering  and Developmental Test and Evaluation to determine the status of their  efforts to implement the Reform Act legislation. We also solicited the  views of current and former DOD developmental testing officials about the  effectiveness of the developmental test and evaluation office. In addition,  we interviewed officials and analyzed pertinent documents related to  workforce and test range issues from 12 DOD test ranges, the Test  Resource Management Center, and cognizant military service systems  engineering, developmental testing, and personnel offices. For purposes  of this report, we use the term systems engineering career field to refer to  two Systems Planning, Research Development, and Engineering career  fields—Systems Engineering and Program Systems Engineering. The  test and evaluation career field is a combination of developmental and  operational testing personnel. Both of these career fields represent a  portion of the total systems engineering and developmental testing  workforce. See appendix I for a more detailed explanation of our scope  and methodology.
We conducted this performance audit from September 2010 to  September 2011 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: DOD significantly downsized its acquisition workforce in the 1990s and  2000s as part of an overall reduction of military and civilian defense  personnel after the end of the Cold War. According to DOD’s April 2010  Strategic Human Capital Plan Update, the department decreased its core  acquisition workforce from about 146,000 people in fiscal year 1998 to  about 126,000 in fiscal year 2008. Meanwhile, the number of major  defense acquisition programs increased from 76 to 93 and total estimated  costs to acquire them increased from nearly $805 billion to more than  $1.6 trillion. The systems engineering and test and evaluation career  fields were affected by the workforce cuts. For example, in 2008 the  Defense Science Board reported that the Army’s test and evaluation  workforce was reduced by more than 55 percent from 1991 through 2007.  In response to cuts in these career fields, DOD reduced its emphasis in  these areas and/or relied on prime contractors to analyze and interpret  developmental testing data. The Defense Science Board also found that  many weapon programs were failing initial operational testing due to a  lack of a disciplined systems engineering approach. Additionally, in 2008  the National Academy of Sciences reported that there were no longer  enough systems engineers to fill programs’ needs. The Academy  observed that DOD cannot outsource its technical and program  management experience and intellect and still expect to acquire new  systems that are both effective and affordable.
Over the past several years, the Congress has called for DOD to improve  its acquisition workforce and increase emphasis on systems engineering  and developmental testing during weapon systems development. For  example, the National Defense Authorization Act for Fiscal Year 2008  required the Secretary of Defense to establish the DOD Acquisition  Workforce Fund for the recruitment, training, and retention of DOD  acquisition personnel, which includes the systems engineering and test  and evaluation career fields. In addition, the Reform Act contains a  number of systems engineering and developmental testing requirements  aimed at helping weapon programs establish a solid foundation prior to  the start of development. For example, the Deputy Assistant Secretaries  are expected to review and approve acquisition planning documents for  major defense acquisition programs, as well as monitor program  activities.
In response to the Reform Act legislation, DOD established new offices  within the Office of the Secretary of Defense for systems engineering and  developmental testing to provide assistance to program offices as they  are developing their acquisition strategies prior to the start of  development and then oversee program office efforts to implement the  strategies. The systems engineering office has about 120 programs in its  portfolio and the developmental test and evaluation office has about     250 programs. In addition, they provide advocacy, oversight and  guidance for their respective workforces throughout DOD. The Secretary  of Defense also announced plans to increase the number of people  performing acquisition activities by almost 20,000 employees through new  hiring actions and by converting contractor positions to government  positions (insourcing) between fiscal years 2009 and 2015. The  Strategic Human Capital Plan Update indicates that 22 percent of the  almost 20,000 position growth will be for systems engineering and 1  percent for test and evaluation, which would increase the size of the  career fields by 16 percent and 5 percent, respectively. The majority of  these new positions will be in the military services at headquarters,  program offices, and test range locations.
The services maintain test ranges for development testing activities.  Currently, 24 ranges are designated as part of the Major Range and Test  Facility Base (MRTFB) because they have unique test capabilities that  are used by multiple services. The Test Resource Management Center is  responsible for ensuring that the MRTFB is adequately funded and  maintained while the services oversee the remaining non-MRTFB ranges  and facilities, which perform more service-specific testing.
section_title: DOD Is Continuing to Implement Reform Act Requirements, but Challenges Remain
paragraphs: The new offices of the Deputy Assistant Secretary of Defense for  Systems Engineering and Developmental Test and Evaluation have  continued to make progress implementing the Reform Act requirements.  Both offices, which provide assistance to program managers and perform  oversight for decision makers, have issued additional policies and  guidance, and are overseeing and providing information on more weapon  acquisition programs than they did last year. The systems engineering  office will be staffed at 163 people in fiscal year 2012 and the  developmental test and evaluation office will be staffed at 63 people,  which is less than the developmental test and evaluation office originally  projected. Yet, the offices are more reliant on contractor support than the  Deputy Assistant Secretaries would like. In addition, many current and  former testing officials continue to believe the developmental test and  evaluation office does not have the resources or influence to effectively  oversee and affect program decisions. However, it is unclear how many  people are needed.
subsections: section_title: Efforts to Implement Reform Act Requirements Are Progressing
paragraphs: DOD established the two offices for systems engineering and  developmental test and evaluation within the Office of the Secretary of  Defense in June 2009. Since then, both offices have increased staffing,  which is enabling them to meet statutory requirements for assisting and  overseeing their portfolios of defense acquisition programs. The table  below shows the actual and planned workforces for both offices.
Most of the staffing increases have been through hiring contractor  employees. While both offices have also increased their government staff,  hiring freezes have curbed their ability to hire additional government  employees and for the developmental test and evaluation office to meet  its authorized staffing goal. The systems engineering office was  authorized 28 government employees for fiscal year 2011, but has not yet  been given permission to advertise for all of the positions because of the  hiring freeze. According to the Deputy Assistant Secretary for  Developmental Test and Evaluation, the office was initially authorized to  have 70 employees, but will be capped at 63 in fiscal year 2011, 4 of  which are detailees from the Test Resource Management Center.
In their fiscal year 2010 joint annual report to the Congress, the Deputy  Assistant Secretaries reported on their ongoing efforts to implement  Reform Act requirements. The table below includes information that was  included in the report or provided to us on their fiscal years 2009 and  2010 activities. As shown, the offices have continued or increased their  activities.
In addition to the progress highlighted in the table above, the offices are  also taking actions in areas that they had not acted upon last year— issuing required guidance on the development and tracking of  performance criteria and exercising a Reform Act option of designating  the Deputy Assistant Secretary of Defense for Developmental Test and  Evaluation for concurrent service as the Director of the Test Resource  Management Center.    Developing performance criteria: The Reform Act requires the Deputy  Assistant Secretaries, in coordination with an official designated by  the Secretary of Defense, who was the Director of the Performance  Assessments and Root Cause Analysis, to jointly issue guidance on  the development and tracking of detailed measurable performance  criteria for major defense acquisition programs. In response to this  requirement, the Deputy Assistant Secretaries, in cooperation with the  Director of Performance Assessments and Root Cause Analysis, have  agreed that each office shall develop guidance within their respective  functional areas in coordination with each other and openly share the  data and findings of those performance criteria in conducting their  oversight.
The Deputy Assistant Secretary for Systems Engineering developed a  set of time-based metrics to assess each program’s ability to execute  its system engineering plans and address risks the office had  identified in prior reviews. The metrics measure program cost,  schedule, staffing, reliability, availability and maintainability, software,  integration, performance and manufacturing, and are to be  incorporated into each program’s systems engineering plan and  evaluated at various points in the development process. Criteria  developed by the Deputy Assistant Secretary for Developmental Test  and Evaluation focus on early acquisition lifecycle activities to ensure  that sound developmental testing planning is performed from the  beginning of development. Other criteria measure program results  and are meant to provide an objective foundation to assess a  program’s subsequent developmental testing performance as it  approaches the production decision and the assessment of  operational test readiness. The office plans to pilot test the metrics on  six programs, with a goal of rolling them out to other programs by  the end of 2011.    Changing leadership of Test Resource Management Center: Effective  April 1, 2011, the Principal Deputy Under Secretary of Defense for  Acquisition, Technology and Logistics designated the Deputy  Assistant Secretary of Defense for Developmental Test and  Evaluation for concurrent service as the Director of the Test Resource  Management Center. DOD had not acted upon this optional Reform  Act provision last year. Both offices will continue to be managed  separately and report to different authorities—developmental test and  evaluation office activities will be reported to the Assistant Secretary  of Defense for Research and Engineering and Test Resource  Management Center activities will be reported directly to the Under  Secretary of Defense for Acquisition, Technology and Logistics.
According to the Deputy Assistant Secretary of Defense for  Developmental Test and Evaluation, the Principal Deputy Under  Secretary for Acquisition, Technology and Logistics asked him to  complete a study in the near future to identify efficiencies that can be  obtained by merging some of the offices’ activities. A study was  started in July 2011. The Deputy Assistant Secretary for  Developmental Test and Evaluation indicated that there could be  some limitations on his ability to streamline management and  reporting activities or shift resources between the organizations  because the Test Resource Management Center is designated by  statute to be a field activity and the organizations are funded  separately. DOD has not studied the possible legal ramifications of  combining the offices.
In their fiscal year 2010 joint annual report to the Congress, the Deputy  Assistant Secretaries also identified several focus areas for improvement.  For example, the systems engineering office plans to reestablish the DOD  Software Working Group to improve DOD’s capability to address systemic  software program issues. Among other things, the developmental test and  evaluation office wants to develop a responsible test organization model  that the services would use to designate the lead government test  organization responsible for overseeing and/or conducting the  developmental test and evaluation for an acquisition program. In addition,  the office wants to issue a policy requiring programs to prioritize use of  government capabilities and to provide a cost-benefit analysis when they  decide to provide funding to use and/or develop test capabilities at prime  contractor sites.
In a departure from the fiscal year 2009 joint annual report, the fiscal  year 2010 report did not contain a discussion of the extent to which  weapon acquisition programs are fulfilling their systems engineering or  test and evaluation master plans. Further, the fiscal year 2010 report did  not include a discussion of test and evaluation waivers or deviations that  programs have received. Instead, the report identified the type of reviews  or engagements both of the offices participated in for various programs.  The Deputy Assistant Secretaries stated that they did not provide the  information in the fiscal year 2010 report because they were directed to  streamline the report. In May 2011, the Senate Armed Services  Committee requested DOD to supplement the fiscal year 2010 report with  this information as required by the Reform Act. DOD has not yet provided  the information.
The Deputy Assistant Secretaries believe they have had a positive  influence on weapon acquisition programs over the past 2 years during  milestone reviews with senior department leaders and through  recommendations to program offices. Although the Assistant Secretary of  Defense for Research and Engineering represents both systems  engineering and developmental test positions at Defense Acquisition  Board meetings, the Deputy Assistant Secretaries said that either they, or  designees, also attend the meetings. They said that they have been  asked by the Principal Deputy for Acquisition, Technology and Logistics  to provide direct input about weapon acquisition programs at some of  these meetings, including discussions on the Gray Eagle unmanned  aircraft system, Ohio Replacement submarine, and the Joint Strike  Fighter aircraft programs. The Deputy Assistant Secretaries also said that  program offices are making changes based on the recommendations  made by their offices during regular program assessments and technical  reviews.
section_title: Organizational Challenges Remain
paragraphs: While the Deputy Assistant Secretaries have made progress  implementing the Reform Act requirements, we identified several  organizational challenges that could limit their effectiveness. A summary  of each of these challenges is presented below.    Reliance on contractor employees: The Deputy Assistant Secretaries  rely heavily on contractors to help perform office activities. In fiscal  year 2010, for example, nearly 85 percent of the staff in the systems  engineering office and 67 percent of the staff in the developmental  test and evaluation office were contractors. Both Deputy Assistant  Secretaries would like to have a larger proportion of government  employees because they believe it is important to maintain a core  cadre of people with the required institutional knowledge and skills to  support current and future program office needs. However, they are  not optimistic about their chances of getting additional government  employees because of a civilian hiring freeze.   Developmental Test and Evaluation Office influence: Current and  former DOD test and evaluation officials continue to believe the  developmental test and evaluation office could be more effective in its  oversight role with the proper influence. For example, they pointed out  that the office’s primary avenue for voicing concerns about weapon  acquisition programs to senior leaders is at overarching integrated  product team meetings that take place in preparation for Defense  Acquisition Board meetings. The integrated product team leader  ultimately decides which organizations will get to present issues at the  Defense Acquisition Board meetings. Current testing officials told us  that in some cases developmental testing issues do not make the  Defense Acquisition Board meeting agendas, which is a concern to  officials who believe the developmental test and evaluation office  should provide independent assessments of weapon acquisition  programs directly to senior leaders. In addition, officials said that the  deputy directors who attend the overarching meetings are not at the  senior executive level like other meeting attendees, which officials  said in some cases has reduced their relative influence during these  meetings. To ensure developmental testing information, such as the  assessment of operational test readiness, receives appropriate  consideration by senior leaders, the Defense Science Board  recommended in 2008 that the office report developmental testing  issues directly to the Deputy Under Secretary of Defense for  Acquisition and Technology. Currently, the office reports through an  intermediary—the Assistant Secretary of Defense for Research and  Engineering.
Even though the systems engineering office reports through the same  reporting channel as the developmental test and evaluation office, the  Deputy Assistant Secretary for Systems Engineering believes his  office has the appropriate amount of influence. This is because the  systems engineering office’s primary emphasis is on assisting  program managers in the development of their systems engineering  plans. In contrast, the Deputy Assistant Secretary for Developmental  Test and Evaluation believes his office should put about equal effort  into assisting and assessing program office activities.     Developmental Test and Evaluation Office resources: Information  provided by the developmental test and evaluation office shows the  office can not provide full coverage of its portfolio of about 250  acquisition programs given its current workforce. For example, in  fiscal year 2010 when the office had 30 people, 89 programs            (36 percent) did not receive any support, including the Air and Missile  Radar program and sub elements of the Early Infantry Brigade  Combat Team program, such as the Small Unmanned Ground  Vehicles and Unmanned Aircraft System.
Although staffing has doubled between fiscal years 2010 and 2011,  the office still has not been able to support all the programs the  Deputy Assistant Secretary for Developmental Test and Evaluation  believes it should. The Deputy Assistant Secretary for Developmental  Test and Evaluation said the office has had to be selective in using its  resources and he has introduced a “triage” strategy for dealing with  the overload of programs relative to the office’s workforce. This  strategy includes dropping virtually all programs below Acquisition  Category I from the developmental testing oversight list and  eliminating oversight of some major automated information systems.  For the most part, the office focuses its efforts on major defense  acquisition programs between milestone B (development start) and  milestone C (the production decision) in order to retain sufficient depth  with programs in development. It is providing minimal coverage to  programs prior to the start of development, which is the most  opportune time to influence a program’s acquisition strategy. Based  on the “triage” strategy, the Deputy Assistant Secretary for  Developmental Test and Evaluation believes only about half of the  current portfolio of about 250 programs would receive the level of  support he believes is needed with a staff of 63 people. On the other  hand, the Deputy Assistant Secretary for Systems Engineering  believes his office, which is supposed to have 163 people in fiscal  year 2012, will have enough staff to oversee its portfolio of 120  programs.
While current and former DOD testing officials provided reasons for  increasing the size of the developmental test and evaluation office,  they could not specify the appropriate size for the office, as they  indicated the issue has not been thoroughly analyzed. Officials  familiar with the establishment of the office told us that three staffing  scenarios were considered prior to the office being established—a  high, medium, and low staffing scenario—but they indicated that no  detailed analysis was done to support any of the scenarios. Under the  high staffing scenario, the developmental test and evaluation office  would have had 250 people, which would have matched the size of  the office of the Director of Operational Test and Evaluation. The  medium level of staffing, 120-150 people, was based on the size of a  legacy developmental testing organization and the low level, which  called for 90 people, was based on an assumption about the number  of programs each person would be responsible for overseeing. As  shown earlier in table 1, the staffing goal is 70 people, which is fewer  than the lowest staffing scenario considered.
Former testing officials believe an opportunity exists to both increase  the developmental testing office’s influence and address resource  concerns by merging Test Resource Management Center and  developmental testing office activities. They believe this would give  the Deputy Assistant Secretary for Developmental Test and  Evaluation the most flexibility in how to allocate resources. They also  pointed out that in the early 1990s, oversight of all developmental test  and evaluation activities had been under one organization that  reported directly to the Under Secretary of Defense for Acquisition.
section_title: Workforce Challenges Could Affect Services’ Systems Engineering and Developmental Testing Efforts
paragraphs: The military services have been increasing the number of people in their  systems engineering and test and evaluation career fields, but challenges  exist that could impede future workforce growth plans as well as testing at  the ranges, if not properly addressed. The services planned to increase  their systems engineering and test and evaluation career fields by about  5,000 people (14 percent) and about 300 people (4 percent), respectively,  between fiscal years 2009 and 2015 through hiring actions and by  insourcing contractor positions. These increases are part of DOD’s overall  efforts to increase the number of people in acquisition career fields. The  services have increased the systems engineering career field by about  half of their projections and exceeded their planned growth for the test  and evaluation career field through the end of fiscal year 2010. However,  budget cuts and a clarification in DOD’s insourcing approach may make  hiring civilians more challenging in the future. The services’  developmental test ranges are also experiencing declining budgets, as  the fiscal year 2012 budget includes cuts of nearly $1.2 billion over the  next 5 years to support accounts that pay for overhead costs. The  services plan to cut range personnel in response to the budget  reductions, which could offset some of the workforce gains they have  already achieved. They have not yet determined how the cuts will be  allocated across the ranges or the impact they will have on meeting  program office needs. Further budget cuts are possible based on the  recent debt ceiling agreement, but details are unknown. Currently, the  services lack common performance metrics that would assist in making  funding decisions.
subsections: section_title: Services Have Increased Their Systems Engineering and Test and Evaluation Career Fields
paragraphs: The services planned to increase their acquisition systems engineering  and test and evaluation career fields by 14 percent and 4 percent,  respectively, between fiscal years 2009 and 2015 through hiring and  insourcing actions. This would increase the overall systems engineering  career field by about 5,000 people from about 35,000 to 40,000 people  and the test and evaluation career field by almost 300 people from about  7,400 to 7,700 people. These new positions would be located at service  headquarters, program offices, and test range locations. It should be  noted that insourcing actions alone do not result in real growth to the  collective number of civilians, military, and contractors performing an  activity because it only involves the transfer of positions from contractors  to the government. Hiring additional civilian employees, on the other  hand, would result in growth, assuming the contractor and military  workforces remain stable.
The following table shows the baseline, goal, and current number of  civilian and military personnel performing systems engineering and test  and evaluation activities for each of the services at the end of fiscal year  2010, as well as the percentage of the growth goal target achieved.
Through the end of fiscal year 2010, the services have made significant  progress towards increasing the two career fields. For example, the  services increased the systems engineering career field by about       2,600 people, about half of the projected growth. Collectively, they have  achieved 94 percent of the growth goal target planned for that career  field. The services achieved 104 percent of the growth goal target  planned for the test and evaluation career field by adding over               600 people. Information provided by the Army and Navy shows about            74 percent of their systems engineering career field growth was through  new hires and the remaining 26 percent was through insourcing. About  86 percent of their test and evaluation career field growth was through  new hires and the remaining 14 percent was through insourcing.
section_title: Budget Cuts and Other Challenges Could Affect Overall Workforce Growth
paragraphs: Recently proposed budget changes would result in modifications to the  services’ workforce growth plans for both acquisition career fields. The  services still plan to increase the systems engineering career field  between fiscal years 2011 and 2015, but with about 800 positions less  than originally planned. As a result, career field growth would be             10 percent instead of 14 percent. On the other hand, the services plan to  hire 400 more people than they originally planned for the test and  evaluation career field, despite the fact that the overall number of DOD  civilians is frozen at fiscal year 2010 levels. This would result in a              6 percent growth to the career field instead of 4 percent.
Service officials stated that achieving additional career field growth,  however, could be difficult because of a recent clarification in DOD’s  insourcing policy. According to a March 2011 memorandum issued jointly  by the Under Secretary of Defense for Acquisition, Technology and  Logistics and the Under Secretary of Defense Comptroller/Chief Financial  Officer, a case-by-case approach will be used for additional insourcing of  acquisition functions based on critical need, whether a function is  inherently governmental, and the benefit demonstrated by a cost-benefit  analysis. The memorandum also states that additional insourcing must be  supported by current budget levels. In cases where added insourcing  would breach the existing civilian ceilings, the proposal and associated  justification must be provided to the Director of Human Capital Initiatives  and then the proposal will be reviewed by the two Under Secretaries of  Defense issuing the memorandum and approved by the Deputy Secretary  of Defense. Of the nearly 2,000 people the Army and Navy now plan to  hire for the systems engineering career field between fiscal years 2011  and 2015, 96 percent is to be achieved through insourcing. And about    40 percent of the over 200 people they plan to hire for the test and  evaluation career field are to be through insourcing.
Although the services have increased their test and evaluation staff, each  of the 12 test ranges we visited experienced a mixture of recruiting, hiring,  training, or retention challenges. For example, Pacific Missile Range  Facility officials stated the range’s location on the island of Kauai, Hawaii,  has made it difficult to recruit personnel due to a lower pay grade  structure and higher costs of living compared to other test ranges. Range  officials stated that it can take months to hire new employees, forcing  many qualified applicants to seek employment opportunities elsewhere.  Test ranges have begun using acquisition workforce expedited hiring  authority, which allows DOD components to streamline the process for  making offers to qualified acquisition personnel. However, ranges have  interpreted this authority differently. Service test and evaluation  executives said they would clarify the policy to the ranges based upon our  observations to ensure the policies are fully understood. Most of the test  range officials we spoke with also had concerns about the timeliness and  quality of the Defense Acquisition University’s training classes. The  Defense Acquisition University is coordinating with the services to  address the increased demand for acquisition training. Finally, some test  ranges are having difficulty retaining engineers or some software  specialties because the private sector can pay them a higher salary.  Officials at Aberdeen Test Center, Maryland, are concerned that many of  their employees will take higher paying jobs with organizations that are  moving into their area as the result of a Base Realignment and Closure  decision.
section_title: Impact of Proposed Budget Cuts on Ranges Has Not Been Determined
paragraphs: The services’ developmental test range budgets are being reduced, but  the full impact of the cuts has yet to be determined. The fiscal year 2012  President’s Budget includes cuts of nearly $1.2 billion (17 percent)  through fiscal year 2015 to the ranges’ institutional funding accounts that  fund operational and overhead expenses such as personnel, facilities,  and equipment costs. According to service officials, the budget cuts are  a result of direction they received late in the fiscal year 2012 budget cycle  to keep their civilian workforce at the same level as they had at the end of  fiscal year 2010. Research, development, test and evaluation accounts  that fund range testing activities absorbed a large portion of the cuts.
As shown in figure 1, the services had already planned to reduce their  MRTFB research, development, test and evaluation funding between  fiscal years 2012 and 2015. Although range budgets reached a peak in  fiscal year 2011, budget estimates for that year projected funding  decreases in subsequent years. According to service officials these  reductions were expected due to expected workload reductions, savings  associated with implementing efficiency initiatives, and a civilian pay  freeze. The fiscal year 2012 budget contains additional, more significant  decreases beyond those forecasted in the previous year’s budget. In  total, this budget provides $5.7 billion for the ranges between fiscal years  2012 and 2015—a 17 percent reduction from the $6.9 billion forecast in  the fiscal year 2011 budget. Most notably, the Army and Air Force range  budgets were each cut over $100 million in fiscal year 2012 alone.
The Chairman of the Senate Armed Services Committee recently  expressed concern that these proposed reductions threaten to seriously  undermine the implementation of the Reform Act’s requirement to rebuild  DOD’s systems engineering and developmental testing organizations and  requested that the proposed cuts be reviewed. The Army and Air Force  are in the process of reviewing these cuts and determining where they will  be made. To minimize the impact on these activities, they are examining  potential reductions to their overhead or administrative staff at  headquarters and field locations. According to service headquarters  officials, if DOD’s proposed budget cuts between fiscal years 2012 and  2015 remain intact, they may have to cut some of their developmental  test capabilities as well as personnel who conduct tests. This could offset  some of the test and evaluation career field gains already achieved over  the past 2 years. Test officials said that reductions in test personnel could  limit the amount of testing performed on weapon acquisition programs,  which could increase the risk associated with those programs and/or  result in an extension of a program’s test schedule. Final decisions on  where to take the cuts have not yet been made. Therefore, we could not  determine the impact funding cuts would have on the ranges’ ability to  meet program office testing needs.
Range officials indicated that prior to these proposed cuts they had  difficulty maintaining their test capabilities. The National Defense  Authorization Act for Fiscal Year 2003 required the Secretary of  Defense by fiscal year 2006 to establish the funding objective of ensuring  that the overhead and institutional costs of the MRTFB ranges were fully  funded through the major test and evaluation investment accounts and to  ensure that DOD customers were charged not more than the direct costs  of testing. The law also required DOD to establish the Test Resource  Management Center and required that the director of that organization  certify whether the services’ proposed budgets for test and evaluation  activities are adequate. To comply with this law, the services increased  their range operating budgets by over 50 percent in fiscal year 2006.  Although MRTFB funding was fairly stable between fiscal years 2006 and  2011, range officials said they have had difficulties maintaining their test  capabilities because the infrastructure is aging and operating costs for  expenses like utilities and fuel have grown at a higher rate than their  overall funding. As a result, officials said they have had to move money  from their range modernization accounts to fund operating costs, only fix  things that are broken or in emergency status, or fund capability upgrades  over several years instead of a shorter period of time. According to range  officials, these challenges are less of a problem for non-MRTFB ranges  like the Redstone Test Center and other test/laboratory activities funded  through working capital fund accounts, where customers pay direct and  overhead costs.
Service officials discussed several strategies or a combination of those  strategies that the services could use to lessen the impact of budget cuts  and funding concerns. One strategy is that the ranges may have to curtail  certain testing, mothball or close test facilities, or consolidate test  capabilities. Another option could be to move certain test capabilities out  of the MRTFB management structure, thereby allowing ranges to charge  customers the full cost of testing. Additionally, on the basis of our range  observations, service officials said more specific guidance for interpreting  financial management regulations on what constitutes direct and  overhead charges for MRTFB operations could be developed to clarify  and standardize the types of costs that can be passed on to customers.  Service test and evaluation executives and the Deputy Assistant  Secretary for Developmental Test and Evaluation believe the ranges’  current policy interpretation is too restrictive, inconsistent across the  MRTFB, and constrains service options as they strive to sustain MRTFB  capabilities. The Test Resource Management Center recently established  a team to study the issue.
section_title: Services Lack Common Range Performance Measures to Aid Decision Making
paragraphs: The services have not implemented common range performance  measures that would help them justify funding and assist them in making  workforce decisions, how best to allocate funding, or make difficult  decisions about mothballing, closing, or consolidating test capabilities, if  necessary. Although ranges collect performance data relevant to their  operations, these indicators may not be useful in making higher-level  infrastructure decisions that cut across several ranges. According to  service officials, it is very difficult to develop a common set of  performance measures because of the uniqueness of each range and its  variable capacity. We have found that performance measures can assist  managers in making decisions about future strategies, planning and  budgeting, identifying priorities, and allocating resources.
Some efforts are under way to provide decision makers more information,  but they are still in process and are not yet approved or implemented. The  Test Resource Management Center has sponsored an effort to develop a  comprehensive set of range metrics. According to the services, early  efforts were not successful and were not well received. While the Air  Force plans to use metrics resulting from the Test Resource Management  Center’s effort, the Army is evaluating the development of a readiness  reporting system and a workload model for its ranges that could provide a  better basis for investment or divestment decisions. The Navy is also in  the process of developing metrics for its MRTFB test capabilities on the  systems’ condition, capacity, competency, and importance. Once  developed, these metrics are expected to assist decision makers in  directing future investment funding and to ensure test capabilities are  adequate to support programs.
section_title: Conclusions
paragraphs: The Reform Act points to the need for DOD to develop more robust  systems engineering and developmental test and evaluation capabilities.  DOD established new organizations and the services developed resource  plans in order to increase emphasis on these activities. From an  organizational standpoint, the offices of the Deputy Assistant Secretaries  for Systems Engineering and Developmental Test and Evaluation are  providing assistance to program managers in developing acquisition  plans and providing oversight of program efforts for the Under Secretary  of Defense for Acquisition, Technology and Logistics and the Congress.  However, the developmental testing office is not as robust or efficient as it  could be in part due to resource and organizational constraints. In  addition, while the military services have made significant progress to  date in increasing their systems engineering and developmental testing  workforce capabilities, planned workforce reductions could offset these  gains.
It is incumbent upon DOD to provide the most effective systems  engineering and developmental testing capability it can afford. It should  have a sound analytical basis for establishing and resourcing that  capability. However, it is not clear that DOD is yet at this point, especially  for developmental testing. DOD has not conducted analysis on the right  size of the developmental test and evaluation office or captured data that  would reinforce range funding and workforce actions, or suggest needed  adjustments. Additionally, the Deputy Assistant Secretary for  Developmental Test and Evaluation believes that there may also be a  statutory provision that limits his ability to achieve efficiencies and  address office challenges. To the extent DOD cannot provide adequate  systems engineering and developmental testing support to its weapons  portfolio, the risks of executing the portfolio within cost and schedule are  increased.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Defense take the following two  actions:    assess the resources and influence needed by the developmental test  and evaluation office to assist and oversee defense acquisition  programs, including    the number of defense acquisition programs that can be  supported by different developmental test and evaluation office  staffing levels, including specifying the total number of personnel,  the mix of government and contractor employees, and the number  of senior executive service personnel needed for each of these  staffing levels;    whether the Test Resource Management Center and the office of  the Deputy Assistant Secretary for Developmental Test and  Evaluation should be combined or resources shifted between  organizations to more effectively support the activities of both  organizations and if so, identify for Congress any statutory  revisions that would be necessary; and  the proper reporting channel, taking into account the decision on  whether or not to combine the organizations, the statutory  oversight requirements, and the level of influence needed to  oversee and assess program office developmental testing and  service budgeting activities.    develop a plan to implement the results of the assessment.
We also recommend that the Secretary of Defense, with input from the  military services, take the following two actions:    develop metrics to assess the MRTFB test capabilities (expanding to  DOD non-MRTFB, and non-DOD government test facilities once an  approved set of metrics are in place supporting the MRTFB), justify  funding, and assist in making decisions on the right-sizing of  personnel, how best to allocate funding, or make future decisions on  whether to mothball, shut down, or consolidate test facilities. These  efforts should be coordinated with the Test Resource Management  Center.  report the impact budget cuts reflected in the fiscal year 2012 budget,  as well as the insourcing policy clarification, will have on their (1) total  workforce (civilians, military, and contractors) that support both of  these activities and (2) ability to meet program office systems  engineering and developmental test and evaluation needs.
section_title: Matter for Congressional Consideration
paragraphs: Contingent upon the results of DOD’s assessment, the Congress may  want to consider revising any applicable statutory provisions necessary to  allow for DOD to combine or shift resources between the Test Resource  Management Center and the office of the Deputy Assistant Secretary for  Developmental Test and Evaluation.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided us with written comments on a draft of this report. DOD  concurred with two recommendations and partially concurred with two  others. DOD offered suggested wording changes to the recommendations  where it partially agreed to offer greater clarity. We agreed with the  suggested changes and reworded our recommendations accordingly.  DOD’s comments appear in appendix II. DOD also provided technical  comments, which we incorporated as appropriate in the report.
In its response, DOD noted that the Deputy Assistant Secretary for  Developmental Test and Evaluation has directed a study to assess the  resources and influence needed by the developmental test and evaluation  office to assist and oversee defense acquisition programs. The Deputy  Assistant Secretary for Development Test and Evaluation will develop a  plan to implement actionable recommendations from the results of the  assessment, with the approval of the Under Secretary of Defense for  Acquisition, Technology and Logistics. The Deputy Assistant Secretary  for Developmental Test and Evaluation also plans to establish a working  group, with participation from the military services, to develop metrics to  assess MRTFB test capabilities, justify funding, and assist in making  decisions on human capital and test facilities management. Finally, the  Deputy Assistant Secretaries for Systems Engineering and  Developmental Test and Evaluation plan to identify any impacts to the  state of the workforce due to funding modifications or DOD’s workforce  policy updates in their joint annual report to the Congress.
We are sending copies of this report to the Secretary of Defense, the  Director of the Office of Management and Budget, and interested  congressional committees. We will also make copies available at no  charge on the GAO web site at http://www.gao.gov.
If you have any questions about this report or need additional information,  please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may   be found on the last page of this report. Key contributors to this report are  listed in appendix III.
section_title: Appendix I: Scope and Methodology
paragraphs: This report examines the military services’ systems engineering and  developmental testing workforce capabilities and the Department of  Defense’s (DOD) efforts to implement Weapon Systems Acquisition  Reform Act of 2009 (Reform Act) requirements. Specifically, we  examined (1) the progress DOD has made in implementing the Reform  Act’s systems engineering and developmental testing requirements and  (2) whether there are challenges at the military service level that could  affect their systems engineering and developmental testing activities.
To measure DOD’s progress in implementing Reform Act requirements  related to systems engineering and developmental testing, we  interviewed officials and collected pertinent documents from the offices of  the Deputy Assistant Secretary of Defense for Systems Engineering and  Developmental Test and Evaluation. Specifically, we collected information  on their efforts to develop additional policy and guidance; review and  approve acquisition planning documents of major acquisition programs;  monitor and review activities of major acquisition programs; and develop  guidance for the development of performance metrics to use on weapon  acquisition programs. We also reviewed their staffing plans and  questioned the Deputy Assistant Secretaries, as well as former DOD  developmental testing experts on whether each of the offices has the  necessary amount of resources and influence to fulfill their missions.
In order to determine whether there are challenges at the military service  level that could affect future systems engineering and developmental  testing activities, we looked at planned and actual workforce growth and  developmental test range activities, and specifically at the following.     We collected and compared the military services’ original workforce  growth projections with their current workforce projections for the  systems engineering and test and evaluation career fields. For this  report, when we use the term systems engineering career field, we  are referring to the Systems Planning, Research Development, and  Engineering—Systems Engineering and Program Systems  Engineering career fields. The Air Force and Army’s original plans  were based on their fiscal year 2008 budget estimate submissions  and the Navy’s was based on its fiscal year 2009 budget estimate  submission. Current plans for each of the services were based on  their fiscal year 2012 budget estimate submissions. Workforce data  covered the fiscal years 2009-2015 timeframe for both the original and  current plans. We interviewed officials within the Office of the  Secretary of Defense and military services to determine the  underlying causes for variances between the two plans and to  determine how the Secretary of Defense’s cost efficiency measures  are affecting new hiring and insourcing plans.
Navy officials stated that they used fiscal year 2009 as their baseline  because before that time, information from various commands was  not centralized and they could not verify the accuracy of the numbers.  Navy officials stated that they have confidence in the validity of the  numbers that support the fiscal year 2009 baseline because factors  that affect workforce counts, such as recodes, decodes,  reassignments, and attrition, are treated consistently.   We compared the services’ baseline workforce data with end of fiscal  year 2010 workforce numbers to determine how much growth  occurred in the two career fields over the past 2 fiscal years. The Air  Force’s and Army’s workforce baseline was the end of fiscal year  2008 and the Navy’s was the end of fiscal year 2009. We assessed  the reliability of these data by interviewing knowledgeable officials in  the Air Force, Army, and Navy about the processes they use to  ensure both the integrity and reliability of their manpower workforce  databases used to track acquisition personnel. We determined that  the data were sufficiently reliable for the purposes of this report.   We conducted site visits to 12 ranges and facilities—11 of these are  designated as part of the Major Range and Test Facility Base  (MRTFB) and one is a non-MRTFB range—to determine challenges  affecting DOD’s ability to conduct developmental testing. We focused  our discussions on a broad range of topics, including funding,  workforce, range facilities and instrumentation, encroachment,  contractor duplication of DOD test facilities, and test resource capacity  and demand. We selected four ranges per service based on  geographical diversity, level of funding, type of testing, and  recommendations from the Office of the Secretary of Defense and the  services. The test ranges we visited were: Aberdeen Test Center,  Maryland; Air Force Flight Test Center, California; Arnold Engineering  Development Center, Tennessee; Atlantic Undersea Test and  Evaluation Center, Bahamas; 46th Test Group, New Mexico; 46th  Test Wing, Florida; High Energy Laser Systems Test Facility, New  Mexico; Naval Air Warfare Center – Aircraft Division, Maryland; Naval  Air Warfare Center – Weapons Division, California; Pacific Missile  Range Facility, Hawaii; Redstone Test Center, Alabama; and White  Sands Missile Range, New Mexico.   Finally, we compared developmental test range funding included in  the President’s Budget Future Year’s Defense Plan for fiscal year  2011 and 2012. We discussed differences between the two plans and  how decreases in the fiscal year 2012 plan would affect  developmental testing activities with officials from the Test Resource  Management Center, the military services, and the office of the  Deputy Assistant Secretary of Defense for Developmental Test and  Evaluation. We also discussed with these officials the progress DOD  and the services have made in developing metrics that could be used  to make workforce and investment decisions.
We conducted this performance audit from September 2010 to  September 2011 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact name above, Bruce Thomas, Assistant Director;  Cheryl Andrew; Rae Ann Sapp; Keith Hudson; Laura Greifner; and Marie  Ahearn made key contributions to this report.