Defense Acquisitions: DOD Can Improve Its Management of Configuration Steering Boards
Released: 2011-07-07T00:00:00

Letter

Background

In July 2007, the USD (AT&L) established CSBs for every current and future major defense acquisition program in development as a measure to limit requirements change and avoid cost increases. The CSBs were to have a broad membership, including senior representatives from the offices of USD (AT&L) and Joint Staff. CSBs were intended to review all requirements and significant technical configuration changes with the potential to adversely affect the program. The USD (AT&L) directed that these changes should generally be rejected or deferred unless funds and schedule adjustments could be identified to mitigate their effects. In addition, program managers were asked to identify options to reduce program cost or moderate requirements, referred to as “descoping” options, on a roughly annual basis. USD (AT&L) also instructed that, while policy would be to keep within planned costs as much as possible even at the expense of scope and content, all expected increases in program costs must be budgeted at the absolute earliest opportunity. USD (AT&L) incorporated CSBs into DOD’s primary acquisition policy— DOD Instruction 5000.02—in December 2008.

In October 2008, Congress enacted the Duncan Hunter National Defense Authorization Act for Fiscal Year 2009, which required the establishment of CSBs for the major defense acquisition programs of the military departments. According to the statute, a CSB must meet at least once each year for each of these programs. The statute also provided direction on CSB membership and responsibilities. It requires CSBs to include the appropriate service acquisition executive as chair and include representatives from USD (AT&L), the Chief of Staff for the armed forces, representatives from other armed forces as appropriate, the Joint Staff, the comptroller of the military department, the military deputy to the service acquisition executive, the program executive officer for the program concerned, and others as appropriate;  prevent unnecessary changes to programs that could have an adverse impact on program cost or schedule, mitigate adverse cost and schedule effects of changes that may be required, and ensure that each program delivers as much planned capability as possible at or below the planned cost and schedule; review and approve or disapprove any proposed changes to program requirements or system configuration with the potential to adversely affect cost and schedule; and review and recommend proposals that could reduce requirements and improve cost and schedule.

In addition, the statute provided program managers the authority to  object to adding new requirements that would be inconsistent with previously established parameters unless approved by the CSB and  propose opportunities to reduce program requirements to improve cost and schedule consistent with program objectives.

In our March 2010 assessment of selected weapon programs, we reported that only 7 of the 42 programs we assessed held CSB meetings in 2009. As a result, in the Senate report accompanying the bill for the Ike Skelton National Defense Authorization Act for Fiscal Year 2011, the Senate Armed Services Committee directed USD (AT&L) to take appropriate steps to ensure that CSBs meet at least once a year to consider the full range of proposed changes to program requirements or system configuration for each major defense acquisition program.

Compliance with the CSB Provisions in Statute Varied by Military Department

The military departments’ compliance with statutory CSB requirements varied. The Air Force and Navy did not fully comply with the requirement to hold annual CSB meetings for all major defense acquisition programs in 2010; the Army did comply. In total, the military departments held an annual CSB meeting for 74 of 96 major defense acquisition programs they managed in 2010. According to our survey results, when the military departments held CSB meetings, 19 programs endorsed requirements or configuration changes. In most of these cases, strategies were developed to mitigate any effect on a program’s cost and schedule—a key provision in the statute and DOD policy. However, key acquisition and requirements personnel were often absent from Air Force and Navy CSB meetings when these issues were discussed. Two major defense acquisition programs—the Ballistic Missile Defense System (BMDS) and the Chemical Demilitarization-Assembled Chemical Weapons Alternatives programs, which are managed by DOD components rather than military departments—are not subject to the CSB provisions in statute, but rather to DOD policy, because the statute only applies to programs overseen by military departments. This policy differs from the statute in that it only requires major defense acquisition programs in development to hold annual CSB reviews and does not require the same members, including the comptroller of the military department.

The Air Force and Navy Did Not Hold CSB Meetings for All Programs, While the Army Did

The Air Force and Navy did not hold CSB meetings for all of their major defense acquisition programs in 2010. The Air Force did not hold CSB meetings for 13 of 31 programs, and the Navy did not hold CSB meetings for 9 of 37 programs. The Army held a CSB meeting for each of its 28 major defense acquisition programs. Of the 96 major defense acquisition programs managed by the military departments, 74 held CSB meetings in 2010 and 22 failed to do so. Table 1 shows how many programs had CSB meetings by military department.

Of the 22 programs that did not have CSB meetings in 2010, 9 programs had meetings in early 2011. In addition, according to the Air Force and Navy, 8 other programs were in the process of being completed or cancelled. Table 2 includes explanations from the Air Force and Navy about why CSB meetings were not held for individual programs.

For each of the military departments, when a CSB meeting reviewed requirements or configuration changes, most were endorsed and strategies to mitigate the effects on a program’s cost and schedule were developed and discussed. However, most of the programs we surveyed did not present requirements or configuration changes to be approved or rejected at their fiscal-year-2010 CSB meetings. Specifically, our survey showed the following results:  Air Force: 6 CSB meetings reviewed requirements or configuration changes, 5 of these meetings endorsed changes, and 4 discussed the cost and schedule effects and ways to mitigate them.  Army: 6 CSB meetings reviewed and endorsed requirements or configuration changes, and 4 of these discussed the cost and schedule effects and ways to mitigate them.  Navy: 10 CSB meetings reviewed requirements or configuration changes; 8 meetings endorsed changes, and 7 of these discussed the cost and schedule effects and ways to mitigate them.

The Navy did not hold CSB reviews for all programs that experienced requirements changes in fiscal-year 2010. According to our survey results, three Navy programs changed system requirements or specifications yet did not hold a CSB meeting. Two of these programs, the Advanced Anti-Radiation Guided Missile and the Remote Minehunting System, held other high-level reviews during this period—two program management reviews and a critical Nunn-McCurdy breach review, respectively—and officials reported that a third program, the Expeditionary Fighting Vehicle, did not conduct its CSB meeting because DOD proposed canceling the program.

Key Acquisition and Requirements Personnel Were Absent from Many CSB Meetings

Key acquisition and requirements personnel were absent from many of the CSB meetings held by the Air Force and Navy in 2010. The CSB provision in the Duncan Hunter National Defense Authorization Act for Fiscal Year 2009 lists seven officials or offices that should be part of a CSB, including the service acquisition executive who should serve as the chairperson of the CSB; representatives from the acquisition, requirements, and funding communities; and others as appropriate. Army CSB meetings held in 2010 included the full array of board members in all but one case. Although USD (AT&L) was invited to the meeting in this case, Army officials reported that the office did not send a representative. The medium of CSB board members’ participation also varied among the military departments. The Army conducted all its CSB meetings in person, whereas both the Air Force and the Navy conducted virtual, otherwise known as paper, CSB meetings for certain programs in 2010 and early 2011.

The Air Force held all of its 2010 CSB meetings without key acquisition participants listed in the CSB statute. According to Air Force officials, their CSB meetings may be chaired by either the service acquisition executive or the principal military deputy to provide for flexibility in scheduling meetings. Generally, the principal military deputy acts as chair in the place of the service acquisition executive and does not attend those meetings that the service acquisition executive chairs. According to the attendee lists provided by the Air Force, only 2 of the 18 CSB meetings held were attended and chaired by the service acquisition executive. At one of those meetings neither the principal military deputy nor a representative of the comptroller was in attendance although officials report that both had been invited. The CSB meetings the service acquisition executive did not attend included numerous discussions of changes that could affect programs’ costs and schedules, including requirements and configuration changes or descoping opportunities. For example, one meeting discussed changes to the Space Based Infrared System’s architecture that could accelerate the program’s delivery of initial capability by 2 years but would cost an additional $45 million.

The Air Force also allows paper CSBs to fulfill the requirement for an annual CSB for programs it believes are stable. A program is eligible to conduct paper CSB meetings if (1) it has a Probability of Program Success score of greater than 80; (2) it has made no requirements and/or significant technical configuration changes since the last CSB that have the potential to affect the cost and schedule of the program; (3) when in production, it is in steady state production but has not reached 90 percent of planned expenditures completed or 90 percent of quantities delivered; and (4) descoping options will not yield any real cost savings. The Air Force did not conduct any paper CSBs in 2010; however, 6 of the 13 Air Force programs that did not hold a CSB meeting in 2010 conducted paper reviews in January 2011. According to Air Force officials, the process for these paper reviews began in December 2010.

The Navy held most of its 2010 CSB meetings without key acquisition and requirements personnel. The Navy has incorporated CSB meetings into the Navy’s gate review process and uses the gate 6 review, with the service acquisition executive or his designee acting as chair, to fulfill the requirement for an annual CSB. However, the Navy’s policy on gate reviews does not include the Joint Staff—a key player in the requirements process and a participant required by statute and DOD policy—as a participant, and at least 22 of the 28 CSB meetings held in 2010 lacked a representative of the Joint Staff. As a result of our review, Navy officials reported that they are revising their policy and procedures for CSBs to ensure the Joint Staff is invited to future CSB meetings.

Navy policy allows the service acquisition executive to delegate the chair to another official within the Navy’s acquisition office, which officials stated provides flexibility in scheduling CSBs. In practice, this resulted in meetings where required members of the CSB did not participate in discussions of requirements, configuration, or descoping. In 2010, the Navy service acquisition executive chaired and attended 12 of the 28 CSB meetings and participated in at least 2 others, both CSBs conducted via paper. According to our review of CSB documentation, six CSB meetings clearly discussed descoping options, and the service acquisition executive did not attend any of the five held in person. The sixth meeting was a paper CSB and it is unclear whether the service acquisition executive participated. When the Navy service acquisition executive or others chair the CSB meeting, the principal military deputy typically does not attend. In addition, at least three CSB meetings in 2010 did not include a representative from USD (AT&L).

The Navy also allows paper CSBs to fulfill the requirement for an annual CSB. In four cases, the Navy used paper CSBs to review requirement and configuration changes sometimes requiring millions of dollars or tens of millions of dollars in additional funding. According to Navy officials, Navy policy allows CSB members to reach decisions on issues of requirements and configuration by circulating briefing slides and memoranda rather than holding an actual meeting; however, there are not clear criteria specifying the circumstances under which a program may hold a paper CSB. Multiple Navy program managers stated that they do not understand which programs are eligible or when and how to request a paper CSB. In one case, a program manager stated that although the program was planning for and preferred a CSB meeting in person, Navy officials changed the format to a paper CSB a few days before the scheduled meeting time.

Two Major Defense Acquisition Programs Are Not Covered By The CSB Statute

Two major defense acquisition programs—the Ballistic Missile Defense System (BMDS) and the Chemical Demilitarization-Assembled Chemical Weapons Alternatives programs, which are managed by DOD components rather than military departments—are not subject to the CSB provisions in statute because the statute only applies to major defense acquisition programs overseen by the military departments. However, DOD acquisition policy, which requires CSBs for all major defense acquisition programs in development, applies to these programs.

The Missile Defense Agency (MDA), which is responsible for the management of BMDS, did not hold a CSB for the system in 2010; however, it did conduct reviews that discussed many of the same issues and included some of the same participants as those required for CSBs. The Program Change Board manages the development, fielding, and integration of BMDS through separate program elements and ensures the integrity of the system as a whole. This board, which is the primary forum for discussing and mitigating changes to program elements’ requirements and configuration, met 42 times in 2010. The Program Change Board is chaired by the equivalent of a service acquisition executive—the director of MDA—and, according to an MDA official, includes the equivalent of the comptroller, the program executive officer, and the program manager. MDA policy also requires USD (AT&L) to be invited to Program Change Boards, and allows for the military services’ participation when deemed appropriate, but does not include the Joint Staff. The Missile Defense Executive Board oversees implementation of strategic plans and reviews the priorities and budget for BMDS as a whole. The Missile Defense Executive Board includes the Joint Staff as well as the MDA director and an array of Office of the Secretary of Defense (OSD) and military service representatives, but according to DOD it does not generally discuss requirements and configuration at the element level. The executive board met seven times in 2010.

The Assistant Secretary of Defense for Nuclear, Chemical, and Biological Defense Programs, who is responsible for the management of the Chemical Demilitarization-Assembled Chemical Weapons Alternatives program, also did not hold a CSB in 2010. However, a similar board—the Chemical Demilitarization Program Strategic Governance Board—met three times in 2010 to discuss program progress, including how it is performing against its requirements and funding issues, including those related to significant cost and schedule growth. In 2010, the Assistant Secretary acted as the chair for this board which also includes representatives from the OSD comptroller, the Joint Staff, and the Army.

DOD Policy Is Not Consistent with the CSB Provisions in Statute

The CSB requirements in DOD’s primary acquisition instruction are not fully consistent with the provisions in statute. Most significantly, the instruction only requires CSB meetings for major defense acquisition programs in development, rather than major defense acquisition programs in development and production. Additionally, the instruction does not include the comptroller as a CSB member. According to USD (AT&L) officials, the CSB provisions in statute may not have been fully incorporated into USD (AT&L)’s December 2008 revision of DOD’s acquisition instruction because the statute was enacted in October 2008 and there was not enough time to reconcile them. USD (AT&L) is in the process of updating the instruction and is considering changes to the CSB requirements. USD (AT&L), according to officials, has also not consistently tracked whether programs are fulfilling the current requirements in DOD policy because the statute makes CSBs a military department responsibility.

CSB Meetings Had Some Positive Effects on Programs’ Efforts to Control Requirements and Costs

Individual programs varied in the extent to which they utilized CSBs to control requirements and mitigate cost and schedule risks. According to our survey results, the majority of CSB meetings neither reviewed requirement changes nor discussed options to reduce requirements or the scope of programs. We found a number of instances in which CSB meetings were effective in mitigating the effect of necessary changes, rejecting other changes, facilitating discussion of requirements, and endorsing descoping options with the potential to improve or preserve cost or schedule. Program managers, however, may be reluctant to recommend descoping options because of cultural biases about the role of a program manager, a preference not to elevate decisions to higher levels of review, and concerns that future funding will be cut. In an effort to increase descoping proposals, the Army and Air Force have issued additional descoping guidance and set savings or budget targets. The perceived effectiveness of the CSB meetings also varied based on the acquisition phase of a program and which CSB members participated. To further increase effectiveness and efficiency of CSBs, some of the military departments have taken steps to coordinate CSB meetings among programs that provide similar capabilities and align CSB meetings with other significant reviews.

Programs Have Had Some Success in Using CSBs to Control and Reduce Requirements

We identified individual examples from each military department in which CSB meetings were used to prevent or reject requirements or configuration changes, mitigate the cost and schedule effects of endorsed changes, facilitate the prioritization of requirements, and provide program managers with opportunities to reduce requirements or suggest other programmatic changes to lower costs and field systems faster. However, most of the program officials who held CSB meetings and responded to our survey reported that CSB meetings were not useful for preventing changes to requirements or configuration, mitigating the potential effects on cost and schedule when changes were endorsed, or recommending ways to improve a program’s cost and schedule by moderating requirements. In interviews with program officials, some explained that they did not utilize the CSB meetings to control requirements because they addressed requirement issues as they arose within the program rather than waiting for their program’s scheduled CSB meeting to occur. Others stated that their program was stable and that there were no requirement changes or descoping options to discuss. According to our survey results, reviews of CSB documentation, and interviews:  26 percent of the programs in our survey with CSB meetings reported that these meetings were useful forums to prevent changes to requirements. Moreover, 35 percent reported that the meetings were useful to make necessary changes to requirements. In an interview, several program officials stated that the mere suggestion of convening a CSB meeting to discuss a new requirement was enough to deter changes.  25 percent of the programs in our survey with CSB meetings reported that these meetings were useful forums to prevent changes to technical configuration. Conversely, 23 percent reported that the meetings were useful to make necessary changes to technical configurations. Our review of minutes and presentations also show at least one CSB meeting that rejected a change that had the potential to adversely affect program cost; the August 2010 CSB review for the LPD 17 amphibious ship program rejected a proposed configuration change that would have added new equipment to the ship at an estimated cost of $26 million.  Some CSB meetings also included discussions of how to prioritize requirements. For example, according to officials, the Air Force used a June 2010 CSB meeting for the Global Hawk—an unmanned surveillance aircraft—to prioritize joint urgent operational needs. According to program officials, the Global Hawk program has received numerous requests to add new capabilities to the platform due to its use in current operations. The program manager stated that the CSB meeting provided the opportunity to present the costs and benefits of those requests to decision makers and receive guidance from them on which ones to pursue or defer.  28 percent of the programs in our survey with CSB meetings reported that these meetings were useful forums to mitigate the potential cost and schedule effects of changes brought to the CSB for consideration. Moreover, 18 percent of programs reported CSB meetings were useful forums to mitigate the potential cost and schedule effects of changes made as a result of the CSB. The Vertical Take Off and Landing Tactical Unmanned Aerial Vehicle program used a CSB meeting to discuss ways to restructure the program in response to cost growth. At the meeting, the members of the CSB encouraged the program manager to go beyond his proposals and investigate changes to program quantities, contract strategy, and operational plans when restructuring the program, in order to reduce cost.  CSB meetings seem to have been effective in mitigating the cost and schedule effects of changes or only endorsing changes that would not affect costs and schedules. Of the 19 programs in our survey in which a CSB meeting endorsed changes to requirements or technical configuration, 1 reported an increase in program cost and 2 reported a delay in the delivery of an initial operational capability.  30 percent of programs in our survey with CSB meetings reported that these meetings were useful forums to offer options to lower costs and field systems faster. Survey results show that descoping options were presented for 19 programs and those options were endorsed for 8 of them. For example, at the December 2009 CSB meeting for the Air Force’s Joint Air-to-Surface Standoff Missile, the program office recommended adopting the extended range version’s lower reliability requirement for the baseline missile. The program office stated the existing baseline requirement, which was 5 percent higher, had the potential to become a cost driver in testing for the program. The CSB endorsed the program office’s recommendation.  Program officials also reported that the exercise of formulating descoping options, regardless of whether or not they were endorsed, helped their office identify and develop mitigation strategies in the event costs increased.

Table 3 provides examples of programs across the military departments that used CSB meetings to endorse requirement, configuration, or other programmatic changes to improve or preserve cost or schedule.

Program managers may be reluctant to recommend descoping options to moderate requirements during a CSB meeting because of cultural biases about the role of a program manager, a preference not to elevate decisions to higher levels of review, and concerns that future funding will be cut. According to several acquisition officials, there is a cultural bias throughout DOD that the role of the program manager is to meet the requirements handed to them, not to seek to reduce them to achieve cost savings. In this context, if a program manager recommends reducing requirements, it may suggest the person is not managing the program or serving the warfighter well. Still others preferred to reduce requirements that were within their span of control through their program’s internal change-management process rather than waiting for a CSB meeting to ask permission. For example, the DDG-51 program office proposed changes to the ships’ configuration to reduce cost by removing or relocating equipment and the CH-53K program avoided cost by relaxing a requirement for self-sealing fuel tanks. Our interviews with program officials also suggest that there may be a reluctance to present descoping options at a CSB meeting because it could be interpreted as an opportunity to reduce the program’s budget.

Army and Air Force Are Encouraging More Descoping Options to Reduce Costs

The Army and Air Force have both taken steps to encourage or require program managers to seek options to lower costs by reducing scope. Acquisition officials noted that the presentation of descoping options and the focus on reducing costs has increased in importance since CSBs were first established, as the budget environment has become more constrained. In a November 2010 memorandum, the Army emphasized the need for program officials to aggressively seek descoping opportunities with the goal of reducing per-unit or total program costs by 5 percent. Army officials stated that the memorandum was signed by senior leaders from the requirements, acquisition, and budgeting communities specifically to address the bias that reducing requirements is unacceptable.

According to officials, the Air Force amended its guidance for CSB meetings to require programs to present three to four descoping options along with the effect of those options on performance and program execution, the dollar amount already invested, and the estimated savings likely to result. Program managers are instructed to treat the descoping options as a budgeting exercise and to present the decisions that would need to be made if the program’s current budget were reduced by 10, 20, and 30 percent. Several program offices told us that forcing programs to present options to reduce requirements or scope led them to spend time preparing options that were not viable or that they would have to recommend against implementing.

CSBs’ Effects Differ for Programs in Development and Production

The types of discussions for which CSBs were useful changed based on whether programs were in development or production. According to our survey results, programs in development found CSB meetings to be more useful than programs in production for making necessary changes to requirements or technical configuration, mitigating the potential cost and schedule effects of changes, and recommending proposals to improve program costs and schedule. Table 4 presents our survey results of program officials’ opinions on the usefulness of CSB meetings.

Programs in development also proposed changes to requirements or configuration, presented options for reducing scope, and had those options endorsed at a higher rate than those in production. Even so, an official for one program in development stated that its CSB meeting was not effective because the program was meeting cost and schedule targets and its requirements were narrowly defined, which decreased opportunities for reducing scope.

According to our survey results, a higher percentage of programs in production reported that CSBs were useful in preventing changes compared to programs in development. We have previously reported that stabilizing a program’s requirements and design well before production is important because changes have increasingly negative effects on cost and schedule the further a program progresses. Program officials were wary about using CSB meetings to try to reduce costs for programs in production either through requirements changes or reductions in scope because the configuration should be locked, the available trade space is probably limited, and potential changes could be disruptive. For instance, the E-2D program reported in its April 2010 CSB meeting that its configuration was extremely stable and, with development and demonstration almost complete, reducing the scope of the programs could prove detrimental because it could lead to redesigns or decreases in capability. Changes at this stage of a program can still have a positive effect on cost if they do not require extensive design changes. For example, the program manager for the Family of Medium Tactical Vehicles—which is well into production with over 40,000 vehicles fielded—recommended removing the self-recovery winch from some vehicles, resulting in savings of $9,535 per vehicle.

Broad Senior Level Participation in CSB Meetings Facilitates Decision Making

CSBs provide a unique opportunity for program managers to address programmatic issues in front of a broad group of high-level decision makers that includes the acquisition, requirements, and funding communities. In some cases, the makeup of the CSB helped to accelerate the resolution of issues and facilitate decision making. For example, the Grey Eagle program utilized its CSB meeting to endorse an increase in the number of active units from 13 to 17. The program office reported that this decision, which otherwise may have taken years to approve and fund, was made and implemented quickly by the CSB because of the senior leadership present. Other program offices stated that the broad membership on CSBs, which includes key stakeholders and other interested parties, helps to create institutional buy-in for programmatic changes. CSB meetings also raised stakeholders’ awareness of cost increases. Specifically, CSB meetings provided the Joint Staff with its first knowledge of cost growth on at least four programs and triggered separate reviews by the Joint Requirements Oversight Council.

When critical stakeholders are absent, the decision-making ability of the CSB may be limited. In particular, some programs with users from across the military services and organizations external to DOD reported that the utility of CSBs was limited when those users were not represented. For example, the primary users of the Air Force’s Global Positioning System IIIA program include the Army, Navy, and other organizations external to DOD. The September 2010 CSB meeting for the system did not include these stakeholders, and program officials stated that as a result, the CSB was not empowered to make significant changes to the program.

Aligning CSB Meetings with Other Reviews May Increase Effectiveness and Efficiency

The decisions made at CSB meetings can affect complementary programs, as well as the funding required for programs. As a result, acquisition and program officials told us there is value in aligning CSB meetings so they are held together with reviews of similar programs or sequencing them to occur before key funding decisions are made. For example, in 2010, the Army grouped programs into capability portfolios, such as aviation or precision fires capabilities, and held one CSB meeting to discuss requirement changes and descoping options for all the programs. These CSB meetings generally occurred after the Army’s capability portfolio reviews—which revalidate, modify, or terminate requirements and ensure the proper allocation of funds across programs—and reviewed, endorsed, and implemented the recommendations coming from them. Holding CSB meetings for capability portfolios can facilitate discussions about interoperability and interdependency and promote an examination of requirements and capabilities across programs, including potential redundancies. Officials also stated that if two well-executed, high-performing programs within the same portfolio were reviewed independently, those discussions might not take place. For example, the Army’s Excalibur—a precision-guided munition—and Guided Multiple Launch Rocket System were both relatively stable programs in production. However, according to officials, during a capability portfolio review, the Army identified an overlap in the two programs’ capabilities and missions and recommended reducing the number of Excalibur munitions to be procured. At the subsequent April 2010 CSB meeting, the Army reviewed and implemented the proposal, which reduced the cost of the Excalibur program by $893.5 million. According to acquisition officials, grouping programs in this manner can also ease the difficulty of scheduling a large number of meetings that require senior leadership participation.

According to program officials, when CSB meetings were aligned with budget deliberations, it enabled an informed discussion of funding issues and rapid changes to program budgets. USD (AT&L)’s 2007 memorandum establishing CSBs stressed the importance of making necessary budget adjustments, especially those involving expected increases in program costs, at the earliest opportunity. In one example, the Army’s November 2009 CSB for the Patriot and Medium Extended Air Defense System programs corresponded with the service’s fiscal-year- 2011 budget-formulation process. Program officials stated that this helped facilitate the transfer of funds and efforts among the two programs, which had been endorsed by senior leaders from the acquisition and funding communities at the CSB. However, it may be functionally challenging to align CSB meetings with the budget formulation process in all cases, as CSB meetings in some cases must be event driven while the budget process is calendar driven.

Conclusions

With the prospect of slowly growing or flat defense budgets for years to come, DOD must get better returns on its weapon system investments than it has in the past. CSBs, which are intended to ensure that a program delivers as much planned capability as possible at or below the expected cost, can be a key tool in furthering this goal. They represent a unique forum that brings together a broad range of high-level decision makers from the acquisition, requirements, and funding communities, who can make and implement decisions quickly.

DOD’s experience with CSBs to date has already demonstrated their potential value—costly new requirements have been rejected, and options to moderate requirements and reduce program costs by millions of dollars have been endorsed. However, the efficiency and effectiveness of CSBs can still be improved. Ensuring key CSB members from the acquisition and requirements community are present at meetings could help build consensus more quickly and make decisions more efficiently. Similarly, while the law is silent on whether paper CSB meetings may be used to meet the annual requirement, holding in-person meetings may be more effective because a paper meeting may not provide the opportunity for in-depth discussion or proper oversight. Holding CSBs in conjunction with capability portfolio reviews and other similar meetings has the potential to expand opportunities to review and rationalize requirements across programs. Improving the connection between CSBs and the budget process and other reviews can help further efforts to match weapon system requirements with funding resources. Reviewing programs at CSBs on a case-by-case basis well into production would help decision makers identify cost savings and shift funding as warfighter needs and funding priorities change. Taken together, these steps have the potential to improve not only the efficiency and effectiveness of CSBs but also the affordability and execution of DOD’s major defense acquisition programs.

Recommendations for Executive Action

We recommend that the Secretary of Defense take the following seven actions directing: the Navy to amend its policy on CSBs to ensure that all statutorily required participants, particularly the Joint Staff, are included; the MDA to amend its policy to ensure that all statutorily required participants for military department CSBs are included in MDA’s Program Change Board, particularly the Joint Staff, if it is to serve as an equivalent review;  USD (AT&L) to amend its acquisition instruction to:  ensure that all statutorily required participants, in particular the comptroller, are included on CSBs; require CSB meetings for major defense acquisition programs in production as well as development but also coordinate with the military departments and the Congress to evaluate the effectiveness of CSB meetings for programs well into production; and  develop the means to better track CSBs and ensure compliance with the requirement that CSBs hold a meeting at least once each year;  USD (AT&L) to work with DOD components to determine whether paper CSBs are as effective as in-person meetings and, if not, amend the acquisition instruction accordingly; and  DOD components to amend their policies to encourage alignment between CSB meetings and other complementary reviews whenever possible.

Agency Comments and Our Evaluation

DOD provided us with written comments on a draft of this report. In its comments, DOD concurred or partially concurred with all seven of our recommendations and agreed to take action to address six of them. The comments are reprinted in appendix II. DOD also provided technical comments, which we addressed in the report, as appropriate.

In concurring with our recommendation that the Navy amend its policy on CSBs to include all statutorily required participants, DOD stated that the Navy has already issued two policy memorandums that do so. DOD also stated that the Navy will continue to issue policy guidance consistent with our recommendation. This will be particularly important as the Navy is currently in the process of revising its primary acquisition instruction.

DOD also concurred with our recommendations to amend its acquisition instruction to ensure that all statutorily required participants are included in CSBs and that meetings occur for programs in development as well as those in production. DOD did not address the portion of our recommendation to coordinate with the military departments and the Congress to evaluate the effectiveness of CSB meetings for programs well into production. Given our mixed findings on the utility of CSB meetings late in production, we continue to believe it would be in the interest of the department to study this issue.

DOD partially concurred with our recommendation that MDA amend its policy to ensure that all statutorily required participants for military department CSBs, in particular the Joint Staff, are included in MDA’s Program Change Board, if it is to serve as an equivalent review. In its comments, DOD stated that Joint Staff participation would provide little value because of the role of the Joint Staff in the acquisition of BMDS. In addition, DOD pointed out that the Joint Staff participates in the Missile Defense Executive Board, a forum in which strategic direction and funding priorities are established. However, we continue to believe that if the Program Change Board is to act as the forum for discussing configuration and requirements changes, it is important that the user communities, as represented by the Joint Staff, participate in these discussions.

DOD partially concurred with our recommendations on improving the tracking of CSB meetings, determining the effectiveness of paper CSBs, and aligning complimentary reviews with CSB meetings, when possible. In its comments, DOD stated that it would address these issues in “best practices” guidance to the military departments. With regard to developing the means to better track CSB meetings and compliance with the requirement to hold a meeting at least once each year, DOD stated the best practices guidance will direct the military departments to ensure adequate tracking vehicles are in place. We continue to believe that USD (AT&L) should play a role in tracking compliance and holding the military departments accountable, given our findings that the military departments did not hold CSBs for all the required programs.

We are sending copies of this report to the Secretary of Defense; the Secretaries of the Army, Navy, and Air Force; USD (AT&L); and the Director of the Office of Management and Budget. In addition, the report will be made available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me at (202) 512-4841. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff members making key contributions to this report are listed in appendix III.

Appendix I: Scope and Methodology

Selection and Classification of Major Defense Acquisition Programs

This report presents information on the Department of Defense’s (DOD) use of Configuration Steering Boards (CSB) for the major defense acquisition program portfolio in 2010. We used the Defense Acquisition Management Information Retrieval system to identify 98 active major defense acquisition programs. We defined an active program as one that issued a selected acquisition report in December 2009. This report presents information on all of these programs. One program, the Ballistic Missile Defense System, is managed by the Missile Defense Agency (MDA), which reports acquisition information on the system by functional elements. We reviewed nine elements and analyzed them separately from the rest of the major programs.

We categorized programs by the five acquisition organizations designated as having oversight—Army, Navy, Air Force, MDA, and the Assistant Secretary of Defense for Nuclear, Chemical, and Biological Defense programs—to assess trends in the use of CSBs. The selected acquisition report for each program designates the program’s acquisition organization. As the lead authority for joint programs rotates among the acquisition organizations as determined by the Office of the Secretary of Defense, we categorized all joint programs according to the service that was designated as the lead authority in the December 2009 selected acquisition report.

All of the programs in our audit fall into one of two phases: engineering manufacturing and development (referred to as development) or production and sustainment (referred to as production). Development generally begins with the initiation of an acquisition program as well as the start of engineering and manufacturing development and generally ends with entry into production. Production generally begins with the decision to enter low-rate initial production. For most programs in our assessment, the placement of programs in one of these two phases was determined by the dates of their Milestone B/II and Milestone C/III decisions. For instance, we categorized programs that have held a Milestone B/II decision but not a Milestone C/III as in the development phase and those that have held a Milestone C/III decision as in the production phase. The dates of milestone decisions for the programs used in the audit were determined through use of the Defense Acquisition Management Information Retrieval system.

Due to the nature of individual programs, select programs were not classified by milestone decision because they either have multiple increments that may begin production in advance of the notional Milestone C/III date, or the programs do not report milestone dates. In these cases, we used the program’s selected acquisition reports to determine the appropriate phase. The Navy often authorizes shipbuilding programs to begin production of the lead ship at Milestone B/II. We classified these programs as in the production phase. As the MDA programs develop systems’ capabilities incrementally instead of following the standard DOD acquisition model, we did not identify acquisition phases for Ballistic Missile Defense System elements.

Compliance with Statutory Requirements

To assess the extent that DOD has complied with the statutory requirements for CSB meetings in 2010, we compared CSB execution to provisions in the statute that call for annual CSB meetings and discussion of specific content. To determine the extent to which DOD complied with the requirement to hold an annual CSB for each program, we analyzed CSB records provided by the acquisition organization we reviewed and, using these records, calculated the number of CSBs held for each program in calendar-year 2010. To determine whether the components established boards that included the statutorily required participants, we analyzed policy and procedure documentation from each of the components as well as attendance lists of CSBs held in calendar-year 2010, provided by the acquisition organizations we reviewed. To identify issues discussed at CSBs and actions resulting from these CSBs, we reviewed CSB documents and questionnaire data and interviewed acquisition officials. We also reviewed and analyzed current and draft documentation related to department and service-level CSB policies, directives, guidance, and instructions to determine if they establish a structure that would facilitate compliance with the statute; examples of these documents include Department of Defense Instruction 5000.02, Department of the Army Pamphlet 70-3 regarding Army Acquisition Procedures, SECNAV Instruction 5000.2D, Air Force Instruction 63-101, and Missile Defense Agency Directive 5010.18 regarding Acquisition Management.

We also interviewed officials representing organizations that participate in CSBs or their equivalents including the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics, Joint Staff, military service and MDA offices, program offices, and capabilities and requirements offices to address department, military service, and MDA policies and execution.

Effectiveness of Configuration Steering Boards

To assess how effective CSBs have been controlling requirements and mitigating cost and schedule risks on programs, we analyzed CSB documentation to identify actions proposed and actions taken as a result of the CSB and their effect on cost, schedule, performance, and system configuration. We also asked program officials in our questionnaire to identify requirement changes or descoping options discussed at the CSB, the impact of decisions made, perceived effectiveness of the CSB, and explanations for not conducting a CSB, if applicable. To further analyze the effectiveness, challenges, and benefits of holding CSBs, we selected 17 programs for interviews. We based our selection on answers to our questionnaire, discussions with officials, and programmatic factors such as acquisition organization and phase. Specifically, we met with program officials at Wright Patterson Air Force Base, Ohio; Redstone Arsenal, Alabama; Washington Navy Yard in Washington DC; the Naval Air Station Patuxent River in Patuxent River, Maryland; and conducted video teleconferences with program officials at Picatinny Arsenal in New Jersey and at Los Angeles Air Force Base in El Segundo, California. We also interviewed acquisition officials, reviewed selected acquisition reports, and examined documentation related to service-level CSB policies, directives, guidance, and instructions to determine whether other reviews or acquisition processes influenced the effectiveness of CSBs.

DOD Major Defense Acquisition Programs Questionnaire

To collect information about DOD’s use of CSBs in fiscal year 2010, we developed and administered a Web-based questionnaire to the program offices of all 98 programs. Fiscal-year data was collected in our survey to be consistent with the Senate report language that contained our mandate. We administered separate questionnaires to nine Ballistic Missile Defense System elements and analyzed the results separately from the rest of the programs in our review. We fielded the survey from October 2010 to December 2010, and after extensive follow-up, we received responses from all 98 programs.

Our questionnaire of the 98 program offices, was not a sample questionnaire, so it has no sampling errors. However, the practical difficulties of conducting any questionnaire may introduce errors, commonly referred to as nonsampling errors. For example, difficulties in interpreting a particular question or limitations in the sources of information available to respondents can introduce unwanted variability into the questionnaire results. We took steps in developing the questionnaire, collecting the data, and analyzing the responses to minimize such nonsampling errors. For example, social science survey specialists designed the questionnaire in collaboration with GAO’s subject-matter experts. We conducted pretests with program managers to check that (1) the questions were clear and unambiguous, (2) terminology was used correctly, (3) the questionnaire did not place an undue burden on agency officials, (4) the information could feasibly be obtained, and (5) the questionnaire was comprehensive and unbiased. For the pretests, we selected programs from each military department and from various phases of the acquisition life cycle. We conducted four pretests. We made changes to the content and format of the questionnaire after each pretest, based on the feedback received. When we analyzed the data, an independent analyst checked all computer programs to reduce risk of error. Since this was a Web-based questionnaire, respondents entered their answers directly into the electronic questionnaire, eliminating the need to key data into a database, minimizing error.

We did not validate the data provided by the program offices, but reviewed the data and performed various checks to determine that the data were reliable enough for our purposes. Where we discovered discrepancies from reviewing responses and interviewing program offices, we clarified the data with the program office and made changes to the questionnaire data accordingly.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, Ronald E. Schwenn, Assistant Director; Noah B. Bleicher; MacKenzie Cooper; Morgan Delaney Ramaker; J. Kristopher Keener; Jean McSween; Kenneth E. Patton; and Brian Schwartz made key contributions to this report.