DOD Weapon Systems: Missed Trade-off Opportunities During Requirements Reviews
Released: 2011-06-16T00:00:00

Letter

Background

DOD uses three interrelated processes to deliver capabilities to the U.S. military: the Joint Capabilities Integration and Development System (JCIDS), which validates gaps in joint warfighting capabilities and requirements that resolve those gaps; the Defense Acquisition System, which develops and fields weapon systems to meet these requirements; and the Planning, Programming, Budgeting and Execution process, which allocates the funding needed to develop, acquire, and field these weapon systems. The JCIDS process is overseen by the JROC, which supports the Chairman of the Joint Chiefs of Staff in advising the Secretary of Defense on joint military capability needs. The JROC is chaired by the Vice Chairman of the Joint Chiefs of Staff, and includes one senior leader from each of the military services, such as the Vice Chief of Staff of the Army or the Vice Chief of Naval Operations.

The JROC has a number of statutory responsibilities related to the identification, validation, and prioritization of joint military requirements. The JROC assists the Chairman of the Joint Chiefs of Staff with a number of tasks, including (1) identifying, assessing, and approving joint military requirements; (2) establishing and assigning priority levels for joint military requirements; and (3) reviewing the estimated level of resources required to fulfill each requirement and ensuring that the resource level is consistent with the requirement’s priority. The JROC also assists acquisition officials in identifying alternatives to any acquisition programs that experience significant cost growth.

Since 2008, Congress has added to the JROC’s statutory responsibilities and increased the number of JROC members and advisors who provide input to it. The National Defense Authorization Act for Fiscal Year 2008 amended the U.S. Code to require that the Under Secretary of Defense for Acquisition, Technology and Logistics (USD AT&L), the Under Secretary of Defense (Comptroller), and the Director of the Office of Program Analysis and Evaluation serve as advisors to the JROC on matters within their authority and expertise. In 2009, WSARA expanded the role of the JROC by directing it to assist the Chairman of the Joint Chiefs of Staff in (1) ensuring that trade-offs among cost, schedule, and performance objectives are considered for joint military requirements; and (2) establishing an objective period of time within which an initial operational capability should be delivered. WSARA also stated that the newly constituted Director of Cost Assessment and Program Evaluation (CAPE) would advise the JROC. The Ike Skelton National Defense Authorization Act for Fiscal Year 2011 allowed the Vice Chairman of the Joint Chiefs of Staff to direct senior leaders from combatant commands to serve as members of the JROC when matters related to the area of responsibility or functions of that command are under consideration. It also added the Under Secretary of Defense for Policy, the Director of Operational Test and Evaluation, and other civilian officials designated by the Secretary of Defense as advisors to the JROC on issues within their authority and expertise.

The JROC is supported in the JCIDS process by two Joint Capabilities Boards (JCB) and seven Functional Capabilities Boards (FCB), each of which is chaired by a general/flag officer or civilian equivalent. JCBs and FCBs are responsible for specific Joint Capability Areas, such as Force Protection, Logistics, or Battlespace Awareness. The JCBs, FCBs, and associated FCB Working Groups review requirements documents prior to JROC reviews. The JCB also serves as the validation authority for requirements documents that are not associated with major defense acquisition programs (MDAP). In some instances, the JROC will not meet in person to approve requirements documents if there are no outstanding issues to discuss.

The JROC and its supporting organizations review requirements documents related to capability gaps and the MDAPs intended to fill those gaps prior to key acquisition milestones. These requirements documents— the Initial Capabilities Documents (ICD), Capability Development Documents (CDD), and Capability Production Documents (CPD)—are submitted by capability sponsors, which are generally the military services, but can also be other DOD agencies or combatant commands. Figure 1 depicts how JCIDS reviews align with the acquisition process.

The ICD is the first requirements document reviewed in JCIDS. It is intended to identify a specific capability gap, or set of gaps, in joint military capabilities that are determined to require a materiel solution as a result of a capabilities-based assessment. DOD policy requires that the JROC validate the ICD prior to a Materiel Development Decision, which is the formal entry point into the acquisition process. The ICD does not contain specific cost, schedule, or performance objectives. Once the JROC validates an ICD, the Milestone Decision Authority, working with appropriate stakeholders, shall determine whether to proceed to a Materiel Development Decision. After the Materiel Development Decision, the capability sponsor initiates an AOA to consider alternative solutions to fulfilling the capability need described in an ICD, and possible trade-offs among cost, schedule, and performance for each alternative are considered.

The CDD is the second requirements document reviewed in JCIDS. It can address capability gaps presented in one or more ICDs. The CDD is intended to define a proposed program’s Key Performance Parameters (KPP), Key System Attributes (KSA), and other performance attributes. KPPs are the system characteristics that the CDD sponsor considers critical to delivering that military capability, while KSAs are system attributes the CDD sponsor considers essential for an effective military capability, but a lower priority than the KPPs. DOD policy calls for the JROC to validate the CDD to inform the Milestone B decision, which marks the official start of an acquisition program and entry into the engineering and manufacturing development phase. The CDD is the first requirements document that contains cost, schedule, and performance objectives.

The CPD is the third and final requirements document reviewed in J It is intended to refine the KPPs, KSAs, and performance attributes validated in the CDD. DOD policy calls for the JROC to validate the CP inform the Milestone C decision, which marks a program’s entry into production. Appendix II identifies the IC the JCB or JROC in fiscal year 2010.

CIDS.

JROC Did Not Always Consider Trade-offs or Influence Trade-off Decisions

The JROC considered trade-offs made by the military services before validating requirements for four of the seven proposed programs it reviewed in fiscal year 2010, and provided input to the military services on the cost, schedule, and performance objectives for two of the seven programs. The JROC’s requirements review was the final step in a long requirements vetting process, with most trade-offs being made by the military services earlier in the process. Key stakeholders from the offices of the Under Secretary of Defense (Comptroller), USD AT&L, Director of CAPE, and the combatant commands were all satisfied with their opportunities to provide input to the JROC; but they provided limited input on trade-offs among cost, schedule, and performance objectives, and used other means to influence trade-offs. Perhaps most importantly, none of the JROC’s requirements reviews align with the AOA, which is where the military services reported making the most significant trade-offs. As a result, a program can spend significant time in technology development before the JROC gets to formally weigh in on these trade-offs through the JCIDS process. The JROC also reviews MDAP requirements after a program enters development and experiences substantial cost growth. DOD and the JROC stated that requirements were not the primary causes of cost growth for the 15 programs reviewed for this purpose in fiscal year 2010 and the JROC did not change any KPPs to mitigate the reported cost growth.

JROC Did Not Always Consider Trade-offs when Validating Requirements for Proposed Programs

The JROC considered trade-offs made by the military services before validating requirements for four of the seven proposed programs it reviewed in fiscal year 2010. On three programs, the JROC did not receive information on the potential cost and schedule implications of each of the alternatives considered. Table 1 summarizes the JROC’s consideration of cost, schedule, and performance objectives for the seven proposed MDAPs it reviewed in fiscal year 2010.

The JROC’s review of the CDD for a proposed program is the final step in a long requirements vetting process, and DOD officials reported that trade- offs typically occur earlier in the process. Each military service conducts its own internal requirements reviews for its proposed programs, which are used to refine requirements documents before they are submitted into JCIDS. Military service officials reported that they make significant trade- offs during these internal reviews, and that KPPs and technical requirements rarely change after requirements documents are submitted into JCIDS because extensive analysis has already been conducted. For the seven proposed MDAPs we reviewed, the military services generally submitted requirements to the JROC that would be fully funded, provide initial capability within 6 years, utilize critical technologies that were nearing maturity, and be acquired using an incremental approach. These characteristics are consistent with provisions in the Weapon Systems Acquisition Reform Act (WSARA) related to how the requirements process should be structured and aspects of GAO’s best practices for weapon system acquisitions.

Two of the proposed program requirements presented to the JROC included major trade-offs among cost, schedule, and performance objectives and revisions to their acquisition approaches that had been made after predecessor programs were cancelled over affordability concerns. The Air Force initiated the HH-60 Recapitalization program after the Combat Search and Rescue Replacement Vehicle (CSAR-X) program was cancelled, and the HH-60 Recapitalization program is expected to decrease cost by changing cabin space, velocity, and range from the CSAR- X requirements. In 2007, the Army, with input from a Functional Capabilities Board, decided to use an incremental acquisition approach for the Ground Soldier System in order to reduce costs, meet schedule demands, and avoid some of the mistakes made during the Land Warrior program, which was cancelled because of funding and cost issues.

Key Stakeholders Provided Limited Input into JCIDS, but Use Other Means to Influence Trade-offs

The JROC received limited input on trade-offs among cost, schedule, and performance objectives from key stakeholders when validating requirements for the seven proposed MDAPs we reviewed from fiscal year 2010. Both WSARA and the National Defense Authorization Act for Fiscal Year 2008 directed the JROC to consult with the Under Secretary of Defense (Comptroller), the USD AT&L, and the Director of CAPE. Additionally, WSARA instructed the JROC to consult with the combatant commands. Officials from these organizations reported that they had ample opportunity to participate in JROC requirements reviews, and Joint Staff officials said efforts to involve these stakeholders preceded WSARA. However, officials from the offices of the Under Secretary of Defense (Comptroller), USD AT&L, and the Director of CAPE also reported that the acquisition and budgeting/funding processes are the primary mechanisms through which they influence programs, rather than JCIDS. For example, CAPE oversees AOAs for MDAPs and has an opportunity to provide input and guidance on AOA considerations. Further, the combatant commands reported that they most often submit prioritized lists of capability gaps directly to the Chairman of the Joint Chiefs of Staff as part of the resource allocation process, which is separate from JCIDS.

Nonetheless, joint stakeholders did provide some significant input during the JROC’s reviews of the seven proposed programs in fiscal year 2010. For example, the Army more fully defined a Ground Soldier System, Increment 1 KPP in response to input from DOD’s Joint Interoperability Test Command, and in another instance, the Army added a KSA to the AIAMD SOS, Increment 2 CDD due to input from the office of the USD AT&L, the Defense Information Systems Agency, and the Joint Staff. Neither of these changes involved trade-offs among cost, schedule, and performance objectives.

JCIDS Reviews Are Not Aligned with the Most Significant Trade-off Decisions

The JROC does not formally review the trade-off decisions made as a result of an AOA until a proposed program’s CDD enters the JCIDS process. According to DOD officials, the most significant trade-offs are made by the military services between ICD and CDD reviews during the AOA, which is intended to compare the operational effectiveness, cost, and risks of a number of alternative potential solutions. For example, during the CVLSP AOA, the Air Force decided to decrease troop transport capacity in order to reduce cost. Alternatively, during the AIAMD SOS AOA, the Army decided to pursue the most costly option reviewed because it provided greater capability. A significant amount of time and resources can be expended before the JROC gets to weigh in on these trade-offs during CDD reviews. For example, the JROC reviewed the AOA summary for JPALS, Increment 2, 4 years after the conclusion of the AOA. During the time between the AOA and the CDD review, the technology intended to enable the chosen alternative is developed. Figure 2 shows the AOA’s relationship to both the requirements and acquisition processes.

Joint Staff officials have stated that establishing a JROC review of the AOA would allow it to provide military advice on trade-offs and the proposed materiel solution before Milestone A, and an ongoing Joint Staff review of JCIDS is considering an increased role for the JROC at this point. According to the Joint Staff, increased JROC engagement at these early stages of the acquisition process is warranted to align it with other elements of recent acquisition reforms. For example, WSARA emphasized that the AOA should fully consider possible trade-offs among cost, schedule, and performance objectives for each alternative considered, and in September 2010, USD AT&L issued a memorandum that emphasized the need for trade-offs from a program’s inception. The memorandum also dictated that affordability targets shall be established at the conclusion of the AOA and that these targets will be treated like KPPs, even though they will be set and managed by the acquisition, not requirements, community.

JROC Did Not Change KPPs when Programs Incurred Substantial Cost Growth

The JROC did not change any KPPs during 15 reviews of programs that reported substantial cost growth in fiscal year 2010. According to the Joint Staff, by holding requirements firm and accepting increased cost and schedule delays, the JROC essentially traded cost and possibly schedule for performance. In fiscal year 2010, the JROC reviewed six programs after they experienced a critical Nunn-McCurdy breach and nine programs as part of the tripwire process. During all 15 reviews, DOD and the JROC stated that requirements were not the primary causes of cost growth. For all six programs that experienced a critical Nunn-McCurdy cost breach, the JROC validated the system’s capabilities as being essential to national security and did not make any changes to their KPPs. For all nine programs that were approaching Nunn-McCurdy thresholds, the JROC did not identify opportunities to mitigate cost growth by modifying requirements. Most of these programs were in production in fiscal year 2010, and changing requirements at this late stage might not have mitigated the reported cost growth. When the JROC reviewed the Family of Advanced Beyond Line-of-Sight Terminals program, which was still in development, it concluded that the program’s requirements could not be met in an affordable manner. The JROC did not immediately defer any of the program’s requirements, but instead requested that USD AT&L identify potential alternatives for the program, including reviewing whether adjustments to performance requirements would be appropriate.

Military Services Did Not Consistently Provide High-Quality Resource Estimates to JROC

The military services did not consistently provide high-quality resource estimates to the JROC to support its review of requirements for 7 proposed programs in fiscal year 2010. We found the estimates presented to the JROC were often unreliable when assessed against best practices criteria. The type of resource estimates the military services presented to the JROC varied from ones that had been validated by the military services’ cost analysis agencies to less rigorous rough-orders-of-magnitude estimates. In most cases, the military services had not effectively conducted uncertainty and sensitivity analyses, which establish confidence levels for resource estimates, based on the knowledge available, and examine the effects of changing assumptions and ground rules. Lacking risk and uncertainty analysis, the JROC cannot evaluate the range of resources that might be necessary to cover increased costs resulting from unexpected design complexity, technology uncertainty, and other issues. The lack of this information affects the JROC’s efforts to ensure that programs are fully funded and its ability to consider the resource implications of cost, schedule, and performance trade-offs.

JROC Received Resource Estimates That Did Not Meet Best Practices

The JROC first receives resource estimates for proposed programs when it reviews CDDs, and when we reviewed the CDD resource estimates presented to the JROC in fiscal year 2010, we found that they were generally unreliable when assessed against our best practices criteria. While most of the resource estimates substantially met our criteria for a comprehensive resource estimate, they generally were not very accurate, credible, or well-documented. Appendix IV includes a list of the best practices against which we assessed these resource estimates.

The type of resource estimates the military services presented to the JROC varied from ones that had been validated by the military services’ cost analysis agencies to less rigorous rough-orders-of-magnitude estimates. According to Joint Staff officials, military services can initiate CDD reviews at any point in the acquisition process prior to program start, even if good resource estimates are not available. For example, the JROC validated the P-8A, Increment 3 CDD more than 2 years before the program was expected to start, before an AOA had been completed, and with a rough-order-of-magnitude estimate. Joint Staff officials reported that they depend on CAPE to review the quality of resource estimates during the JCIDS process, but CAPE cost assessment officials told us that they rarely participate in JCIDS reviews.

Regardless of the type of resource estimate, uncertainty and sensitivity analysis can establish confidence levels for resource estimates, based on the knowledge available at the time, and examine the effects of changing assumptions and ground rules, including those related to trade-offs among cost, schedule, and performance objectives. The military services sponsoring the requirements generally did not effectively meet best practices for uncertainty and sensitivity analyses using the knowledge they had available to them for any of the seven resource estimates we reviewed. Figure 3 summarizes our assessment of the resource estimates presented to the JROC against our best practices criteria.

Five of the seven CDD resource estimates substantially met our criteria for a comprehensive resource estimate. The resource estimates generally completely defined their respective programs, and included most, if not all, life-cycle costs. The Ship to Shore Connector, CVLSP, and JPALS, Increment 2 resource estimates also effectively documented all cost- influencing ground rules and assumptions, although the other resource estimates did not. Additionally, only the Ship to Shore Connector’s work breakdown structure effectively met our criteria, which require that work breakdown structures are product-oriented and at an appropriate level of detail. If a resource estimate does not specifically break out common costs, such as government-furnished equipment costs, or does not include an associated work breakdown structure dictionary, cost estimators cannot ensure that the estimate includes all relevant costs.

The HH-60 Recapitalization and P-8A, Increment 3 resource estimates did not effectively meet any of our best practices for a comprehensive resource estimate. Unless resource estimates account for all costs, they cannot enhance decision making by allowing for design trade-off studies to be evaluated on a total cost, technical, and performance basis. Additionally, unless ground rules and assumptions are clearly documented, the resource estimate will not have a basis for resolving areas of potential risk.

Most Resource Estimates Did Not Substantially Meet Accuracy Criteria

Only two of the seven CDD resource estimates substantially met our criteria for an accurate resource estimate, while three partially met the criteria, and two did not meet or minimally met the criteria. We found that the Ship to Shore Connector, CVLSP, AIAMD SOS, Increment 2, and the Ground Soldier System, Increment 1 resource estimates contained few, if any, minor mistakes, and that the Ship to Shore Connector, CVLSP, and JPALS, Increment 2 resource estimates were appropriately adjusted for inflation. Additionally, we found that the Ship to Shore Connector and JPALS, Increment 2 resource estimates were based on historical records of actual experiences from other comparable programs.

However, we generally found that the resource estimates were not consistent with our best practices. Accurate resource estimates are rooted in historical data, which provide cost estimators with insight into actual costs of similar programs, and can be used to challenge optimistic assumptions and bring more realism to a resource estimate. Unless an estimate is based on an assessment of the most likely costs, and reflects the degree of uncertainty given all of the risks considered, management will not be able to make well-informed decisions.

Most Resource Estimates Were Not Credible

Four of the seven CDD resource estimates did not meet or minimally met our criteria for a credible resource estimate, and only the Ship to Shore Connector resource estimate substantially met the criteria. The Ship to Shore Connector and AIAMD SOS, Increment 2 resource estimates included sensitivity analyses that identified a range of possible costs based on varying assumptions, parameters, and data inputs, but none of the other resource estimates included this analysis. As a best practice, sensitivity analysis should be included in all resource estimates because it examines the effects of changing assumptions and ground rules. Since uncertainty cannot be avoided, it is necessary to identify the cost elements that represent the most risk and, if possible, cost estimators should quantify that risk. When an agency fails to conduct sensitivity analysis to identify the effect of uncertainties associated with different assumptions, this increases the chance that decisions will be made without a clear understanding of the impact on cost.

Additionally, only the Ship to Shore Connector resource estimate effectively met our best practices for risk and uncertainty analysis. For management to make good decisions, the program estimate must reflect the degree of uncertainty so that a level of confidence can be given about the estimate. An estimate without risk and uncertainty analysis is unrealistic because it does not assess the variability in the resource estimate from effects such as schedules slipping, missions changing, and proposed solutions not meeting users’ needs. Lacking risk and uncertainty analysis, management cannot determine a defensible level of contingency reserves that is necessary to cover increased costs resulting from unexpected design complexity, technology uncertainty, and other issues.

Further, none of the planned programs effectively met our criteria for an independent cost estimate when they were reviewed by the JROC. An independent cost estimate is considered one of the best and most reliable resource estimate validation methods because it provides an independent view of expected program costs that tests the program office and service estimates for reasonableness. Without an independent cost estimate, decision makers lack insight into a program’s potential costs because these estimates frequently use different methods and are less burdened with organizational bias. Moreover, independent cost estimates tend to incorporate adequate risk, and therefore tend to be more conservative by forecasting higher costs than the program office. A program estimate that has not been reconciled with an independent cost estimate has an increased risk of proceeding underfunded because an independent cost estimate provides an objective and unbiased assessment of whether the program estimate can be achieved. Alternatively, programs can reinforce the credibility of their resource estimates through cross-checking, which determines whether alternative cost estimating methods produce similar results. However, only the Ship to Shore Connector resource estimate effectively met our best practices for cross-checking.

Most Resource Estimates Were Not Well-documented

Only the JPALS, Increment 2 resource estimate substantially met our criteria for a well-documented resource estimate, while four of the seven CDD resource estimates partially met our criteria, and two of the resource estimates did not meet or minimally met the criteria. The JPALS, Increment 2 and CVLSP resource estimates sufficiently described the calculations performed and estimating methodologies used to derive each program element’s cost. Additionally, the JPALS, Increment 2, Ship to Shore Connector, and AIAMD SOS, Increment 2 documentation clearly discusses the technical baseline description, and the data in the technical baseline are consistent with the resource estimate. However, none of the documents effectively described how the resource estimates were developed in a manner that a cost analyst unfamiliar with the program could understand what was done and replicate it.

We generally found that the resource estimates were not consistent with our best practices for a well-documented resource estimate. Documentation is essential for validating and defending a resource estimate. Without a well-documented resource estimate, a convincing argument of an estimate’s validity cannot be presented, and decision makers’ questions cannot be effectively answered. Poorly documented resource estimates cannot explain the rationale of the methodology or the calculations underlying the cost elements. Further, a well-documented resource estimate is essential for an effective independent review to ensure that the resource estimate is valid and credible. Unless the estimate is fully documented, it will not support reconciliation with an independent cost estimate, hindering understanding of cost elements and their differences.

Military Services Generally Presented Resource Estimates That Were Fully Funded to JROC

The JROC required the military services to show that the proposed programs were fully funded to the resource estimates presented by the military services before it validated requirements for five of the seven proposed MDAPs we reviewed from fiscal year 2010; the two other proposed MDAPs were funded at more than 97 and 99 percent respectively. However, we found that these resource estimates were generally unreliable, which undermined the JROC’s efforts. In 2007, the JROC issued guidance instructing the military services to commit to funding the requirements that the JROC validates. The guidance emphasized the need for full funding in an effort to facilitate sound fiscal and risk decisions. However, the JROC does not explicitly consider a requirement’s affordability in a broader context during JCIDS reviews. DOD funding plans are captured in the future-years defense program, which presents resource information for the current year and the following 4 years. The future-years defense program is updated twice per year to reflect the military services’ input and the budget the President submits to Congress. Statute and DOD acquisition policy also require programs to be fully funded through the period covered by the future- years defense program.

One of the seven proposed MDAPs we reviewed from fiscal year 2010 included a funding shortfall when its requirements were being reviewed through JCIDS, but its CDD was not approved until the shortfall had been addressed. Specifically, when the JCB reviewed the CVLSP CDD, the funding plan included a $1.3 billion shortfall through fiscal year 2015. The JCB chairman directed the Air Force to modify the program’s funding plan before proceeding to the JROC review. When the Air Force briefed the JROC on the CVLSP CDD approximately 8 months later, it presented a funding plan that fully funded the program through the future-years defense program time frame. The revised funding plan also included more money for the program beyond the future-years defense program time frame, and the total program cost increased from $14.2 billion to $15.2 billion.

Despite JROC efforts to ensure programs are fully funded, the military services retain primary control over their budgets, and ultimately, JROC decisions are influential but not binding. When the JCB reviewed the JPALS, Increment 2 CDD, it requested clarification on the Air Force’s funding plan, and emphasized the need for full funding prior to program start. The funding plan presented to the JCB included a $77.7 million shortfall through fiscal year 2015, and the Air Force had cut JPALS funding in the past. Following the JCB review, the JROC issued a decision memorandum that documented the Air Force’s commitment to fully funding JPALS, Increment 2. However, in fiscal years 2011 and 2012, the Air Force only funded approximately 30 percent of the resource estimate presented to the JCB.

JROC Did Not Consistently Prioritize Requirements and Capability Gaps

The JROC does not currently prioritize requirements, consider redundancies across proposed programs, or prioritize and analyze capability gaps in a consistent manner. As a result, the Joint Staff is missing an opportunity to improve military service and departmentwide portfolio management efforts. A portfolio management approach to weapon system investments would involve taking a disciplined, integrated approach to prioritizing needs and allocating resources in order to eliminate redundancies, gain efficiencies, and achieve a balanced mix of executable programs. According to Army, Air Force, and Navy officials, having a better understanding of warfighter priorities from the JROC would be useful to inform both portfolio management efforts and service budgets. A DOD review team examining the JCIDS process is considering changes that would address the prioritization of requirements. During its review of the capability gaps presented in 12 ICDs in fiscal year 2010, the JROC did receive some information on priorities and potential redundancies; however, the sponsors presented this information in an inconsistent manner, making it difficult for the JROC to assess the relative priority of capability gaps across different ICDs.

JROC Does Not Prioritize Requirements or Consider Redundancies Across Proposed Programs

Under the current JCIDS process, the JROC does not prioritize requirements or consider redundancies across proposed programs during CDD reviews. In the National Defense Authorization Act for Fiscal Year 2008, Congress amended the U.S. Code to direct the JROC to help assign priority levels for joint military requirements and ensure that resource levels associated with those requirements are consistent with the level of priority. The House Armed Services Committee report accompanying the authorization act stated that clear JROC priorities and budget guidance would allow for joint decision making, as opposed to service-centric budget considerations. In addition, we have previously recommended that DOD develop an analytic approach within JCIDS to better prioritize and balance the capability needs of the military services, combatant commands, and other defense components. According to the Joint Staff and military service officials, prioritization across programs still primarily occurs through the Planning, Programming, Budgeting and Execution process, which is the responsibility of the military services and the Office of the Under Secretary of Defense (Comptroller).

The JCIDS manual does not currently require an analysis of potential redundancies during CDD reviews. In our recently issued report on government duplication, we noted that service-driven requirements and funding processes continue to hinder integration and efficiency and contribute to unnecessary duplication in addressing warfighter needs. We have also previously reported that ineffective collaboration precluded opportunities for commonality in unmanned aircraft systems. In fiscal year 2010, the JROC met to consider joint efficiencies between two such systems: the Navy’s Broad Area Maritime Surveillance system and the Air Force’s Global Hawk system. The JROC requested that the Navy and Air Force ensure that a common component was interoperable between the two systems, and that the Air Force consider an all-weather capability developed by the Navy. The JROC has also supported joint development efforts for these programs and requested annual status updates. According to Broad Area Maritime Surveillance program officials, the Air Force and Navy programs are investigating commonality opportunities, including sense-and-avoid capabilities, a consolidated maintenance hub, and basing options for both systems. The JROC did not meet to consider any other joint efficiencies across military services in fiscal year 2010.

The Joint Staff has acknowledged that the JROC should play a larger role in prioritizing needs and addressing redundancies. In July 2010, the Vice Chairman of the Joint Chiefs of Staff initiated a review of the JCIDS process. One of the goals of the review team was to develop metrics and criteria to ensure the JCIDS process has the ability to rank or prioritize needs. The review team’s charter states that these metrics must enable more structured reviews of portfolio gaps and redundancies. According to the Joint Staff, the review team is considering a number of recommendations including asking the JROC to prioritize requirements based on the urgency and significance of the need. This list of priorities could be used to inform military service budgets. Joint Staff officials have also stated that redundancies may be addressed more directly in the future as part of an enhanced portfolio management effort.

Lack of JROC Prioritization Results in Missed Opportunities to Manage Portfolios Better

We have previously reported that DOD has not taken a portfolio management approach to weapon system investments, which would involve taking a disciplined, integrated approach to prioritizing needs and allocating resources in order to eliminate redundancies, gain efficiencies, and achieve a balanced mix of executable programs. In September 2010, USD AT&L issued guidance intended to increase efficiencies and eliminate redundancies, and it presented the Army’s portfolio management activities as an example to emulate. The Army uses capability portfolio reviews of capability gaps and proposed and existing programs to revalidate, modify, or terminate requirements and ensure the proper allocation of funds between them. The Army has established 17 portfolios, including aviation, air and missile defense, and combat vehicle modernization. An Army official involved in the portfolio reviews said that he has requested on several occasions for the Joint Staff to prioritize warfighter needs; however, the JROC has not done so. Instead, the Army relies on its own prioritization information during the portfolio reviews to help determine the capability areas where it is willing to assume risk. Air Force and Navy officials have also stated that they could benefit from JROC prioritization of requirements, and that this information would be useful in order to better allocate resources during their budget formulation activities.

Sponsors Do Not Prioritize Capability Gaps or Analyze Potential Redundancies in a Consistent Manner

The JROC has required that capability sponsors prioritize capability gaps and identify redundancies when developing ICDs, and capability sponsors generally complied with these requirements in the 12 validated ICDs we reviewed from fiscal year 2010. However, the sponsors presented this information in an inconsistent manner, making it difficult for the JROC and the military services to assess priorities and redundancies across ICDs or use this information to inform resource allocation decisions. For example, the Electronic Health Record ICD prioritized its gaps in numerical order from 1 to 10, but the Command and Control On-The-Move ICD labeled half its gaps medium priority and the other half high priority. The JCIDS operation manual provides limited guidance on how capability sponsors should prioritize the gaps, stating only that the prioritization should be based on the potential for operational risk associated with the shortfalls. The JCIDS manual also directs capability sponsors to identify redundancies and assess whether the overlap is operationally acceptable or whether it should be evaluated as part of the trade-offs to satisfy capability gaps. Three of the 12 validated ICDs we reviewed from fiscal year 2010 did not address redundancies. Furthermore, only one of these ICDs presented to the JROC in fiscal year 2010 included an evaluation of the overlaps. The JROC did not address these omissions when it validated the documents.

Conclusions

In the last several years, Congress has passed legislation to give the JROC a greater role in prioritizing military requirements and shaping sound acquisition programs by encouraging cost, schedule, and performance trade-offs. Taken together, these steps have the potential to improve the affordability and execution of DOD’s portfolio of major defense acquisition programs. However, the JROC has largely left prioritization and trade-off decisions to the military services, despite having a unique, joint perspective, which would allow it to look across the entire department to identify efficiencies and potential redundancies. To more effectively leverage its unique perspective, the JROC would have to change the way it views its role, more regularly engage the acquisition community in trade-off discussions at early acquisition milestones, and more effectively scrutinize the quality of the resource estimates presented by the military services. Until it does so, the JROC will only be a marginal player in DOD’s efforts to align the department’s available resources with its warfighting requirements.

Recommendations for Executive Action

To enhance the JROC’s role in DOD-wide efforts to deliver better value to the taxpayer and warfighter, we recommend that the Vice Chairman of the Joint Chiefs of Staff, as chairman of the JROC, take the following five actions: Establish a mechanism to review the final AOA report prior to Milestone A to ensure that trade-offs have been considered and to provide military advice on these trade-offs and the proposed materiel solution to the Milestone Decision Authority.

Require that capability sponsors present resource estimates that have been reviewed by a military service’s cost analysis organization to ensure best practices are being followed.

Require that capability sponsors present key results from sensitivity and uncertainty analyses, including the confidence levels associated with resource estimates, based on the program’s current level of knowledge.

Assign priority levels to the CDDs based on joint force capability gaps and redundancies against current and anticipated threats, and provide these prioritization levels to the Under Secretary of Defense (Comptroller) and the military services to be used for resource allocation purposes.

Modify the JCIDS operations manual to require that CDDs discuss potential redundancies across proposed and existing programs, and address these redundancies when validating requirements.

Agency Comments and Our Evaluation

The Joint Staff provided us written comments on a draft of this report. The comments are reprinted in appendix V. The Joint Staff also provided technical comments, which we addressed in the report, as appropriate.

In its comments, the Joint Staff partially concurred with all five of our recommendations, generally agreeing that there is a need to take action to address the issues we raised, but differing in terms of the specific actions that should be taken.

The Joint Staff partially concurred with our recommendation that the Vice Chairman of the Joint Chiefs of Staff, as chairman of the JROC, establish a mechanism to review the final AOA report prior to Milestone A to ensure that trade-offs have been considered and to provide military advice on these trade-offs and the proposed materiel solution to the Milestone Decision Authority. The Joint Staff noted that its ongoing review of JCIDS will include a recommendation that AOA results be briefed to FCBs. However, the FCB will only elevate these briefings to the JCB or JROC on an exception basis. The Joint Staff explained that this approach would allow the JROC to provide more informed advice to a Milestone Decision Authority without adding another round of staffing, an additional JCIDS document, or an official validation of AOA results. We agree that the Joint Staff should seek to implement this recommendation in the most efficient and effective way possible; however, given our finding that the most significant trade-off decisions are made as a result of an AOA, we continue to believe that the results should be reviewed by the JROC.

The Joint Staff partially concurred with our recommendation that the Vice Chairman of the Joint Chiefs of Staff require that capability sponsors present resource estimates that have been reviewed by a military service’s cost analysis organization to ensure best practices are being followed. The Joint Staff stated that program office cost estimates are compared to independent cost estimates during CDD reviews. However, none of the seven CDD cost estimates we reviewed effectively met our criteria for an independent cost estimate. As a result, we believe that the Joint Staff needs to take additional action to ensure that resource estimates presented by capability sponsors have been reviewed by a military service’s cost analysis organization. The Joint Staff also stated that its ongoing review of JCIDS will examine how to highlight this area during CDD reviews.

The Joint Staff partially concurred with our recommendation that the Vice Chairman of the Joint Chiefs of Staff require that capability sponsors present key results from sensitivity and uncertainty analyses, including the confidence levels associated with resource estimates, based on the program’s current level of knowledge. The Joint Staff stated that our recommendation needs further study to understand the expected outcomes and the required authorities for the JROC, and its ongoing review of JCIDS will examine how to highlight this area. We believe that the JROC cannot fully consider trade-offs or the affordability of a proposed program unless it receives information on the risk and uncertainty associated with resource estimates; it does not need additional authority to require capability sponsors to present the results of this type of analysis before it approves proposed requirements. The Joint Staff also noted that the Director, CAPE, has cost analysis responsibilities for resource estimates. CAPE cost assessment officials reported that they rarely participated in JCIDS reviews. As a result, the JROC may have to be more proactive in reaching out to CAPE to help it understand the risk and uncertainty associated with the resource estimates it receives.

The Joint Staff partially concurred with our recommendation that the Vice Chairman of the Joint Chiefs of Staff assign priority levels to CDDs based on joint force capability gaps and redundancies against current and anticipated threats, and provide these prioritization levels to the Under Secretary of Defense (Comptroller) and the military services to be used for resource allocation purposes. The Joint Staff agreed that the identification of joint priorities could enhance a number of processes, including program and budget reviews. It noted that its ongoing review of JCIDS will recommend a prioritization framework through which CDDs will inherit priority levels based on the requirements and capability gaps identified in ICDs or Joint Urgent Operational Needs. However, the Joint Staff argued against prioritizing based on CDDs directly because it would provide less flexibility. We believe that the proposed approach could be effective if the Joint Staff addresses the inconsistencies we found in the way ICDs prioritize gaps. In addition, we continue to believe that the prioritization framework should facilitate an examination of priorities across CDDs.

The Joint Staff partially concurred with our recommendation that the Vice Chairman of the Joint Chiefs of Staff modify the JCIDS operations manual to require that CDDs discuss potential redundancies across proposed and existing programs, and address these redundancies when validating requirements. The Joint Staff stated that its ongoing review of JCIDS will address this issue by establishing unique requirements as a higher priority than unnecessarily redundant requirements, and by establishing a post- AOA review, which could also be used to identify unnecessary redundancies. The Joint Staff did not address whether it would update the JCIDS operations manual as recommended and stated that reviewing assessments of redundancies in CDDs would be late in the JCIDS process. We believe that potential redundancies should be discussed at multiple points, including during CDD reviews, because we found that several years can pass between the conclusion of an AOA and this review. During that time, new redundancy issues could emerge.

We are sending copies of this report to the Secretary of Defense; the Chairman and Vice Chairman of the Joint Chiefs of Staff; the Secretaries of the Army, Navy, and Air Force; and the Director of the Office of Management and Budget. In addition, the report will be made available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me at (202) 512-4841. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff members making key contributions to this report are listed in Appendix VI.

Appendix I: Scope and Methodology

To conduct our work, we reviewed relevant sections of Title 10 of the U.S. Code, the Weapon Systems Acquisition Reform Act of 2009 (WSARA), and the National Defense Authorization Act for Fiscal Year 2008 to establish the role of the Joint Requirements Oversight Council (JROC) in considering trade-offs among cost, schedule, and performance objectives; reviewing the estimated level of resources needed to fulfill these requirements; and prioritizing requirements. We also reviewed Department of Defense (DOD), Joint Staff, and military service guidance documents, as well as those for the Joint Capabilities Integration and Development System (JCIDS) for developing and validating military requirements, to determine how these roles have been implemented in policy. To determine how these policies have been implemented in practice, we analyzed information and capability documents contained in the Joint Staff’s Knowledge Management/Decision Support tool. To do so, we first established how many requirements documents—Initial Capabilities Documents (ICD), Capability Development Documents (CDD), and Capability Production Documents (CPD)—were reviewed by the JROC and Joint Capabilities Board (JCB) during fiscal year 2010. We selected fiscal year 2010 as our time frame because WSARA was enacted in May 2009, and this would allow for any changes the JROC would implement as result of this legislation. We then focused our analysis on the unclassified requirement documents reviewed by the JROC and JCB in fiscal year 2010 which identified capability gaps or defined performance requirements for new major defense acquisition programs: 13 ICDs and 7 CDDs. We assessed these documents, as well as briefings presented to the JROC or the JCB, associated meeting minutes, and JROC decision memos. We also examined 15 JROC reviews of programs that incurred substantial cost growth after program start in fiscal year 2010 to determine if cost, schedule, and performance trade-offs were made. We chose this time period to allow for any changes the JROC would implement as result of the enactment of WSARA in May 2009.

To determine the extent to which the JROC has considered trade-offs among cost, schedule, and performance objectives within programs, we reviewed the seven CDDs submitted to the JROC and analyzed the information presented on trade-offs. We focused on CDDs because they are the first requirements documents that contain cost, schedule, and performance objectives. We also examined JROC decision memos to identify whether the JROC provided input on cost, schedule, and performance objectives for the seven proposed programs and analyses of alternatives (AOA) conducted by the military services prior to JROC reviews. We also met with officials from the Joint Staff; Department of the Air Force; Department of the Army; Department of the Navy; Office of the Director of Cost Assessment and Program Evaluation (CAPE); Office of the Under Secretary of Defense (Comptroller); Office of the Assistant Secretary of Defense for Research and Engineering; and respective program offices about these issues. To obtain combatant command views on their participation in the joint requirements process since the implementation of WSARA, we developed a survey administered to DOD’s 10 combatant commands. The survey addressed a range of topics related to the joint requirements process, including the means for combatant commands to provide information on their capability needs. To understand the Joint Staff’s ongoing internal JCIDS review, we assessed the review charter and met with the Joint Staff officials managing the review to discuss the recommendations from the review and how they might affect the JROC’s consideration of trade-offs. We also observed joint requirements meetings and reviewed prior GAO reports.

To determine the quality and effectiveness of efforts to estimate the level of resources needed to fulfill joint military requirements, we assessed the resource estimates used to support the seven unclassified proposed major defense acquisition programs reviewed by the JROC in fiscal year 2010 against the best practices in our cost estimating guide. We used these criteria to determine the extent to which these resource estimates were credible, well documented, accurate, and comprehensive. We scored each best practice as either being Not Met—DOD provided no evidence that satisfies any of the criterion, Minimally Met—DOD provided evidence that satisfies a small portion of the criterion, Partially Met—DOD provided evidence that satisfies about half of the criterion, Substantially Met—DOD provided evidence that satisfies a large portion of the criterion, and Met— DOD provided complete evidence that satisfies the entire criterion. We determined the overall assessment rating by assigning each individual rating a number: Not Met = 1, Minimally Met = 2, Partially Met = 3, Substantially Met = 4, and Met = 5. Then, we took the average of the individual assessment ratings to determine the overall rating for each of the four characteristics. To perform this analysis, we obtained and analyzed program resource estimate supporting documentation, including service cost positions, technical descriptions, work breakdown structures, technology readiness assessments, program schedules, and AOA reports. We also interviewed program and cost estimating officials, when necessary, to gather additional information on these resource estimates and the cost models used to produce them. Each program was also provided with a copy of our assessment of their resource estimates for review and comment.

To determine the extent to which the JROC prioritized requirements and capability gaps, we reviewed the 13 ICDs and 7 CDDs submitted to the JROC and any discussions of priorities and redundancies contained in each document. We also met with officials from the Joint Staff; Department of the Air Force; Department of the Army; Department of the Navy; and Office of the Under Secretary of Defense (Comptroller) to discuss the extent to which the JROC and its supporting bodies have addressed prioritization issues. To understand the Joint Staff’s ongoing internal JCIDS review, we assessed the review charter and met with the Joint Staff officials managing the review to discuss the recommendations from the review and how they might affect the JROC’s prioritization of requirements. We also observed joint requirements meetings and reviewed prior GAO reports.

We conducted this performance audit from June 2010 to June 2011 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Requirements Documents Reviewed by the JCB or JROC in Fiscal Year 2010

In fiscal year 2010, the Joint Capabilities Boards (JCB) and Joint Requirements Oversight Council (JROC) combined to review a total of 45 new requirements documents, including 11 that were classified, 2 that were information technology programs, and 8 documents that were not associated with major defense acquisition programs. The remaining 24 requirements documents are identified in figure 4.

Appendix III: JROC Cost Breach Reviews in Fiscal Year 2010

The Joint Requirements Oversight Council (JROC) conducted 15 reviews following cost breaches in fiscal year 2010—6 Nunn-McCurdy reviews and 9 tripwire reviews. Table 2 identifies these reviews.

Appendix IV: Best Practice Criteria for Resource Estimates

Table 3 below presents the best practice criteria against which we assessed the resource estimates presented to the Joint Requirements Oversight Council during fiscal year 2010 Capability Development Document (CDD) reviews.

Appendix V: Comments from the Department of Defense

Appendix VI: GAO Contact and Acknowledgments

Acknowledgments

In addition to the contact named above, Ronald E. Schwenn, Assistant Director; Noah B. Bleicher; Stephen V. Marchesani; Kenneth E. Patton; Karen A. Richey; Anna K. Russell; and Nathan A. Tranquilli made key contributions to this report.