DOD Weapon Systems: Missed Trade-off Opportunities During Requirements Reviews
Published: 2011-06-16T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Weapon Systems Acquisition Reform Act of 2009 (WSARA) directed the Joint Requirements Oversight Council (JROC) to ensure trade-offs among cost, schedule, and performance objectives are considered as part of its requirements review process. WSARA also directed GAO to assess the implementation of these requirements. This report addresses (1) the extent to which the JROC has considered trade-offs within programs, (2) the quality of resource estimates presented to the JROC, and (3) the extent to which the JROC is prioritizing requirements and capability gaps. To do so, GAO analyzed requirement documents reviewed by the JROC in fiscal year 2010, which identified capability gaps or performance requirements for new major defense acquisition programs. GAO also assessed resource estimates presented to the JROC against best practices criteria in the GAO Cost Estimating and Assessment Guide.
section_title: What GAO Found
paragraphs: The JROC considered trade-offs made by the military services before validating requirements for four of the seven proposed programs it reviewed in fiscal year 2010. According to DOD officials, the most significant trade-offs are made by the military services during the analysis of alternatives (AOA), which occurs between the JROC's review of an Initial Capabilities Document (ICD) and its review of a Capability Development Document (CDD). The AOA is intended to compare the operational effectiveness, cost, and risks of a number of alternative potential solutions. The JROC does not formally review the trade-off decisions made as a result of an AOA until it reviews a proposed program's CDD. As a result, the JROC does not have an opportunity to provide military advice on trade-offs and the proposed solution before it is selected, and a significant amount of time and resources can be expended in technology development before the JROC gets to formally weigh in. The military services did not consistently provide high-quality resource estimates to the JROC for proposed programs in fiscal year 2010. GAO found the estimates presented to the JROC were often unreliable when assessed against best practices criteria. In most cases, the military services had not effectively conducted uncertainty and sensitivity analyses or examined the effects of changing assumptions and ground rules, all of which could further the JROC's efforts to ensure that programs are fully funded and provide a sound basis for making cost, schedule, and performance trade-offs. The JROC does not currently prioritize requirements, consider redundancies across proposed programs, or prioritize and analyze capability gaps in a consistent manner. As a result, the Joint Staff is missing an opportunity to improve the management of DOD's joint portfolio of weapon programs. According to Army, Air Force, and Navy officials, having a better understanding of warfighter priorities from the JROC would be useful to inform both portfolio management efforts and service budgets. A DOD review team examining the JROC's requirements review process is considering changes that would address the prioritization of requirements on a departmentwide basis.
section_title: What GAO Recommends
paragraphs: GAO recommends that the JROC establish a mechanism to review AOA results earlier in the acquisition process, require higher quality resource estimates from requirements sponsors, prioritize requirements across proposed programs, and address potential redundancies during requirements reviews. The Joint Staff partially concurred with GAO's recommendations and generally agreed with their intent, but differed with GAO on how to implement them.

section_title: Letter
paragraphs: With the prospect of slowly growing or flat defense budgets for years to  come, the Department of Defense (DOD) must get better returns on its  weapon system investments and find ways to deliver more capability to  the warfighter for less than it has in the past. In this environment, DOD’s  capacity to make effective trade-offs among cost, schedule, and  performance objectives when developing and validating weapon system  requirements and acquiring those systems will be important to achieving a  balance between DOD’s weapon system investments and resources  available to it.
The Weapon Systems Acquisition Reform Act of 2009 (WSARA) took  several steps to encourage DOD to engage in a more robust discussion of  trade-offs among cost, schedule, and performance objectives before  beginning a new weapon system program. First, WSARA directed DOD’s  Joint Requirements Oversight Council (JROC), which validates joint  military requirements, to ensure trade-offs among cost, schedule, and  performance objectives are considered as part of its process for assessing  and prioritizing requirements. Additionally, WSARA stated that the  requirements development process must be structured to enable  incremental, evolutionary, or spiral acquisition approaches, and that  acquisition, budget, and cost estimating officials should be provided an  opportunity to develop resource estimates and raise cost and schedule  issues before performance objectives are established. Finally, WSARA  amended the U.S. Code to require the JROC to seek and consider, among  other things, the input of combatant commanders when assisting the  Chairman of the Joint Chiefs of Staff in identifying, assessing, and  approving joint military requirements; considering trade-offs; and  establishing and assigning priority levels, among other areas.
WSARA also directed GAO to assess the implementation of these  requirements. This report addresses (1) the extent to which the JROC has  considered trade-offs among cost, schedule, and performance objectives  within programs; (2) the quality and effectiveness of efforts to estimate the  level of resources needed to fulfill joint military requirements; and (3) the  extent to which the JROC is prioritizing requirements and capability gaps.  In addition, the Ike Skelton National Defense Authorization Act for Fiscal  Year 2011 requires GAO to conduct a comprehensive review of the JROC’s  requirements validation process and report its results in 2012.
To conduct our work, we focused on JROC activities in fiscal year 2010.  We chose this time period to allow for any changes the JROC would  implement as a result of the enactment of WSARA in May 2009. To  determine the extent to which the JROC has considered cost, schedule,  and performance trade-offs within programs, we reviewed the seven  Capability Development Documents (CDD) submitted to the JROC in  fiscal year 2010, JROC decision memos related to the CDDs, and analyses  of alternatives (AOA) conducted by the military services prior to JROC  reviews. We focused on CDDs because they are the first requirements  documents that contain cost, schedule, and performance objectives.  Additionally, we reviewed documentation from 15 JROC reviews of  programs that incurred substantial cost growth after program start to  determine if cost, schedule, and performance trade-offs were made. To  determine the quality and effectiveness of efforts to estimate the level of  resources needed to fulfill joint military requirements, we assessed the  resource estimates used to support the 7 CDDs presented to the JROC for  approval. To do so, we applied GAO’s best practice criteria for cost  estimates and reviewed supporting documentation. Each program was  also provided with a copy of our assessment of their resource estimates  for review and comment. To determine the extent to which the JROC  prioritized requirements and capability gaps, we reviewed the 13 Initial  Capabilities Documents (ICD) and 7 CDDs submitted to the JROC in fiscal  year 2010, and any discussions of priorities and redundancies contained in  each document. We also interviewed officials from DOD, the Joint Staff,  and military service headquarters about the extent to which the JROC and  its supporting bodies have addressed prioritization issues.
We conducted this performance audit from June 2010 to June 2011 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings based on our audit  objectives. Appendix I contains detailed information on our scope and  methodology.
subsections: section_title: Background
paragraphs: DOD uses three interrelated processes to deliver capabilities to the U.S.  military: the Joint Capabilities Integration and Development System  (JCIDS), which validates gaps in joint warfighting capabilities and  requirements that resolve those gaps; the Defense Acquisition System,  which develops and fields weapon systems to meet these requirements;  and the Planning, Programming, Budgeting and Execution process, which  allocates the funding needed to develop, acquire, and field these weapon  systems. The JCIDS process is overseen by the JROC, which supports the  Chairman of the Joint Chiefs of Staff in advising the Secretary of Defense  on joint military capability needs. The JROC is chaired by the Vice  Chairman of the Joint Chiefs of Staff, and includes one senior leader from  each of the military services, such as the Vice Chief of Staff of the Army or  the Vice Chief of Naval Operations.
The JROC has a number of statutory responsibilities related to the  identification, validation, and prioritization of joint military requirements.  The JROC assists the Chairman of the Joint Chiefs of Staff with a number  of tasks, including (1) identifying, assessing, and approving joint military  requirements; (2) establishing and assigning priority levels for joint  military requirements; and (3) reviewing the estimated level of resources  required to fulfill each requirement and ensuring that the resource level is  consistent with the requirement’s priority. The JROC also assists  acquisition officials in identifying alternatives to any acquisition programs  that experience significant cost growth.
Since 2008, Congress has added to the JROC’s statutory responsibilities  and increased the number of JROC members and advisors who provide  input to it. The National Defense Authorization Act for Fiscal Year 2008  amended the U.S. Code to require that the Under Secretary of Defense for  Acquisition, Technology and Logistics (USD AT&L), the Under Secretary  of Defense (Comptroller), and the Director of the Office of Program  Analysis and Evaluation serve as advisors to the JROC on matters within  their authority and expertise. In 2009, WSARA expanded the role of the  JROC by directing it to assist the Chairman of the Joint Chiefs of Staff in  (1) ensuring that trade-offs among cost, schedule, and performance  objectives are considered for joint military requirements; and (2)  establishing an objective period of time within which an initial operational  capability should be delivered. WSARA also stated that the newly  constituted Director of Cost Assessment and Program Evaluation (CAPE)  would advise the JROC. The Ike Skelton National Defense Authorization  Act for Fiscal Year 2011 allowed the Vice Chairman of the Joint Chiefs of  Staff to direct senior leaders from combatant commands to serve as  members of the JROC when matters related to the area of responsibility or  functions of that command are under consideration. It also added the  Under Secretary of Defense for Policy, the Director of Operational Test  and Evaluation, and other civilian officials designated by the Secretary of  Defense as advisors to the JROC on issues within their authority and  expertise.
The JROC is supported in the JCIDS process by two Joint Capabilities  Boards (JCB) and seven Functional Capabilities Boards (FCB), each of  which is chaired by a general/flag officer or civilian equivalent. JCBs and  FCBs are responsible for specific Joint Capability Areas, such as Force  Protection, Logistics, or Battlespace Awareness. The JCBs, FCBs, and  associated FCB Working Groups review requirements documents prior to  JROC reviews. The JCB also serves as the validation authority for  requirements documents that are not associated with major defense  acquisition programs (MDAP). In some instances, the JROC will not meet  in person to approve requirements documents if there are no outstanding  issues to discuss.
The JROC and its supporting organizations review requirements  documents related to capability gaps and the MDAPs intended to fill those  gaps prior to key acquisition milestones. These requirements documents— the Initial Capabilities Documents (ICD), Capability Development  Documents (CDD), and Capability Production Documents (CPD)—are  submitted by capability sponsors, which are generally the military  services, but can also be other DOD agencies or combatant commands.  Figure 1 depicts how JCIDS reviews align with the acquisition process.
The ICD is the first requirements document reviewed in JCIDS. It is  intended to identify a specific capability gap, or set of gaps, in joint  military capabilities that are determined to require a materiel solution as a  result of a capabilities-based assessment. DOD policy requires that the  JROC validate the ICD prior to a Materiel Development Decision, which is  the formal entry point into the acquisition process. The ICD does not  contain specific cost, schedule, or performance objectives. Once the JROC  validates an ICD, the Milestone Decision Authority, working with  appropriate stakeholders, shall determine whether to proceed to a  Materiel Development Decision. After the Materiel Development  Decision, the capability sponsor initiates an AOA to consider alternative  solutions to fulfilling the capability need described in an ICD, and possible  trade-offs among cost, schedule, and performance for each alternative are  considered.
The CDD is the second requirements document reviewed in JCIDS. It can  address capability gaps presented in one or more ICDs. The CDD is  intended to define a proposed program’s Key Performance Parameters  (KPP), Key System Attributes (KSA), and other performance attributes.  KPPs are the system characteristics that the CDD sponsor considers critical to delivering that military capability, while KSAs are system  attributes the CDD sponsor considers essential for an effective military  capability, but a lower priority than the KPPs. DOD policy calls for the  JROC to validate the CDD to inform the Milestone B decision, which  marks the official start of an acquisition program and entry into the  engineering and manufacturing development phase. The CDD is the first requirements document that contains cost, schedule, and performance  objectives.
The CPD is the third and final requirements document reviewed in J It is intended to refine the KPPs, KSAs, and performance attributes  validated in the CDD. DOD policy calls for the JROC to validate the CP inform the Milestone C decision, which marks a program’s entry into  production. Appendix II identifies the IC the JCB or JROC in fiscal year 2010.
CIDS.
section_title: JROC Did Not Always Consider Trade-offs or Influence Trade-off Decisions
paragraphs: The JROC considered trade-offs made by the military services before  validating requirements for four of the seven proposed programs it  reviewed in fiscal year 2010, and provided input to the military services on  the cost, schedule, and performance objectives for two of the seven  programs. The JROC’s requirements review was the final step in a long  requirements vetting process, with most trade-offs being made by the  military services earlier in the process. Key stakeholders from the offices  of the Under Secretary of Defense (Comptroller), USD AT&L, Director of  CAPE, and the combatant commands were all satisfied with their  opportunities to provide input to the JROC; but they provided limited  input on trade-offs among cost, schedule, and performance objectives, and  used other means to influence trade-offs. Perhaps most importantly, none  of the JROC’s requirements reviews align with the AOA, which is where  the military services reported making the most significant trade-offs. As a  result, a program can spend significant time in technology development  before the JROC gets to formally weigh in on these trade-offs through the  JCIDS process. The JROC also reviews MDAP requirements after a  program enters development and experiences substantial cost growth.  DOD and the JROC stated that requirements were not the primary causes  of cost growth for the 15 programs reviewed for this purpose in fiscal year  2010 and the JROC did not change any KPPs to mitigate the reported cost  growth.
subsections: section_title: JROC Did Not Always Consider Trade-offs when Validating Requirements for Proposed Programs
paragraphs: The JROC considered trade-offs made by the military services before  validating requirements for four of the seven proposed programs it  reviewed in fiscal year 2010. On three programs, the JROC did not receive  information on the potential cost and schedule implications of each of the  alternatives considered. Table 1 summarizes the JROC’s consideration of  cost, schedule, and performance objectives for the seven proposed MDAPs  it reviewed in fiscal year 2010.
The JROC’s review of the CDD for a proposed program is the final step in  a long requirements vetting process, and DOD officials reported that trade- offs typically occur earlier in the process. Each military service conducts  its own internal requirements reviews for its proposed programs, which  are used to refine requirements documents before they are submitted into  JCIDS. Military service officials reported that they make significant trade- offs during these internal reviews, and that KPPs and technical  requirements rarely change after requirements documents are submitted  into JCIDS because extensive analysis has already been conducted. For  the seven proposed MDAPs we reviewed, the military services generally  submitted requirements to the JROC that would be fully funded, provide  initial capability within 6 years, utilize critical technologies that were  nearing maturity, and be acquired using an incremental approach. These  characteristics are consistent with provisions in the Weapon Systems  Acquisition Reform Act (WSARA) related to how the requirements process  should be structured and aspects of GAO’s best practices for weapon  system acquisitions.
Two of the proposed program requirements presented to the JROC  included major trade-offs among cost, schedule, and performance  objectives and revisions to their acquisition approaches that had been  made after predecessor programs were cancelled over affordability  concerns. The Air Force initiated the HH-60 Recapitalization program after  the Combat Search and Rescue Replacement Vehicle (CSAR-X) program  was cancelled, and the HH-60 Recapitalization program is expected to  decrease cost by changing cabin space, velocity, and range from the CSAR- X requirements. In 2007, the Army, with input from a Functional  Capabilities Board, decided to use an incremental acquisition approach for  the Ground Soldier System in order to reduce costs, meet schedule  demands, and avoid some of the mistakes made during the Land Warrior  program, which was cancelled because of funding and cost issues.
section_title: Key Stakeholders Provided Limited Input into JCIDS, but Use Other Means to Influence Trade-offs
paragraphs: The JROC received limited input on trade-offs among cost, schedule, and  performance objectives from key stakeholders when validating  requirements for the seven proposed MDAPs we reviewed from fiscal year  2010. Both WSARA and the National Defense Authorization Act for Fiscal  Year 2008 directed the JROC to consult with the Under Secretary of  Defense (Comptroller), the USD AT&L, and the Director of CAPE.  Additionally, WSARA instructed the JROC to consult with the combatant  commands. Officials from these organizations reported that they had  ample opportunity to participate in JROC requirements reviews, and Joint  Staff officials said efforts to involve these stakeholders preceded WSARA.  However, officials from the offices of the Under Secretary of Defense  (Comptroller), USD AT&L, and the Director of CAPE also reported that  the acquisition and budgeting/funding processes are the primary  mechanisms through which they influence programs, rather than JCIDS.  For example, CAPE oversees AOAs for MDAPs and has an opportunity to  provide input and guidance on AOA considerations. Further, the  combatant commands reported that they most often submit prioritized  lists of capability gaps directly to the Chairman of the Joint Chiefs of Staff  as part of the resource allocation process, which is separate from JCIDS.
Nonetheless, joint stakeholders did provide some significant input during  the JROC’s reviews of the seven proposed programs in fiscal year 2010.  For example, the Army more fully defined a Ground Soldier System,  Increment 1 KPP in response to input from DOD’s Joint Interoperability  Test Command, and in another instance, the Army added a KSA to the  AIAMD SOS, Increment 2 CDD due to input from the office of the USD  AT&L, the Defense Information Systems Agency, and the Joint Staff.  Neither of these changes involved trade-offs among cost, schedule, and  performance objectives.
section_title: JCIDS Reviews Are Not Aligned with the Most Significant Trade-off Decisions
paragraphs: The JROC does not formally review the trade-off decisions made as a  result of an AOA until a proposed program’s CDD enters the JCIDS  process. According to DOD officials, the most significant trade-offs are  made by the military services between ICD and CDD reviews during the  AOA, which is intended to compare the operational effectiveness, cost,  and risks of a number of alternative potential solutions. For example,  during the CVLSP AOA, the Air Force decided to decrease troop transport  capacity in order to reduce cost. Alternatively, during the AIAMD SOS  AOA, the Army decided to pursue the most costly option reviewed  because it provided greater capability. A significant amount of time and  resources can be expended before the JROC gets to weigh in on these  trade-offs during CDD reviews. For example, the JROC reviewed the AOA  summary for JPALS, Increment 2, 4 years after the conclusion of the AOA.  During the time between the AOA and the CDD review, the technology  intended to enable the chosen alternative is developed. Figure 2 shows the  AOA’s relationship to both the requirements and acquisition processes.
Joint Staff officials have stated that establishing a JROC review of the AOA  would allow it to provide military advice on trade-offs and the proposed  materiel solution before Milestone A, and an ongoing Joint Staff review of  JCIDS is considering an increased role for the JROC at this point.  According to the Joint Staff, increased JROC engagement at these early  stages of the acquisition process is warranted to align it with other  elements of recent acquisition reforms. For example, WSARA emphasized  that the AOA should fully consider possible trade-offs among cost,  schedule, and performance objectives for each alternative considered,  and in September 2010, USD AT&L issued a memorandum that  emphasized the need for trade-offs from a program’s inception. The  memorandum also dictated that affordability targets shall be established at  the conclusion of the AOA and that these targets will be treated like KPPs,  even though they will be set and managed by the acquisition, not  requirements, community.
section_title: JROC Did Not Change KPPs when Programs Incurred Substantial Cost Growth
paragraphs: The JROC did not change any KPPs during 15 reviews of programs that  reported substantial cost growth in fiscal year 2010. According to the Joint  Staff, by holding requirements firm and accepting increased cost and  schedule delays, the JROC essentially traded cost and possibly schedule  for performance. In fiscal year 2010, the JROC reviewed six programs after  they experienced a critical Nunn-McCurdy breach and nine programs as  part of the tripwire process. During all 15 reviews, DOD and the JROC  stated that requirements were not the primary causes of cost growth. For  all six programs that experienced a critical Nunn-McCurdy cost breach,  the JROC validated the system’s capabilities as being essential to national  security and did not make any changes to their KPPs. For all nine  programs that were approaching Nunn-McCurdy thresholds, the JROC did  not identify opportunities to mitigate cost growth by modifying  requirements. Most of these programs were in production in fiscal year  2010, and changing requirements at this late stage might not have  mitigated the reported cost growth. When the JROC reviewed the Family  of Advanced Beyond Line-of-Sight Terminals program, which was still in  development, it concluded that the program’s requirements could not be  met in an affordable manner. The JROC did not immediately defer any of  the program’s requirements, but instead requested that USD AT&L identify  potential alternatives for the program, including reviewing whether  adjustments to performance requirements would be appropriate.
section_title: Military Services Did Not Consistently Provide High-Quality Resource Estimates to JROC
paragraphs: The military services did not consistently provide high-quality resource  estimates to the JROC to support its review of requirements for 7  proposed programs in fiscal year 2010. We found the estimates presented  to the JROC were often unreliable when assessed against best practices  criteria. The type of resource estimates the military services presented to  the JROC varied from ones that had been validated by the military  services’ cost analysis agencies to less rigorous rough-orders-of-magnitude  estimates. In most cases, the military services had not effectively  conducted uncertainty and sensitivity analyses, which establish  confidence levels for resource estimates, based on the knowledge  available, and examine the effects of changing assumptions and ground  rules. Lacking risk and uncertainty analysis, the JROC cannot evaluate the  range of resources that might be necessary to cover increased costs  resulting from unexpected design complexity, technology uncertainty, and  other issues. The lack of this information affects the JROC’s efforts to  ensure that programs are fully funded and its ability to consider the  resource implications of cost, schedule, and performance trade-offs.
subsections: section_title: JROC Received Resource Estimates That Did Not Meet Best Practices
paragraphs: The JROC first receives resource estimates for proposed programs when it  reviews CDDs, and when we reviewed the CDD resource estimates  presented to the JROC in fiscal year 2010, we found that they were  generally unreliable when assessed against our best practices criteria.  While most of the resource estimates substantially met our criteria for a  comprehensive resource estimate, they generally were not very accurate,  credible, or well-documented. Appendix IV includes a list of the best  practices against which we assessed these resource estimates.
The type of resource estimates the military services presented to the JROC  varied from ones that had been validated by the military services’ cost  analysis agencies to less rigorous rough-orders-of-magnitude estimates.  According to Joint Staff officials, military services can initiate CDD  reviews at any point in the acquisition process prior to program start, even  if good resource estimates are not available. For example, the JROC  validated the P-8A, Increment 3 CDD more than 2 years before the  program was expected to start, before an AOA had been completed, and  with a rough-order-of-magnitude estimate. Joint Staff officials reported  that they depend on CAPE to review the quality of resource estimates  during the JCIDS process, but CAPE cost assessment officials told us that  they rarely participate in JCIDS reviews.
Regardless of the type of resource estimate, uncertainty and sensitivity  analysis can establish confidence levels for resource estimates, based on  the knowledge available at the time, and examine the effects of changing  assumptions and ground rules, including those related to trade-offs among  cost, schedule, and performance objectives. The military services  sponsoring the requirements generally did not effectively meet best  practices for uncertainty and sensitivity analyses using the knowledge they  had available to them for any of the seven resource estimates we  reviewed. Figure 3 summarizes our assessment of the resource estimates  presented to the JROC against our best practices criteria.
Five of the seven CDD resource estimates substantially met our criteria for  a comprehensive resource estimate. The resource estimates generally  completely defined their respective programs, and included most, if not  all, life-cycle costs. The Ship to Shore Connector, CVLSP, and JPALS,  Increment 2 resource estimates also effectively documented all cost- influencing ground rules and assumptions, although the other resource  estimates did not. Additionally, only the Ship to Shore Connector’s work  breakdown structure effectively met our criteria, which require that work  breakdown structures are product-oriented and at an appropriate level of  detail. If a resource estimate does not specifically break out common  costs, such as government-furnished equipment costs, or does not include  an associated work breakdown structure dictionary, cost estimators  cannot ensure that the estimate includes all relevant costs.
The HH-60 Recapitalization and P-8A, Increment 3 resource estimates did  not effectively meet any of our best practices for a comprehensive  resource estimate. Unless resource estimates account for all costs, they  cannot enhance decision making by allowing for design trade-off studies  to be evaluated on a total cost, technical, and performance basis.  Additionally, unless ground rules and assumptions are clearly  documented, the resource estimate will not have a basis for resolving  areas of potential risk.
subsections: section_title: Most Resource Estimates Did Not Substantially Meet Accuracy Criteria
paragraphs: Only two of the seven CDD resource estimates substantially met our  criteria for an accurate resource estimate, while three partially met the  criteria, and two did not meet or minimally met the criteria. We found that  the Ship to Shore Connector, CVLSP, AIAMD SOS, Increment 2, and the  Ground Soldier System, Increment 1 resource estimates contained few, if  any, minor mistakes, and that the Ship to Shore Connector, CVLSP, and  JPALS, Increment 2 resource estimates were appropriately adjusted for  inflation. Additionally, we found that the Ship to Shore Connector and  JPALS, Increment 2 resource estimates were based on historical records  of actual experiences from other comparable programs.
However, we generally found that the resource estimates were not  consistent with our best practices. Accurate resource estimates are rooted  in historical data, which provide cost estimators with insight into actual  costs of similar programs, and can be used to challenge optimistic  assumptions and bring more realism to a resource estimate. Unless an  estimate is based on an assessment of the most likely costs, and reflects  the degree of uncertainty given all of the risks considered, management  will not be able to make well-informed decisions.
section_title: Most Resource Estimates Were Not Credible
paragraphs: Four of the seven CDD resource estimates did not meet or minimally met  our criteria for a credible resource estimate, and only the Ship to Shore  Connector resource estimate substantially met the criteria. The Ship to  Shore Connector and AIAMD SOS, Increment 2 resource estimates  included sensitivity analyses that identified a range of possible costs based  on varying assumptions, parameters, and data inputs, but none of the  other resource estimates included this analysis. As a best practice,  sensitivity analysis should be included in all resource estimates because it  examines the effects of changing assumptions and ground rules. Since  uncertainty cannot be avoided, it is necessary to identify the cost elements  that represent the most risk and, if possible, cost estimators should  quantify that risk. When an agency fails to conduct sensitivity analysis to  identify the effect of uncertainties associated with different assumptions,  this increases the chance that decisions will be made without a clear  understanding of the impact on cost.
Additionally, only the Ship to Shore Connector resource estimate  effectively met our best practices for risk and uncertainty analysis. For  management to make good decisions, the program estimate must reflect  the degree of uncertainty so that a level of confidence can be given about  the estimate. An estimate without risk and uncertainty analysis is  unrealistic because it does not assess the variability in the resource  estimate from effects such as schedules slipping, missions changing, and  proposed solutions not meeting users’ needs. Lacking risk and uncertainty  analysis, management cannot determine a defensible level of contingency  reserves that is necessary to cover increased costs resulting from  unexpected design complexity, technology uncertainty, and other issues.
Further, none of the planned programs effectively met our criteria for an  independent cost estimate when they were reviewed by the JROC. An  independent cost estimate is considered one of the best and most reliable  resource estimate validation methods because it provides an independent  view of expected program costs that tests the program office and service  estimates for reasonableness. Without an independent cost estimate,  decision makers lack insight into a program’s potential costs because  these estimates frequently use different methods and are less burdened  with organizational bias. Moreover, independent cost estimates tend to  incorporate adequate risk, and therefore tend to be more conservative by  forecasting higher costs than the program office. A program estimate that  has not been reconciled with an independent cost estimate has an  increased risk of proceeding underfunded because an independent cost  estimate provides an objective and unbiased assessment of whether the  program estimate can be achieved. Alternatively, programs can reinforce  the credibility of their resource estimates through cross-checking, which  determines whether alternative cost estimating methods produce similar  results. However, only the Ship to Shore Connector resource estimate  effectively met our best practices for cross-checking.
section_title: Most Resource Estimates Were Not Well-documented
paragraphs: Only the JPALS, Increment 2 resource estimate substantially met our  criteria for a well-documented resource estimate, while four of the seven  CDD resource estimates partially met our criteria, and two of the resource  estimates did not meet or minimally met the criteria. The JPALS,  Increment 2 and CVLSP resource estimates sufficiently described the  calculations performed and estimating methodologies used to derive each  program element’s cost. Additionally, the JPALS, Increment 2, Ship to  Shore Connector, and AIAMD SOS, Increment 2 documentation clearly  discusses the technical baseline description, and the data in the technical  baseline are consistent with the resource estimate. However, none of the  documents effectively described how the resource estimates were  developed in a manner that a cost analyst unfamiliar with the program  could understand what was done and replicate it.
We generally found that the resource estimates were not consistent with  our best practices for a well-documented resource estimate.  Documentation is essential for validating and defending a resource  estimate. Without a well-documented resource estimate, a convincing  argument of an estimate’s validity cannot be presented, and decision  makers’ questions cannot be effectively answered. Poorly documented  resource estimates cannot explain the rationale of the methodology or the  calculations underlying the cost elements. Further, a well-documented  resource estimate is essential for an effective independent review to  ensure that the resource estimate is valid and credible. Unless the estimate  is fully documented, it will not support reconciliation with an independent  cost estimate, hindering understanding of cost elements and their  differences.
section_title: Military Services Generally Presented Resource Estimates That Were Fully Funded to JROC
paragraphs: The JROC required the military services to show that the proposed  programs were fully funded to the resource estimates presented by the  military services before it validated requirements for five of the seven  proposed MDAPs we reviewed from fiscal year 2010; the two other  proposed MDAPs were funded at more than 97 and 99 percent  respectively. However, we found that these resource estimates were  generally unreliable, which undermined the JROC’s efforts. In 2007, the  JROC issued guidance instructing the military services to commit to  funding the requirements that the JROC validates. The guidance  emphasized the need for full funding in an effort to facilitate sound fiscal  and risk decisions. However, the JROC does not explicitly consider a  requirement’s affordability in a broader context during JCIDS reviews.  DOD funding plans are captured in the future-years defense program,  which presents resource information for the current year and the  following 4 years. The future-years defense program is updated twice per  year to reflect the military services’ input and the budget the President  submits to Congress. Statute and DOD acquisition policy also require  programs to be fully funded through the period covered by the future- years defense program.
One of the seven proposed MDAPs we reviewed from fiscal year 2010  included a funding shortfall when its requirements were being reviewed  through JCIDS, but its CDD was not approved until the shortfall had been  addressed. Specifically, when the JCB reviewed the CVLSP CDD, the  funding plan included a $1.3 billion shortfall through fiscal year 2015. The  JCB chairman directed the Air Force to modify the program’s funding plan  before proceeding to the JROC review. When the Air Force briefed the  JROC on the CVLSP CDD approximately 8 months later, it presented a  funding plan that fully funded the program through the future-years  defense program time frame. The revised funding plan also included more  money for the program beyond the future-years defense program time  frame, and the total program cost increased from $14.2 billion to $15.2  billion.
Despite JROC efforts to ensure programs are fully funded, the military  services retain primary control over their budgets, and ultimately, JROC  decisions are influential but not binding. When the JCB reviewed the  JPALS, Increment 2 CDD, it requested clarification on the Air Force’s  funding plan, and emphasized the need for full funding prior to program  start. The funding plan presented to the JCB included a $77.7 million  shortfall through fiscal year 2015, and the Air Force had cut JPALS funding  in the past. Following the JCB review, the JROC issued a decision  memorandum that documented the Air Force’s commitment to fully  funding JPALS, Increment 2. However, in fiscal years 2011 and 2012, the  Air Force only funded approximately 30 percent of the resource estimate  presented to the JCB.
section_title: JROC Did Not Consistently Prioritize Requirements and Capability Gaps
paragraphs: The JROC does not currently prioritize requirements, consider  redundancies across proposed programs, or prioritize and analyze  capability gaps in a consistent manner. As a result, the Joint Staff is  missing an opportunity to improve military service and departmentwide  portfolio management efforts. A portfolio management approach to  weapon system investments would involve taking a disciplined, integrated  approach to prioritizing needs and allocating resources in order to  eliminate redundancies, gain efficiencies, and achieve a balanced mix of  executable programs. According to Army, Air Force, and Navy officials,  having a better understanding of warfighter priorities from the JROC  would be useful to inform both portfolio management efforts and service  budgets. A DOD review team examining the JCIDS process is considering  changes that would address the prioritization of requirements. During its  review of the capability gaps presented in 12 ICDs in fiscal year 2010, the  JROC did receive some information on priorities and potential  redundancies; however, the sponsors presented this information in an  inconsistent manner, making it difficult for the JROC to assess the relative  priority of capability gaps across different ICDs.
subsections: section_title: JROC Does Not Prioritize Requirements or Consider Redundancies Across Proposed Programs
paragraphs: Under the current JCIDS process, the JROC does not prioritize  requirements or consider redundancies across proposed programs during  CDD reviews. In the National Defense Authorization Act for Fiscal Year  2008, Congress amended the U.S. Code to direct the JROC to help assign  priority levels for joint military requirements and ensure that resource  levels associated with those requirements are consistent with the level of  priority. The House Armed Services Committee report accompanying the  authorization act stated that clear JROC priorities and budget guidance  would allow for joint decision making, as opposed to service-centric  budget considerations. In addition, we have previously recommended  that DOD develop an analytic approach within JCIDS to better prioritize  and balance the capability needs of the military services, combatant  commands, and other defense components. According to the Joint Staff  and military service officials, prioritization across programs still primarily  occurs through the Planning, Programming, Budgeting and Execution  process, which is the responsibility of the military services and the Office  of the Under Secretary of Defense (Comptroller).
The JCIDS manual does not currently require an analysis of potential  redundancies during CDD reviews. In our recently issued report on  government duplication, we noted that service-driven requirements and  funding processes continue to hinder integration and efficiency and  contribute to unnecessary duplication in addressing warfighter needs. We  have also previously reported that ineffective collaboration precluded  opportunities for commonality in unmanned aircraft systems. In fiscal  year 2010, the JROC met to consider joint efficiencies between two such  systems: the Navy’s Broad Area Maritime Surveillance system and the Air  Force’s Global Hawk system. The JROC requested that the Navy and Air  Force ensure that a common component was interoperable between the  two systems, and that the Air Force consider an all-weather capability  developed by the Navy. The JROC has also supported joint development  efforts for these programs and requested annual status updates. According  to Broad Area Maritime Surveillance program officials, the Air Force and  Navy programs are investigating commonality opportunities, including  sense-and-avoid capabilities, a consolidated maintenance hub, and basing  options for both systems. The JROC did not meet to consider any other  joint efficiencies across military services in fiscal year 2010.
The Joint Staff has acknowledged that the JROC should play a larger role  in prioritizing needs and addressing redundancies. In July 2010, the Vice  Chairman of the Joint Chiefs of Staff initiated a review of the JCIDS  process. One of the goals of the review team was to develop metrics and  criteria to ensure the JCIDS process has the ability to rank or prioritize  needs. The review team’s charter states that these metrics must enable  more structured reviews of portfolio gaps and redundancies. According to  the Joint Staff, the review team is considering a number of  recommendations including asking the JROC to prioritize requirements  based on the urgency and significance of the need. This list of priorities  could be used to inform military service budgets. Joint Staff officials have  also stated that redundancies may be addressed more directly in the future  as part of an enhanced portfolio management effort.
section_title: Lack of JROC Prioritization Results in Missed Opportunities to Manage Portfolios Better
paragraphs: We have previously reported that DOD has not taken a portfolio  management approach to weapon system investments, which would  involve taking a disciplined, integrated approach to prioritizing needs and  allocating resources in order to eliminate redundancies, gain efficiencies,  and achieve a balanced mix of executable programs. In September 2010,  USD AT&L issued guidance intended to increase efficiencies and eliminate  redundancies, and it presented the Army’s portfolio management activities  as an example to emulate. The Army uses capability portfolio reviews of  capability gaps and proposed and existing programs to revalidate, modify,  or terminate requirements and ensure the proper allocation of funds  between them. The Army has established 17 portfolios, including aviation,  air and missile defense, and combat vehicle modernization. An Army  official involved in the portfolio reviews said that he has requested on  several occasions for the Joint Staff to prioritize warfighter needs;  however, the JROC has not done so. Instead, the Army relies on its own  prioritization information during the portfolio reviews to help determine  the capability areas where it is willing to assume risk. Air Force and Navy  officials have also stated that they could benefit from JROC prioritization  of requirements, and that this information would be useful in order to  better allocate resources during their budget formulation activities.
section_title: Sponsors Do Not Prioritize Capability Gaps or Analyze Potential Redundancies in a Consistent Manner
paragraphs: The JROC has required that capability sponsors prioritize capability gaps  and identify redundancies when developing ICDs, and capability  sponsors generally complied with these requirements in the 12 validated  ICDs we reviewed from fiscal year 2010. However, the sponsors  presented this information in an inconsistent manner, making it difficult  for the JROC and the military services to assess priorities and  redundancies across ICDs or use this information to inform resource  allocation decisions. For example, the Electronic Health Record ICD  prioritized its gaps in numerical order from 1 to 10, but the Command and  Control On-The-Move ICD labeled half its gaps medium priority and the  other half high priority. The JCIDS operation manual provides limited  guidance on how capability sponsors should prioritize the gaps, stating  only that the prioritization should be based on the potential for operational  risk associated with the shortfalls. The JCIDS manual also directs  capability sponsors to identify redundancies and assess whether the  overlap is operationally acceptable or whether it should be evaluated as  part of the trade-offs to satisfy capability gaps. Three of the 12 validated  ICDs we reviewed from fiscal year 2010 did not address redundancies.  Furthermore, only one of these ICDs presented to the JROC in fiscal year  2010 included an evaluation of the overlaps. The JROC did not address  these omissions when it validated the documents.
section_title: Conclusions
paragraphs: In the last several years, Congress has passed legislation to give the JROC  a greater role in prioritizing military requirements and shaping sound  acquisition programs by encouraging cost, schedule, and performance  trade-offs. Taken together, these steps have the potential to improve the  affordability and execution of DOD’s portfolio of major defense  acquisition programs. However, the JROC has largely left prioritization  and trade-off decisions to the military services, despite having a unique,  joint perspective, which would allow it to look across the entire  department to identify efficiencies and potential redundancies. To more  effectively leverage its unique perspective, the JROC would have to change  the way it views its role, more regularly engage the acquisition community  in trade-off discussions at early acquisition milestones, and more  effectively scrutinize the quality of the resource estimates presented by the  military services. Until it does so, the JROC will only be a marginal player  in DOD’s efforts to align the department’s available resources with its  warfighting requirements.
section_title: Recommendations for Executive Action
paragraphs: To enhance the JROC’s role in DOD-wide efforts to deliver better value to  the taxpayer and warfighter, we recommend that the Vice Chairman of the  Joint Chiefs of Staff, as chairman of the JROC, take the following five  actions:    Establish a mechanism to review the final AOA report prior to  Milestone A to ensure that trade-offs have been considered and to  provide military advice on these trade-offs and the proposed materiel  solution to the Milestone Decision Authority.
Require that capability sponsors present resource estimates that have  been reviewed by a military service’s cost analysis organization to  ensure best practices are being followed.
Require that capability sponsors present key results from sensitivity  and uncertainty analyses, including the confidence levels associated  with resource estimates, based on the program’s current level of  knowledge.
Assign priority levels to the CDDs based on joint force capability gaps  and redundancies against current and anticipated threats, and provide  these prioritization levels to the Under Secretary of Defense  (Comptroller) and the military services to be used for resource  allocation purposes.
Modify the JCIDS operations manual to require that CDDs discuss  potential redundancies across proposed and existing programs, and  address these redundancies when validating requirements.
section_title: Agency Comments and Our Evaluation
paragraphs: The Joint Staff provided us written comments on a draft of this report. The  comments are reprinted in appendix V. The Joint Staff also provided  technical comments, which we addressed in the report, as appropriate.
In its comments, the Joint Staff partially concurred with all five of our  recommendations, generally agreeing that there is a need to take action to  address the issues we raised, but differing in terms of the specific actions  that should be taken.
The Joint Staff partially concurred with our recommendation that the Vice  Chairman of the Joint Chiefs of Staff, as chairman of the JROC, establish a  mechanism to review the final AOA report prior to Milestone A to ensure  that trade-offs have been considered and to provide military advice on  these trade-offs and the proposed materiel solution to the Milestone  Decision Authority. The Joint Staff noted that its ongoing review of JCIDS  will include a recommendation that AOA results be briefed to FCBs.  However, the FCB will only elevate these briefings to the JCB or JROC on  an exception basis. The Joint Staff explained that this approach would  allow the JROC to provide more informed advice to a Milestone Decision  Authority without adding another round of staffing, an additional JCIDS  document, or an official validation of AOA results. We agree that the Joint  Staff should seek to implement this recommendation in the most efficient  and effective way possible; however, given our finding that the most  significant trade-off decisions are made as a result of an AOA, we continue  to believe that the results should be reviewed by the JROC.
The Joint Staff partially concurred with our recommendation that the Vice  Chairman of the Joint Chiefs of Staff require that capability sponsors  present resource estimates that have been reviewed by a military service’s  cost analysis organization to ensure best practices are being followed. The  Joint Staff stated that program office cost estimates are compared to  independent cost estimates during CDD reviews. However, none of the  seven CDD cost estimates we reviewed effectively met our criteria for an  independent cost estimate. As a result, we believe that the Joint Staff  needs to take additional action to ensure that resource estimates  presented by capability sponsors have been reviewed by a military  service’s cost analysis organization. The Joint Staff also stated that its  ongoing review of JCIDS will examine how to highlight this area during  CDD reviews.
The Joint Staff partially concurred with our recommendation that the Vice  Chairman of the Joint Chiefs of Staff require that capability sponsors  present key results from sensitivity and uncertainty analyses, including the  confidence levels associated with resource estimates, based on the  program’s current level of knowledge. The Joint Staff stated that our  recommendation needs further study to understand the expected  outcomes and the required authorities for the JROC, and its ongoing  review of JCIDS will examine how to highlight this area. We believe that  the JROC cannot fully consider trade-offs or the affordability of a  proposed program unless it receives information on the risk and  uncertainty associated with resource estimates; it does not need additional  authority to require capability sponsors to present the results of this type  of analysis before it approves proposed requirements. The Joint Staff also  noted that the Director, CAPE, has cost analysis responsibilities for  resource estimates. CAPE cost assessment officials reported that they  rarely participated in JCIDS reviews. As a result, the JROC may have to be  more proactive in reaching out to CAPE to help it understand the risk and  uncertainty associated with the resource estimates it receives.
The Joint Staff partially concurred with our recommendation that the Vice  Chairman of the Joint Chiefs of Staff assign priority levels to CDDs based  on joint force capability gaps and redundancies against current and  anticipated threats, and provide these prioritization levels to the Under  Secretary of Defense (Comptroller) and the military services to be used for  resource allocation purposes. The Joint Staff agreed that the identification  of joint priorities could enhance a number of processes, including program  and budget reviews. It noted that its ongoing review of JCIDS will  recommend a prioritization framework through which CDDs will inherit  priority levels based on the requirements and capability gaps identified in  ICDs or Joint Urgent Operational Needs. However, the Joint Staff argued  against prioritizing based on CDDs directly because it would provide less  flexibility. We believe that the proposed approach could be effective if the  Joint Staff addresses the inconsistencies we found in the way ICDs  prioritize gaps. In addition, we continue to believe that the prioritization  framework should facilitate an examination of priorities across CDDs.
The Joint Staff partially concurred with our recommendation that the Vice  Chairman of the Joint Chiefs of Staff modify the JCIDS operations manual  to require that CDDs discuss potential redundancies across proposed and  existing programs, and address these redundancies when validating  requirements. The Joint Staff stated that its ongoing review of JCIDS will  address this issue by establishing unique requirements as a higher priority  than unnecessarily redundant requirements, and by establishing a post- AOA review, which could also be used to identify unnecessary  redundancies. The Joint Staff did not address whether it would update the  JCIDS operations manual as recommended and stated that reviewing  assessments of redundancies in CDDs would be late in the JCIDS process.  We believe that potential redundancies should be discussed at multiple  points, including during CDD reviews, because we found that several years  can pass between the conclusion of an AOA and this review. During that  time, new redundancy issues could emerge.
We are sending copies of this report to the Secretary of Defense; the  Chairman and Vice Chairman of the Joint Chiefs of Staff; the Secretaries of  the Army, Navy, and Air Force; and the Director of the Office of  Management and Budget. In addition, the report will be made available at  no charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact me at (202) 512-4841. Contact points for our offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Staff members making key contributions to this report are  listed in Appendix VI.
section_title: Appendix I: Scope and Methodology
paragraphs: To conduct our work, we reviewed relevant sections of Title 10 of the U.S.  Code, the Weapon Systems Acquisition Reform Act of 2009 (WSARA), and  the National Defense Authorization Act for Fiscal Year 2008 to establish  the role of the Joint Requirements Oversight Council (JROC) in  considering trade-offs among cost, schedule, and performance objectives;  reviewing the estimated level of resources needed to fulfill these  requirements; and prioritizing requirements. We also reviewed Department  of Defense (DOD), Joint Staff, and military service guidance documents, as  well as those for the Joint Capabilities Integration and Development  System (JCIDS) for developing and validating military requirements, to  determine how these roles have been implemented in policy. To determine  how these policies have been implemented in practice, we analyzed  information and capability documents contained in the Joint Staff’s  Knowledge Management/Decision Support tool. To do so, we first  established how many requirements documents—Initial Capabilities  Documents (ICD), Capability Development Documents (CDD), and  Capability Production Documents (CPD)—were reviewed by the JROC  and Joint Capabilities Board (JCB) during fiscal year 2010. We selected  fiscal year 2010 as our time frame because WSARA was enacted in May  2009, and this would allow for any changes the JROC would implement as  result of this legislation. We then focused our analysis on the unclassified  requirement documents reviewed by the JROC and JCB in fiscal year 2010  which identified capability gaps or defined performance requirements for  new major defense acquisition programs: 13 ICDs and 7 CDDs. We  assessed these documents, as well as briefings presented to the JROC or  the JCB, associated meeting minutes, and JROC decision memos. We also  examined 15 JROC reviews of programs that incurred substantial cost  growth after program start in fiscal year 2010 to determine if cost,  schedule, and performance trade-offs were made. We chose this time  period to allow for any changes the JROC would implement as result of  the enactment of WSARA in May 2009.
To determine the extent to which the JROC has considered trade-offs  among cost, schedule, and performance objectives within programs, we  reviewed the seven CDDs submitted to the JROC and analyzed the  information presented on trade-offs. We focused on CDDs because they  are the first requirements documents that contain cost, schedule, and  performance objectives. We also examined JROC decision memos to  identify whether the JROC provided input on cost, schedule, and  performance objectives for the seven proposed programs and analyses of  alternatives (AOA) conducted by the military services prior to JROC  reviews. We also met with officials from the Joint Staff; Department of the  Air Force; Department of the Army; Department of the Navy; Office of the  Director of Cost Assessment and Program Evaluation (CAPE); Office of  the Under Secretary of Defense (Comptroller); Office of the Assistant  Secretary of Defense for Research and Engineering; and respective  program offices about these issues. To obtain combatant command views  on their participation in the joint requirements process since the  implementation of WSARA, we developed a survey administered to DOD’s  10 combatant commands. The survey addressed a range of topics related  to the joint requirements process, including the means for combatant  commands to provide information on their capability needs. To  understand the Joint Staff’s ongoing internal JCIDS review, we assessed  the review charter and met with the Joint Staff officials managing the  review to discuss the recommendations from the review and how they  might affect the JROC’s consideration of trade-offs. We also observed joint  requirements meetings and reviewed prior GAO reports.
To determine the quality and effectiveness of efforts to estimate the level  of resources needed to fulfill joint military requirements, we assessed the  resource estimates used to support the seven unclassified proposed major  defense acquisition programs reviewed by the JROC in fiscal year 2010  against the best practices in our cost estimating guide. We used these  criteria to determine the extent to which these resource estimates were  credible, well documented, accurate, and comprehensive. We scored each  best practice as either being Not Met—DOD provided no evidence that  satisfies any of the criterion, Minimally Met—DOD provided evidence that  satisfies a small portion of the criterion, Partially Met—DOD provided  evidence that satisfies about half of the criterion, Substantially Met—DOD  provided evidence that satisfies a large portion of the criterion, and Met— DOD provided complete evidence that satisfies the entire criterion. We  determined the overall assessment rating by assigning each individual  rating a number: Not Met = 1, Minimally Met = 2, Partially Met = 3,  Substantially Met = 4, and Met = 5. Then, we took the average of the  individual assessment ratings to determine the overall rating for each of  the four characteristics. To perform this analysis, we obtained and  analyzed program resource estimate supporting documentation, including  service cost positions, technical descriptions, work breakdown structures,  technology readiness assessments, program schedules, and AOA reports.  We also interviewed program and cost estimating officials, when  necessary, to gather additional information on these resource estimates  and the cost models used to produce them. Each program was also  provided with a copy of our assessment of their resource estimates for  review and comment.
To determine the extent to which the JROC prioritized requirements and  capability gaps, we reviewed the 13 ICDs and 7 CDDs submitted to the  JROC and any discussions of priorities and redundancies contained in  each document. We also met with officials from the Joint Staff;  Department of the Air Force; Department of the Army; Department of the  Navy; and Office of the Under Secretary of Defense (Comptroller) to  discuss the extent to which the JROC and its supporting bodies have  addressed prioritization issues. To understand the Joint Staff’s ongoing  internal JCIDS review, we assessed the review charter and met with the  Joint Staff officials managing the review to discuss the recommendations  from the review and how they might affect the JROC’s prioritization of  requirements. We also observed joint requirements meetings and reviewed  prior GAO reports.
We conducted this performance audit from June 2010 to June 2011 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
section_title: Appendix II: Requirements Documents Reviewed by the JCB or JROC in Fiscal Year 2010
paragraphs: In fiscal year 2010, the Joint Capabilities Boards (JCB) and Joint  Requirements Oversight Council (JROC) combined to review a total of 45  new requirements documents, including 11 that were classified, 2 that  were information technology programs, and 8 documents that were not  associated with major defense acquisition programs. The remaining 24  requirements documents are identified in figure 4.
section_title: Appendix III: JROC Cost Breach Reviews in Fiscal Year 2010
paragraphs: The Joint Requirements Oversight Council (JROC) conducted 15 reviews  following cost breaches in fiscal year 2010—6 Nunn-McCurdy reviews and  9 tripwire reviews. Table 2 identifies these reviews.
section_title: Appendix IV: Best Practice Criteria for Resource Estimates
paragraphs: Table 3 below presents the best practice criteria against which we  assessed the resource estimates presented to the Joint Requirements  Oversight Council during fiscal year 2010 Capability Development  Document (CDD) reviews.
section_title: Appendix V: Comments from the Department of Defense
section_title: Appendix VI: GAO Contact and Acknowledgments
subsections: section_title: Acknowledgments
paragraphs: In addition to the contact named above, Ronald E. Schwenn, Assistant  Director; Noah B. Bleicher; Stephen V. Marchesani; Kenneth E. Patton;  Karen A. Richey; Anna K. Russell; and Nathan A. Tranquilli made key  contributions to this report.