Defense Management: Guidance and Progress Measures Are Needed to Realize Benefits from Changes in DOD's Joint Requirements Process
Published: 2012-02-24T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Defenses Joint Requirements Oversight Council (JROC) is charged with assisting in the prioritization of capability needs from a joint perspective and helping guide investments. The JROC is supported by the Joint Capabilities Integration and Development System (JCIDS) process. However, a congressional committee and GAO have expressed concerns about the extent to which JCIDS has been effective in prioritizing capability needs. The Ike Skelton National Defense Authorization Act for Fiscal Year 2011 required GAO to provide a report on the effectiveness of JCIDS in several areas. In addition to responding to this direction, GAO has more broadly evaluated the extent to which (1) the Joint Staff has developed and implemented an analytic approach to prioritize capability needs and (2) the JROC has considered aspects of the availability and operational support of weapon systemscalled sustainmentwhen validating the requirements of proposed capability solutions. To do so, GAO analyzed capability documents, reviewed relevant guidance and law, and interviewed officials.
section_title: What GAO Found
paragraphs: After studying the Joint Capabilities Integration and Development System (JCIDS) process since September 2010, the Joint Staff began initiating actions in October 2011 to better prioritize capability needs and align those needs with available budgetary resources. Specifically, according to Joint Staff officials, the Joint Requirements Oversight Council (JROC) has begun to consider the benefits and affordability of new capabilities within the context of joint capability areas and to evaluate possible duplication before validating new capability requirements. The Joint Staff has begun to implement a new approach to support JROC prioritization of capability needs, but the new approach is still evolving and has not been fully developed and clearly documented. New guidance does not clearly outline goals of the new approach, develop and communicate the analytic approach envisioned to support JROC decision making, or set out criteria and accompanying measures of progress. GAO previously reported that JCIDSs ability to prioritize needs could be improved if it had an analytic approach to reviewing and validating proposals that would help ensure that the most important capability needs of the department are addressed. Until the Joint Staff takes steps to fully develop, document, and institutionalize the new analytic approach, it is not clear whether the current momentum for improving the JCIDS process will be sustained.
JCIDS guidance in effect through December 2011 required that sponsors of potential major defense acquisition programs address sustainment information in capability development documents according to four metricsmateriel availability, operational availability, reliability, and ownership cost. Each of these metrics includes a set of potentially reportable criteria or data, which are listed as review criteria and are suggested, but not clearly required by the guidance, to be included in the metric. Based on GAOs analysis of six capability development documents, GAO found that all of the documents provided information on the four required sustainment metrics, but the completeness of information for all of the metrics key criteria varied. Further, in some cases information that should have been included, according to Department of Defense officials, was not provided. The Joint Staff issued updated JCIDS guidance in January 2012, but the guidance still does not clearly require program sponsors to report on the individual criteria for each of the four sustainment metrics. Without complete and detailed information on each of the individual criteria elements, the JROC may not have the information it needs to make the most informed decisions when validating the requirements of proposed solutions intended to mitigate capability gaps.
section_title: What GAO Recommends
paragraphs: GAO recommends that the Vice Chairman of the Joint Chiefs of Staff (1) revise and implement JCIDS guidance to reflect recent changes to the process and establish criteria and measures for determining the relative importance of capability needs and (2) require program sponsors to address each criterion in JCIDS guidance related to sustainment in capability documents. DOD partially concurred with GAOs recommendations.

section_title: Letter
paragraphs: Like the rest of the federal government, the Department of Defense  (DOD) is operating in a constrained budget environment and facing  difficult decisions about how to invest its resources to meet national  security objectives. The Joint Requirements Oversight Council (JROC)  directly and indirectly supports the Chairman of the Joint Chiefs of Staff’s  role as the principal military advisor to the President and other senior  national security officials. The JROC consists of the Vice Chairman of the  Joint Chiefs of Staff and other senior military officials and, among other  duties, is expected to assist in the prioritization of joint military  requirements and ensure that resource levels needed to fulfill those  requirements are consistent with the level of priority. These duties can  help ensure that proposed weapon systems are prioritized to create an  affordable portfolio. The JROC uses the Joint Capabilities Integration and  Development System (JCIDS), a process created in 2003 to guide the  development of capabilities from a joint perspective, to help it identify  capability gaps and validate the requirements of proposed capability  solutions to mitigate those gaps. JCIDS operates in conjunction with two  other DOD processes—the acquisition process, which facilitates the  development and acquisition of proposed capabilities, and the budgeting  process, through which these capabilities are funded. The development  and acquisition of weapon systems consume a substantial portion of  DOD’s budget. From fiscal years 2011 through 2016, DOD plans to spend  almost $390 billion on its current portfolio of major defense acquisition  programs. Moreover, DOD spends billions of dollars each year to sustain  these weapon systems. As we have reported, costs to operate and  support these systems make up at least 70 percent of a system’s life  cycle costs. The House Armed Services Committee and GAO have  expressed concerns about the extent to which JCIDS has been effective  in assigning priorities to capabilities proposed to meet the needs of the  joint force.
A provision in the Ike Skelton National Defense Authorization Act for  Fiscal Year 2011 required GAO to provide a report on the effectiveness of  JCIDS in several areas, such as prioritizing joint requirements and  considering information on costs of sustaining future programs. In  appendix I, we have reprinted the elements of the provision to be covered  by the report and our response to each one. As part of our analysis, we  also evaluated (1) the extent to which the Joint Staff has developed and  implemented an analytic approach to prioritize capability needs and   (2) the extent to which the JROC has considered aspects of the  availability and operational support requirements of weapon systems— called sustainment—when validating the requirements of proposed  capability solutions.
To assess the extent to which the Joint Staff has developed and  implemented an analytic approach to prioritize capability needs, we  reviewed relevant law and Joint Staff policy documents and related  guidance on the roles and responsibilities of the JCIDS process and the  JROC as it pertains to prioritization. We interviewed Joint Staff officials to  discuss changes to its approach for managing requirements within  capability portfolios and making recommendations for trade-offs among  alternatives. We also reviewed prior GAO reports that discussed  capability development and requirements prioritization and compared the  current efforts to minimize risks and identify unnecessary overlap and  duplication with what we reported in the past. To assess the extent to  which the JROC has considered aspects of the availability and  operational support requirements of weapon systems—called  sustainment—when validating the requirements of proposed capability  solutions, we reviewed DOD and Joint Staff policy documents and related  guidance on the requirement to develop sustainment metrics for capability  documents for certain programs processed through JCIDS. We  interviewed DOD and Joint Staff officials to discuss the preparation,  presentation, and consideration of sustainment data. We also selected a  nonprobability sample of six requirements documents for programs that  have been required to report sustainment metrics to determine whether  the documents contained the required metrics and the supporting  information included with the metrics.
We conducted this performance audit from April 2011 through February  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. Additional details on our  scope and methodology are in appendix II.
subsections: section_title: Background
paragraphs: JCIDS; the Planning, Programming, Budgeting and Execution process;  and the Defense Acquisition System broadly make up DOD’s overall  defense acquisition management framework. JCIDS was implemented in  2003 to guide future defense programs from a joint capabilities  perspective. JCIDS is one of the first steps in DOD’s acquisition  processes; JCIDS participants work to identify and determine whether to  validate the need for capabilities proposed by the services, the defense  agencies, and the combatant commands. Once a requirement is  validated, the services rely on the DOD’s Planning, Programming,  Budgeting and Execution process, through which DOD allocates financial  resources across the department—including the services—to identify  funding for validated capability solutions. DOD then manages the  development and procurement of proposed capabilities through the  Defense Acquisition System.
DOD implemented the JCIDS process in 2003 in an effort to assist the  JROC by changing DOD’s requirements validation process from a  service-specific perspective to a joint capabilities perspective. The JROC,  which is chaired by the Vice Chairman of the Joint Chiefs of Staff,  consists of a general or admiral from each of the military services and  may include combatant commanders or deputy commanders when  directed by the JROC Chairman. The JROC is charged with assisting the  Chairman of the Joint Chiefs of Staff with a number of tasks, including   (1) identifying, assessing, and approving joint military requirements to  meet the national military strategy; (2) establishing and assigning priority  levels for joint military requirements; and (3) reviewing the estimated level  of resources required to fulfill each joint military requirement and ensuring  that the resource level is consistent with the requirement’s priority, among  others. The JROC also assists acquisition officials in identifying  alternatives to any acquisition programs that meet joint military  requirements for the purposes of certain statutory provisions, addressing  matters such as cost growth. In 2009, the Weapon Systems Acquisition  Reform Act expanded the role of the JROC by directing it to assist the  Chairman of the Joint Chiefs of Staff in ensuring that trade-offs among  cost, schedule, and performance objectives are considered for joint  military requirements and establishing an objective for the overall period  of time within which an initial operational capability should be delivered to  meet each joint military requirement.
The JROC reviews requirements for programs designated as JROC- interest based on their expected cost and complexity and, under guidance in effect through December 2011, also reviewed programs at  the request of certain senior DOD officials. Within JCIDS, the JROC is  supported in its duty to review and validate joint capability needs by the  Joint Capabilities Board and six Functional Capabilities Boards. The  Joint Capabilities Board is chaired by the Director of the Joint Staff’s  Directorate for Force Structure, Resources, and Assessment, and each  Functional Capabilities Board is chaired by a general/flag officer or civilian  equivalent. The Joint Capabilities Board reviews capability documents  before they are passed on to the JROC for its review and also serves as  the validation authority for certain programs that do not reach JROC- interest thresholds, although the JROC may review any JCIDS document  or other issues requiring joint resolution. Functional Capabilities Boards  are responsible for reviewing proposed requirements specific to joint  capability areas, such as protection, logistics, or battlespace awareness.
In JCIDS, the JROC and its supporting organizations review requirements  documents related to capability gaps and the major defense acquisition  programs intended to fill those gaps prior to key acquisition milestones.  These requirements documents—initial capabilities documents, capability  development documents, and capability production documents for  materiel solutions and change recommendations for nonmateriel  solutions—are submitted into the JCIDS process by capability sponsors.The initial capabilities document identifies a specific capability gap, or set  of gaps, and if a materiel solution is required, helps inform the initial  stages of the acquisition process, which include an analysis of the  alternative solutions to fulfilling the capability need and the selection of a  preferred system concept. When the technology development phase of  the acquisition process is complete, a program sponsor completes a  capability development document that includes more detail on the desired  capabilities of the proposed system and defines the system’s key  performance parameters or attributes against which the delivered  increment of capability will be measured. Finally, the sponsor prepares a  capability production document to describe the actual performance of the  system that will deliver the required capability. Figure 1 depicts how  JCIDS reviews align with the acquisition process.
The House Armed Services Committee and a panel established by the  committee have discussed long-standing challenges with the JCIDS  process and the JROC’s fulfillment of its statutory responsibilities. The  House Armed Services Committee, in a report accompanying a bill for the  National Defense Authorization Act for Fiscal Year 2008, described a  legislative provision that would allow for joint decision making as opposed  to service-centric budget considerations by incorporating clear priorities  and budget guidance into the JROC process. In 2009, the House Armed  Services Committee established a panel on defense acquisition reform  because of a sense that the acquisition system was not responsive  enough for today’s needs, not rigorous enough in protecting taxpayers,  and not disciplined enough in the acquisition of weapon systems for  tomorrow’s wars. The panel received testimony that the Joint Staff lacked  some of the analytical expertise necessary to ensure that the JCIDS  process rigorously vets proposed requirements. Additionally, since 2008  we have reported on these challenges. We reported in 2008 that the  JCIDS process was not effective in prioritizing capability gaps, and we  noted that capability needs continued to be proposed and defined by the  services with little involvement from the joint community. We  recommended that the Secretary of Defense direct the Chairman of the  Joint Chiefs of Staff to develop an analytic approach within JCIDS to  better prioritize and balance the capability needs of the military services,  combatant commands, and other defense components. DOD partially  agreed with our recommendation but did not fully implement it, and  prioritization remains service driven. More recently, in June 2011 we  reported that the JROC did not always consider trade-offs among cost,  schedule, and performance objectives; prioritize requirements; consider  redundancies across proposed programs; or prioritize and analyze  capability gaps in a consistent manner. We recommended that the  JROC require higher-quality resource estimates from requirements  sponsors to ensure that best practices are being followed, provide a  sound basis to ensure that trade-offs are considered, prioritize  requirements across proposed programs, and address potential  redundancies during requirements reviews, among other steps. DOD  partially agreed with our recommendations and commented that  improvements to the quality of resource estimates would be addressed in  upcoming changes to the JCIDS process.
In May 2011, we also reported that combatant command officials raised  concerns that JCIDS focuses more on long-term service-centric capability  gaps than on combatant commands’ more immediate and largely joint  gaps. JCIDS was designed as a deliberate process to meet longer-term  joint needs. To address urgent needs, DOD established a separate  process—the joint urgent operational needs process—in 2005. The joint  urgent operational needs process was intended to respond to needs  associated with combat operations in Afghanistan and Iraq and the War  on Terror. The revised JCIDS guidance canceled separate guidance for  joint urgent operational needs and incorporates and describes the joint  urgent operational needs process. Urgent operational needs, as defined  by the new JCIDS guidance, are capability requirements needed for  ongoing or anticipated contingency operations that if left unfulfilled could  potentially result in loss of life or critical mission failure. For this report,  we focus on requirements that have not been identified as urgent and  instead follow the deliberative JCIDS process.
section_title: The Joint Staff Has Initiated Efforts to Prioritize Capability Needs, but Implementation Processes Are Not Fully Developed and Clearly Documented
paragraphs: The Joint Staff is undertaking efforts to improve the ability to prioritize  capability needs from a joint perspective through JCIDS and to align  those needs with available budget resources. However, implementation  processes for JCIDS’s new approach to managing requirements and  considering affordability are still evolving and have not been fully  developed and clearly documented.
subsections: section_title: Changes in JCIDS Processes Are Intended to Improve Prioritization of Capability Needs and Guide Future Investments
paragraphs: Determining priorities among joint requirements has been a responsibility  of the JROC since Congress amended section 181 of Title 10 of the U.S.  Code in 2008 to require the JROC to assist in establishing and assigning  priority levels for joint military requirements and to help ensure that  resource levels associated with those requirements are consistent with  the level of priority. DOD officials acknowledge that JCIDS has been  ineffective in helping the JROC carry out these responsibilities. We have  previously reported that JCIDS’s ability to align resources to balance  competing needs could be improved if it had an analytic approach that  provided a means to review and validate proposals to ensure that the  most important capability needs of the department are being addressed.  We further said that such an approach should establish criteria and  measures for identifying capability gaps and determining the relative  importance of capability needs. Finally, the approach should result in  measurable progress in allocating resources in order to eliminate  redundancies, gain efficiencies, and achieve a balanced mix of  executable programs.
DOD officials told us that downward pressure on the defense budget has  led the Joint Staff to change how the JCIDS process is used to strengthen  its ability to support JROC members in making trade-off decisions among  requirements and balancing risks across the force within expected  resources. In fall 2011, according to officials, the incoming Vice Chairman  of the Joint Chiefs of Staff, as the Chairman of the JROC, began to make  changes in the JCIDS processes to focus on what capabilities currently  exist and weigh the benefits of investing in new capabilities with their  estimated costs early in the review process. The Joint Staff issued draft  guidance in October 2011 and began implementation based on the draft  guidance. The Joint Staff issued the final guidance in January 2012.  Descriptions of these changes and their implementation follow:
New capabilities will be considered as part of a “capability  portfolio approach.” Under the portfolio approach, officials stated  that JROC members are to ensure that proposed investments in  capabilities address joint needs or they will not be validated to  proceed to the acquisition process. In addition to validating capability  proposals, according to Joint Staff officials, the JROC has begun  examining how the services’ existing programs can support joint  operations to reduce duplication of capabilities. As of December 2011,  according to Joint Staff officials, the last five closed meetings of the  JROC began conversations about how to meet requirements by  considering available capabilities and the costs and benefits of  proposed programs. As a result, according to Joint Staff officials, at  least five classified programs have been reviewed and altered by  either comparing redundant capabilities, reducing capacity, adjusting  delivery schedules, or directing follow-on analysis before moving  programs forward. To support JROC decision making, officials  reported that Functional Capabilities Boards will be tasked with  examining requirements, their associated capability gaps, and  proposed solutions within their capability portfolios, and independently  assessing how a proposed capability fits into its corresponding joint  capability area. The Functional Capabilities Boards previously had  responsibility for identifying, assessing, and prioritizing (if required)  joint capability needs proposals within assigned joint capability areas  but, according to officials, have not always carried out these  responsibilities. Previously, Functional Capabilities Boards acted  primarily as technical reviewers of requirements documents, and  program sponsors briefed the JROC on the attributes of the program.  However, the new guidance does not specify how the independent  assessment is to be conducted, and it is too soon to tell how the  Functional Capabilities Boards will respond to the new requirement.  Officials reported that Functional Capabilities Boards are expected to  develop methodologies on a case-by-case basis. DOD officials said  that the analytic approach and uses of analytic information will evolve  over time.
DOD has previously attempted to manage capabilities  departmentwide through a portfolio approach, but has never fully  implemented the approach. In 2006, DOD established an effort to  manage resources across capability areas by establishing capability  portfolio managers to enable DOD to develop and manage  capabilities across the department rather than by military service or  individual program, and by doing so, to improve the interoperability of  future capabilities, minimize capability redundancies and gaps, and  maximize capability effectiveness. However, as we reported in 2008,  capability portfolio managers make recommendations on capability  development issues within their portfolios but do not have  independent decision-making authority. In 2011, Joint Staff officials  told us that they were unaware of capability portfolio managers’ active  involvement in the JCIDS process.
Attendance at JROC meetings will be limited to key decision  makers and stakeholders. Beginning in October 2011, the JROC  Chairman began to limit attendance at JROC meetings to facilitate  candid discussion among senior leaders about priorities for joint  requirements and alternative solutions. Joint Staff officials told us that  previously, meetings were open to a broad range of interested parties  and service sponsors provided briefings on their proposals for new  capabilities. Under the new approach being implemented, a  representative of one of the Functional Capabilities Boards will  provide a briefing on the proposal to the JROC. The representatives  would then present the board’s independent assessment of benefits,  costs, and risk for the JROC to discuss and decide upon. According to  JCIDS officials, JROC members are expected to make decisions from  the perspective of the joint force and avoid taking a service-centric  approach.
Affordability of proposals will be a primary factor in validation  decisions. JROC members have always been expected to consider  the resource implications of validation decisions, but officials stated  that until recently, these considerations have not been a focus  because capabilities were not competing with each other for funding.  According to officials, the Functional Capabilities Boards have been  directed to take similar steps to ensure that capability proposals not  only meet technical requirements but also represent the most efficient  alternative for providing a capability within the joint capability area  without creating duplication or overlap. According to JCIDS officials,  the JROC is also reconsidering previous validation decisions and  asking for changes to proposals to minimize costs. We reviewed  recent Functional Capabilities Board briefings to the JROC, which  provided information on how needs might be met with current  capabilities and alternatives that might meet needs while minimizing  costs. However, it is too soon to assess how the JROC will consider  affordability of programs when making validation decisions.
section_title: New JCIDS Processes Are Not Fully Developed and Clearly Documented
paragraphs: The Vice Chairman of the Joint Chiefs of Staff has sought some policy  changes and, according to officials, provided other direction to implement  changes in the JCIDS process, but the new approach to managing  requirements and considering affordability is still evolving and has not  been fully developed and clearly documented. According to Joint Staff  officials, the dynamic fiscal environment and the evolutionary method  being used to develop the new approach and implementation processes  make it important that decision makers maintain flexibility in decision  making. We believe that the new approach has promise in positioning the  JROC to more accurately identify capability gaps and trade-offs, but it has  not been fully developed to include steps to ensure that the approach is  fully implemented, that the intent is fully communicated to all stakeholders  involved, and that the results of the new approach will be measurable.
We have previously reported that key practices for results-oriented  management involve leadership from top officials as well as the  involvement of stakeholders at all levels throughout a period of  transition. We have also reported that in order to demonstrate a  successful results-oriented framework, officials must include clearly  defined measures to assess intended outcomes. As shown in table 1,  key practices for supporting change should include, among other actions,  obtaining and sustaining support from senior leadership to facilitate the  transformation, establishing clear lines of communication between all  affected parties, and demonstrating value and credibility of new  processes through the use of metrics.
According to Joint Staff officials, the Chairman of the JROC has been a  driving force in positioning the JROC to take on the responsibility of  aligning needs and balancing risk and resources, which fulfills one of the  key steps for results-oriented management, but the approach is still new  and officials have not completed all of the steps that facilitate institutional  acceptance and implementation of the new approach. The Joint Staff has  begun the process of change by articulating a clear rationale for  change—that the JROC can more effectively represent the warfighters’  requirements and make strategic trade-off decisions as budgets stay flat  or decrease by taking a more active role in shaping an affordable joint  force. However, best practices for managing a results-oriented change  state that goals and procedures should be communicated to stakeholders  throughout the organization so that they understand how they should  implement the new approach and how the organization will measure  progress. The Joint Staff issued guidance that outlines new procedures  intended to establish an approach for prioritizing capabilities from a joint  perspective and to increase the timeliness of JCIDS reviews by  categorizing proposals according to the level of urgency of the need and  streamlining procedures for urgent needs. However, the guidance does  not clearly outline criteria and measures for demonstrating progress  toward meeting the goal of aligning needs with available resources or  clearly communicate the goals and the analytic approach envisioned to  support JROC decision making. Further, the guidance does not describe  how the proposed change will affect the services, combatant commands,  and other stakeholders. Finally, the JCIDS guidance does not establish  criteria and measures for demonstrating progress toward the goal of  creating a balanced portfolio of programs that takes into account needs,  risks, and available resources, nor do other documents provided to us by  DOD. Measures such as the proportion of requirements that address joint  priority needs versus service-specific needs, the savings obtained  through the elimination of redundant capabilities, and the comparison of  estimated costs of a proposed program with the actual costs of operating  the program over its life cycle could be helpful in assessing whether the  process is balancing requirements with available resources or whether  further adjustments to the process are needed.
Considering new capabilities across the department in the context of joint  capability areas can help DOD begin to identify priorities for future  investment. However, unless the Joint Staff takes steps to define and  institutionalize the new approach by adhering to the key principles of  results-oriented management, it is not clear whether the current  momentum in implementing an analytic approach through JCIDS will be  sustained.
section_title: JROC Has Not Always Had Complete Information on Future Costs to Sustain Proposed Capabilities before Approving Documentation for Program Development
paragraphs: Even though sustainment costs make up a significant portion of the total  ownership costs of a weapon system, the JROC has not always had  complete information on such costs when validating documentation used  in the decision to initiate program development. “Sustainment” as a  category represents a range of activities intended to ensure the  availability of weapon systems and to support their operations, including  some logistics and personnel services. During the identification of  capability gaps and consideration of selected alternatives, it is difficult for  the sponsors to provide detailed information on program capabilities and  cost estimates. As a major defense acquisition program moves toward  the development stage, JCIDS requires that more complete and accurate  sustainment information be presented in capability development  documents. According to DOD officials, decision makers need more  accurate cost information to assess whether the benefit of a proposed  capability justifies the cost of sustaining the capability over its life.  JCIDS guidance requires that sponsors of potential major defense  acquisition programs include sustainment information in capability  development documents, which detail proposed solutions to fulfill  capability needs.
The JROC has generally relied on sponsor-provided assessments of  sustainment information in the capability development documents to  make its validation decisions, but these documents have not always  included all the information suggested in JCIDS guidance or sufficient  detail to enable the JROC to assess the quality of the information. A DOD   regarding the development of sustainment information suggests  manual that when sustainment requirements and underlying assumptions are not  clearly documented, subsequent decisions about the project may be  based on incorrect assumptions. Prior GAO workjoint warfighting capabilities and proposals to fulfill the gaps should be  clearly identified to decision-making bodies, such as the JROC, to inform  deliberations. Further, information should be complete so those making  the important decisions may do so as effectively as possible.   suggests that gaps in  JCIDS guidance requires sponsors of major defense acquisition programs  to address sustainment based on four metrics—materiel availability,  operational availability, reliability, and ownership cost (renamed operation  and support cost in January 2012). The guidance includes a series of  criteria, which are listed as review criteria in the guidance. The criteria  provide additional information on each metric. For example, JCIDS  guidance lists as review criteria for the materiel availability metric whether  there is a clear definition and accounting for the intended service life of  the program, an identification of planned downtime, and a comparison of  downtime value with experiences of other analogous systems, among  other criteria. Table 2 outlines examples of key review criteria within each  of the four sustainment metrics under the guidance in effect through  December 2011.
Program sponsors provide initial information on the sustainment metrics  for proposed capability solutions when they submit a capability  development document, one of three capability documents the JROC  considers in its review and potential validation of capability proposals.  Officials from both the Joint Staff and the Office of the Under Secretary of  Defense for Acquisition, Technology and Logistics review the capability  development documents and, according to officials, verify that all required  sustainment elements have been included before the documents are  validated by the JROC. Officials also provide their independent  assessments of the quality of the cost estimates to the JROC.from the Office of the Under Secretary of Defense for Acquisition,  Technology and Logistics said they work with program sponsors to  ensure that life cycle sustainment planning and costs are as accurate as  possible. Officials from the Office of the Under Secretary of Defense for  Acquisition, Technology and Logistics and the Joint Staff told us that they  consider reported sustainment information important to a program’s  development and review all reported information.
The JCIDS Manual notes that listed criteria, information, and activities  cannot necessarily be applied to all systems. Sponsors have a degree of  latitude in determining which items are applicable for their specific  concept, technology, system, or combination of these. For example, a  program sponsor for a major defense acquisition program is required to  report a measure for operational availability, but would not necessarily  have to report on the respective criteria, such as addressing downtime  associated with failure, including recovery time or movement of  maintenance teams to the work site. Because the guidance does not  specifically require program sponsors to report on the individual criteria,  they generally include some, but not all, of the individual criteria.
Our analysis of six capability development documents found that all of the  documents provided information on all of the required sustainment  metrics. However, we found that the completeness of information  reported for all of the metrics’ key criteria varied. Specifically, none of the  documents included complete information for each of the four  sustainment metrics’ review criteria elements. In addition, each of the  documents had some common omissions; for example, none of the six  capability documents we reviewed included information on all nine of the  ownership cost metric’s criteria elements. Further, several of these  documents only included information on one criteria element for a single  metric, and none reported information on all of the elements for any of the  metrics. Finally, when information on the metrics’ key criteria was  provided, the level of detail varied among the documents. For example,  for some criteria, some documents provided a paragraph of supporting  information and analysis whereas others provided single-sentence  responses that did not provide as substantial detail for the supporting  criteria. According to officials, for the capability documents we reviewed,  there were cases in which information on some of the suggested criteria  should have been included but were not.
Joint Staff officials and officials from the Office of the Under Secretary of  Defense for Acquisition, Technology and Logistics noted that while criteria  for each of the required sustainment metrics may not be applicable to  every program, it would be beneficial to the JROC if the services reported  on the criteria for each metric outlined in the guidance or indicated a  reason why a specific criterion was not applicable. Officials from the  Office of the Under Secretary of Defense for Acquisition, Technology and  Logistics told us that when they conduct their reviews of sponsor-reported  sustainment information, not all of the supporting documentation they  need for a thorough independent assessment is available in the capability  development document. These officials said that they can generally find  detailed documentation on sustainment planning and costs from sources  outside of the JCIDS process, but that the information is not always  readily available within the JCIDS database. Ultimately, review efficiency  could be improved if all the information were available in the JCIDS  database. Updated JCIDS guidance issued in January 2012 still does not  clearly require program sponsors to report on the individual criteria for  each of the sustainment metrics. The Joint Staff is developing a new  reporting tool intended to provide a standard format for reporting  sustainment information. The tool will require program sponsors to at  least minimally address each of the four sustainment metrics in order to  submit the capability development document for review through JCIDS,  according to officials. However, officials stated that this tool will not  require that sponsors address each of the individual criteria elements  within the four sustainment metrics.
Without complete and detailed information on each of the individual  criteria elements, the JROC may not be in the best position to weigh the  costs and benefits of a proposal within a capability portfolio. The quality  and completeness of the data that sponsors provide through the  capability development document in the JCIDS process will become more  important as the JROC increases its examination of the benefits of  programs balanced against their associated costs. As we have previously  reported, incomplete and inaccurate sustainment information has been a  long-standing problem for DOD. In November 2009, a DOD team  assessing weapon system acquisition reform reported that DOD lacked  valid, measurable sustainment information to accurately assess how  programmatic decisions affected life cycle costs and made  recommendations to improve weapon system life cycle sustainment.Until the JROC requires program sponsors to report complete  sustainment information, including both the overall metrics and the  supporting criteria, the JROC may not always have the complete and  detailed information it needs to make the most informed decisions.
section_title: Conclusions
paragraphs: The prospect of declining budgets has amplified the need for DOD to  prioritize among capability gaps and to use its resources to maximize the  capabilities of the joint force. The Chairman of the JROC has begun to  take steps to better balance risks across the joint force by examining  proposals for new capabilities within the context of existing joint capability  areas and to consider affordability, including sustainment costs, as a  factor in validating requirements proposals. The revised approach is new  and evolving, but in order for it to achieve the intended results of  prioritizing capability needs and aligning those needs with available  resources, the Joint Staff needs to take steps to fully develop the  approach and document it more explicitly. Specifically, DOD does not yet  have a documented implementation plan with measures of success that  support change. In addition, having good sustainment information is a key  element needed to improve JCIDS’s success over the long term.  Sustainment costs historically represent 70 percent of a system’s life  cycle costs, but DOD has been making decisions with incomplete  information on sustainment and does not require that sponsors address  all of the criteria outlined in JCIDS guidance. Until the JROC has  developed and fully documented an approach for prioritizing capability  needs and aligning these needs with available resources and has  complete sustainment information associated with the operation of new  capabilities, it will not be in the best position to align resources with  priorities or balance costs with benefits in affordable investment plans.
section_title: Recommendations for Executive Action
paragraphs: To help sustain momentum for efforts to bring a capability portfolio  approach to the JCIDS process and to improve the quality of sustainment  information reported in capability development documents, we  recommend that the Vice Chairman of the Joint Chiefs of Staff, as the  JROC Chairman, take the following two actions:
Revise and implement guidance to reflect changes to the JCIDS  process as well as to establish criteria and measures for determining  the relative importance of capability needs across capability areas and  assessing progress.
Explicitly require that program sponsors address each of the criteria  outlined for the individual sustainment metrics when submitting  capability development documents.
section_title: Agency Comments and Our Evaluation
paragraphs: The Joint Staff provided written comments on a draft of this report. In its  comments, the Joint Staff said our report represented a fair and objective  assessment of the JCIDS process. It partially agreed with both of our  recommendations, citing ongoing and planned changes to the joint  requirements development process. However, the comments did not  detail any specific steps that DOD plans to take to address our  recommendations. The comments are reprinted in their entirety in  appendix III. The Joint Staff also provided technical comments, which we  have incorporated into the report as appropriate.
The Joint Staff partially concurred with our recommendation that the Vice  Chairman of the Joint Chiefs of Staff revise and implement guidance to  reflect changes to the JCIDS process as well as to establish criteria and  measures for determining the relative importance of capability needs  across capability areas and assessing progress. In its written response,  the Joint Staff described recent initiatives to substantially change the joint  requirements development process to require that capability requirements  be evaluated within a capability portfolio by Functional Capability Boards,  the Joint Capabilities Board, and the JROC. The Joint Staff also  discussed its planned efforts to improve prioritization of capability needs  and stated that JROC reviews will incorporate an evolving portfolio  assessment tool. The Joint Staff expects that the departmentwide  priorities outlined in DOD’s strategic guidance as well as a revised  process for assessing capability gaps and combatant command priorities  will enable the JROC to make more informed decisions about priorities.  While we agree that the Joint Staff has taken important steps to enable  prioritization of capabilities, such as addressing prioritization in a new  enclosure in its revised JCIDS Manual, the enclosure does not explicitly  outline implementation processes. We continue to believe that clear  guidance that establishes criteria for determining priority levels and  measures for demonstrating progress will be essential in sustaining  momentum toward the goal of creating a balanced portfolio of programs  that takes into account needs, risks, and available resources. Moreover,  providing guidance that fully documents the new procedures for assigning  priority levels to capability gaps is an essential step toward clarifying how  the procedures will be implemented.
The Joint Staff also partially concurred with our recommendation that the  Vice Chairman of the Joint Chiefs of Staff explicitly require that program  sponsors address each of the criteria outlined for the individual  sustainment metrics when submitting capability development documents.  According to its written comments, the Joint Staff criteria for the  sustainment metrics were designed to guide the development of  requirements, but were not intended to be prescriptive because individual  programs are unique and criteria applicable to one problem may not apply  to another. We agree that each of the criteria may not be applicable to  every program. However, if program sponsors addressed each criterion in  some manner, including explaining that a criterion is not applicable to the  program, the JROC would be assured that program sponsors considered  all criteria when developing program proposals. Further, the Joint Staff  commented that JCIDS reviews of capability development documents by  Joint Staff and Office of the Secretary of Defense officials ensure that a  document is thoroughly vetted for consideration by the JROC. It  suggested that the inclusion of analyses and rationale for sustainment  metrics development in capability development documents might be  duplicative because this information is contained in acquisition documents  that exist outside of JCIDS. However, as we noted in our report, the  documents that contain the analysis and rationale for the required  sustainment metrics are not necessarily reviewed by or available to the  JROC members during their consideration of a capability development  document. We continue to believe that the inclusion of a sponsor- provided rationale for each metric criterion would enhance the  thoroughness and efficiency of the JROC’s review of sustainment  information through JCIDS.
We are sending copies of this report to interested congressional  committees, the Secretary of Defense, the Vice Chairman of the Joint  Chiefs of Staff, and other interested parties. In addition, the report is  available at no charge on the GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-3489 or pendletonj@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix IV.
section_title: Appendix I: Observations on Reporting Provisions in the National Defense Authorization Act for Fiscal Year 2011
paragraphs: Section 862 of the Ike Skelton National Defense Authorization Act for  Fiscal Year 2011 requires the Comptroller General to carry out a  comprehensive review of the Joint Capabilities Integration and  Development System (JCIDS) and to submit to the congressional defense  committees a report on the review. This appendix, in conjunction with the  letter, addresses each of the reporting provisions as described in the act.  Specifically, section 862 requires the following contents for the review:  Purpose. The purpose of the review is to evaluate the effectiveness of  JCIDS in achieving the following objectives:
Timeliness in delivering capability to the warfighter.
Efficient use of the investment resources of the Department of  Defense (DOD).
Control of requirements creep.
Responsiveness to changes occurring after the approval of a  requirements document environment, the emergence of new capabilities, or changes in the  resources estimated to procure or sustain a capability).   (including changes to the threat
Development of the personnel skills, capacity, and training needed for  an effective and efficient requirements process.
Matters considered. In performing the review, the Comptroller General is  required to gather information on and consider the following matters:
The time that requirements documents take to receive approval  through JCIDS.
The quality of cost information considered in JCIDS and the extent of  its consideration.
The term requirements document is defined in the statute as “a document produced in  JCIDS that is provided for an acquisition program to guide the subsequent development,  production, and testing of the program and that—(A) justifies the need for a materiel  approach, or an approach that is a combination of materiel and non-materiel, to satisfy  one or more specific capability gaps; (B) details the information necessary to develop an  increment of militarily useful, logistically supportable, and technically mature capability,  including key performance parameters; or (C) identifies production attributes required for a  single increment of a program.” See § 862(c)(2).
The extent to which JCIDS establishes a meaningful level of priority  for requirements.
The extent to which JCIDS is considering trade-offs between cost,  schedule, and performance objectives.
The quality of information on sustainment considered in JCIDS and  the extent to which sustainment information is considered.
An evaluation of the advantages and disadvantages of designating a  commander of a unified combatant command for each requirements  document for which the Joint Requirements Oversight Council (JROC)  is the validation authority to provide a joint evaluation task force to  participate in a materiel solution and to provide input to the analysis of alternatives; participate in testing  (including limited user tests and prototype testing);provide input on  a concept of operations and doctrine; provide end user feedback  to the resource sponsor; and participate, through the combatant  commander concerned, in any alteration of the requirement for  such solution.
Section 862 also provided definitions for JCIDS, requirements document,  requirements creep, and materiel solution.
Tables 3 through 12 contain our response to each of the requirements  mandated by the Ike Skelton National Defense Authorization Act for  Fiscal Year 2011. The provision mandating our report defined the JCIDS  process by referring to the JCIDS guidance in effect from March 2009  through December 2011. Accordingly, our response to the mandated  elements as presented in this appendix generally focuses on JCIDS  under that guidance. However, we also provide some information relating  to JCIDS as described in the revised guidance issued in January 2012. In  addition, our assessments generally focused on those programs that  were determined to be JROC-interest, or those that were designated as  major defense acquisition programs or major automated information  systems and capabilities that have a potentially significant impact on  interoperability in allied and coalition operations. Generally, these  programs have greater costs or are more complex than smaller programs,  and therefore provide an opportunity to assess the effectiveness of more  aspects of the JCIDS process.
section_title: Appendix II: Scope and Methodology
paragraphs: To assess the extent to which the Joint Staff has developed and  implemented an analytic approach to prioritize capability needs, we  reviewed relevant legislation and Joint Staff guidance on the roles and  requirements of the JCIDS process and the JROC as it pertains to  assigning levels of priority to capability proposals. Specifically, we  reviewed section 181 of the U.S. Code, Title 10; Chairman of the Joint  Chiefs of Staff Instruction 3170.01G; Chairman of the Joint Chiefs of Staff  Instruction 3170.01H (which we reviewed in draft form); and the JCIDS  Manual. We then compared the current and prior versions of the  instruction and manual to identify changes in the guidance with respect to  prioritization of capability proposals. We met with officials from the Joint  Staff; Department of the Air Force; Department of the Army; Department  of the Navy; and Office of the Under Secretary of Defense for Acquisition,  Technology and Logistics to discuss their perspectives on the  implementation of changes to JCIDS with respect to prioritizing capability  requirements. In order to understand how the JROC is implementing its  new approach for prioritizing capabilities, we reviewed briefing materials  presented at a JROC forum in November 2011. To corroborate our  understanding of the documents we reviewed, we conducted interviews  with Joint Staff and Office of the Under Secretary of Defense for  Acquisition, Technology and Logistics officials. To understand the Joint  Staff’s recent internal review of JCIDS, we reviewed the charter and  recommendations, and met with Joint Staff officials to discuss how those  recommendations from the review might affect the JROC’s prioritization of  capability proposals. We also reviewed prior reports by GAO, the House  Armed Services Committee, and the Defense Business Board that  discussed prioritization of capability proposals through JCIDS, and  compared the JROC’s current efforts to prioritize with what has been  reported in the past. We assessed whether the guidance in the JCIDS  Manual and JCIDS instruction (in draft form during our review and issued  in January 2012) on prioritization meets the intent of recommendations  contained in our prior reports.
To assess the extent to which the JROC has considered aspects of the  availability and operational support requirements of weapon systems— called sustainment—when validating the requirements of proposed  capability solutions, we reviewed relevant DOD and Joint Staff policy  documents and related guidance outlining the requirement to develop and  report sustainment metrics for capability documents. Specifically, we  reviewed the reporting requirements for major defense acquisition  programs processed through JCIDS.requirements for capability development documents and to understand  the JCIDS process, we reviewed the JCIDS Manual enclosure pertaining  to sustainment and instructions from the Chairman of the Joint Chiefs of  Staff. We also reviewed prior GAO work on related topics. Further, we  interviewed DOD and Joint Staff officials to discuss preparation,  presentation, and consideration of sustainment data.
We also conducted a case study analysis of select capability  development documents that included sustainment information. We  sought a universe of all capability development documents subject to  reporting the sustainment key performance parameter validated since  2007, when JCIDS began requiring program sponsors to include this  information. We initially obtained a universe of 22 JCIDS capability  development documents from the Office of the Under Secretary of  Defense for Acquisition, Technology and Logistics/Logistics and Materiel  Readiness. We relied on this provided list of documents because the  Joint Staff’s Knowledge Management/Decision Support database did not  produce reliable results of all requirements documents containing  sustainment information. We narrowed the list by eliminating programs  that have been downgraded, truncated, or canceled; programs in which  the sustainment data were not at the key performance parameter level or  were more narrowly defined by only one type of platform; programs  whose information was contained in capability production documents  rather than capability development documents; and programs whose  capability development documents were entered in JCIDS before the full  implementation of the sustainment key performance parameter  requirement. Additionally, several programs did not have supporting  documentation that would allow a review; Joint Staff officials we met with  stated that these documents were not available either because of a  misidentification of the type of capability document that was being  reviewed (a capability production document as opposed to a capability  development document), or because the document was not included in  the Joint Staff’s database for JROC review. These factors led to a refined  universe of 12 requirements documents. We then randomly selected  capability development documents for two programs per service—Army,  Air Force, and Navy—resulting in a total of six programs to serve as case  studies. Because this was a nonprobability sample of programs, the  results are not generalizable to all programs; however, they are illustrative  of the kinds of issues that are possible in such programs. In order to  assess reported sustainment information in the six selected cases, we  performed a content analysis of the documentation available for the six  cases. Two GAO analysts independently reviewed each of the six  capability development documents, assessing whether each of the  individual elements of the JCIDS Manual sustainment metrics was  included, coding the inclusion of each metric as “yes,” “no,” “partial,” and  “don’t know.” The two analysts then discussed and reconciled all initial  disagreements regarding the assigned codes. We then discussed the  results of this content analysis with officials from the Joint Staff and the  Office of the Under Secretary of Defense for Acquisition, Technology and  Logistics/Logistics and Materiel Readiness to verify that the results of our  analysis were valid.
We conducted this performance audit from April 2011 through February  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Margaret Morgan, Assistant  Director; Melissa Blanco; Mae Jones; Kate Lenane; Jennifer Madison;  Ron Schwenn; Michael Shaughnessy; Michael Silver; Jennifer Spence;  Amie Steele; and Kristy Williams made key contributions to this report.