Defense Management: Guidance and Progress Measures Are Needed to Realize Benefits from Changes in DOD's Joint Requirements Process
Released: 2012-02-24T00:00:00

Letter

Background

JCIDS; the Planning, Programming, Budgeting and Execution process; and the Defense Acquisition System broadly make up DOD’s overall defense acquisition management framework. JCIDS was implemented in 2003 to guide future defense programs from a joint capabilities perspective. JCIDS is one of the first steps in DOD’s acquisition processes; JCIDS participants work to identify and determine whether to validate the need for capabilities proposed by the services, the defense agencies, and the combatant commands. Once a requirement is validated, the services rely on the DOD’s Planning, Programming, Budgeting and Execution process, through which DOD allocates financial resources across the department—including the services—to identify funding for validated capability solutions. DOD then manages the development and procurement of proposed capabilities through the Defense Acquisition System.

DOD implemented the JCIDS process in 2003 in an effort to assist the JROC by changing DOD’s requirements validation process from a service-specific perspective to a joint capabilities perspective. The JROC, which is chaired by the Vice Chairman of the Joint Chiefs of Staff, consists of a general or admiral from each of the military services and may include combatant commanders or deputy commanders when directed by the JROC Chairman. The JROC is charged with assisting the Chairman of the Joint Chiefs of Staff with a number of tasks, including (1) identifying, assessing, and approving joint military requirements to meet the national military strategy; (2) establishing and assigning priority levels for joint military requirements; and (3) reviewing the estimated level of resources required to fulfill each joint military requirement and ensuring that the resource level is consistent with the requirement’s priority, among others. The JROC also assists acquisition officials in identifying alternatives to any acquisition programs that meet joint military requirements for the purposes of certain statutory provisions, addressing matters such as cost growth. In 2009, the Weapon Systems Acquisition Reform Act expanded the role of the JROC by directing it to assist the Chairman of the Joint Chiefs of Staff in ensuring that trade-offs among cost, schedule, and performance objectives are considered for joint military requirements and establishing an objective for the overall period of time within which an initial operational capability should be delivered to meet each joint military requirement.

The JROC reviews requirements for programs designated as JROC- interest based on their expected cost and complexity and, under guidance in effect through December 2011, also reviewed programs at the request of certain senior DOD officials. Within JCIDS, the JROC is supported in its duty to review and validate joint capability needs by the Joint Capabilities Board and six Functional Capabilities Boards. The Joint Capabilities Board is chaired by the Director of the Joint Staff’s Directorate for Force Structure, Resources, and Assessment, and each Functional Capabilities Board is chaired by a general/flag officer or civilian equivalent. The Joint Capabilities Board reviews capability documents before they are passed on to the JROC for its review and also serves as the validation authority for certain programs that do not reach JROC- interest thresholds, although the JROC may review any JCIDS document or other issues requiring joint resolution. Functional Capabilities Boards are responsible for reviewing proposed requirements specific to joint capability areas, such as protection, logistics, or battlespace awareness.

In JCIDS, the JROC and its supporting organizations review requirements documents related to capability gaps and the major defense acquisition programs intended to fill those gaps prior to key acquisition milestones. These requirements documents—initial capabilities documents, capability development documents, and capability production documents for materiel solutions and change recommendations for nonmateriel solutions—are submitted into the JCIDS process by capability sponsors.The initial capabilities document identifies a specific capability gap, or set of gaps, and if a materiel solution is required, helps inform the initial stages of the acquisition process, which include an analysis of the alternative solutions to fulfilling the capability need and the selection of a preferred system concept. When the technology development phase of the acquisition process is complete, a program sponsor completes a capability development document that includes more detail on the desired capabilities of the proposed system and defines the system’s key performance parameters or attributes against which the delivered increment of capability will be measured. Finally, the sponsor prepares a capability production document to describe the actual performance of the system that will deliver the required capability. Figure 1 depicts how JCIDS reviews align with the acquisition process.

The House Armed Services Committee and a panel established by the committee have discussed long-standing challenges with the JCIDS process and the JROC’s fulfillment of its statutory responsibilities. The House Armed Services Committee, in a report accompanying a bill for the National Defense Authorization Act for Fiscal Year 2008, described a legislative provision that would allow for joint decision making as opposed to service-centric budget considerations by incorporating clear priorities and budget guidance into the JROC process. In 2009, the House Armed Services Committee established a panel on defense acquisition reform because of a sense that the acquisition system was not responsive enough for today’s needs, not rigorous enough in protecting taxpayers, and not disciplined enough in the acquisition of weapon systems for tomorrow’s wars. The panel received testimony that the Joint Staff lacked some of the analytical expertise necessary to ensure that the JCIDS process rigorously vets proposed requirements. Additionally, since 2008 we have reported on these challenges. We reported in 2008 that the JCIDS process was not effective in prioritizing capability gaps, and we noted that capability needs continued to be proposed and defined by the services with little involvement from the joint community. We recommended that the Secretary of Defense direct the Chairman of the Joint Chiefs of Staff to develop an analytic approach within JCIDS to better prioritize and balance the capability needs of the military services, combatant commands, and other defense components. DOD partially agreed with our recommendation but did not fully implement it, and prioritization remains service driven. More recently, in June 2011 we reported that the JROC did not always consider trade-offs among cost, schedule, and performance objectives; prioritize requirements; consider redundancies across proposed programs; or prioritize and analyze capability gaps in a consistent manner. We recommended that the JROC require higher-quality resource estimates from requirements sponsors to ensure that best practices are being followed, provide a sound basis to ensure that trade-offs are considered, prioritize requirements across proposed programs, and address potential redundancies during requirements reviews, among other steps. DOD partially agreed with our recommendations and commented that improvements to the quality of resource estimates would be addressed in upcoming changes to the JCIDS process.

In May 2011, we also reported that combatant command officials raised concerns that JCIDS focuses more on long-term service-centric capability gaps than on combatant commands’ more immediate and largely joint gaps. JCIDS was designed as a deliberate process to meet longer-term joint needs. To address urgent needs, DOD established a separate process—the joint urgent operational needs process—in 2005. The joint urgent operational needs process was intended to respond to needs associated with combat operations in Afghanistan and Iraq and the War on Terror. The revised JCIDS guidance canceled separate guidance for joint urgent operational needs and incorporates and describes the joint urgent operational needs process. Urgent operational needs, as defined by the new JCIDS guidance, are capability requirements needed for ongoing or anticipated contingency operations that if left unfulfilled could potentially result in loss of life or critical mission failure. For this report, we focus on requirements that have not been identified as urgent and instead follow the deliberative JCIDS process.

The Joint Staff Has Initiated Efforts to Prioritize Capability Needs, but Implementation Processes Are Not Fully Developed and Clearly Documented

The Joint Staff is undertaking efforts to improve the ability to prioritize capability needs from a joint perspective through JCIDS and to align those needs with available budget resources. However, implementation processes for JCIDS’s new approach to managing requirements and considering affordability are still evolving and have not been fully developed and clearly documented.

Changes in JCIDS Processes Are Intended to Improve Prioritization of Capability Needs and Guide Future Investments

Determining priorities among joint requirements has been a responsibility of the JROC since Congress amended section 181 of Title 10 of the U.S. Code in 2008 to require the JROC to assist in establishing and assigning priority levels for joint military requirements and to help ensure that resource levels associated with those requirements are consistent with the level of priority. DOD officials acknowledge that JCIDS has been ineffective in helping the JROC carry out these responsibilities. We have previously reported that JCIDS’s ability to align resources to balance competing needs could be improved if it had an analytic approach that provided a means to review and validate proposals to ensure that the most important capability needs of the department are being addressed. We further said that such an approach should establish criteria and measures for identifying capability gaps and determining the relative importance of capability needs. Finally, the approach should result in measurable progress in allocating resources in order to eliminate redundancies, gain efficiencies, and achieve a balanced mix of executable programs.

DOD officials told us that downward pressure on the defense budget has led the Joint Staff to change how the JCIDS process is used to strengthen its ability to support JROC members in making trade-off decisions among requirements and balancing risks across the force within expected resources. In fall 2011, according to officials, the incoming Vice Chairman of the Joint Chiefs of Staff, as the Chairman of the JROC, began to make changes in the JCIDS processes to focus on what capabilities currently exist and weigh the benefits of investing in new capabilities with their estimated costs early in the review process. The Joint Staff issued draft guidance in October 2011 and began implementation based on the draft guidance. The Joint Staff issued the final guidance in January 2012. Descriptions of these changes and their implementation follow:

New capabilities will be considered as part of a “capability portfolio approach.” Under the portfolio approach, officials stated that JROC members are to ensure that proposed investments in capabilities address joint needs or they will not be validated to proceed to the acquisition process. In addition to validating capability proposals, according to Joint Staff officials, the JROC has begun examining how the services’ existing programs can support joint operations to reduce duplication of capabilities. As of December 2011, according to Joint Staff officials, the last five closed meetings of the JROC began conversations about how to meet requirements by considering available capabilities and the costs and benefits of proposed programs. As a result, according to Joint Staff officials, at least five classified programs have been reviewed and altered by either comparing redundant capabilities, reducing capacity, adjusting delivery schedules, or directing follow-on analysis before moving programs forward. To support JROC decision making, officials reported that Functional Capabilities Boards will be tasked with examining requirements, their associated capability gaps, and proposed solutions within their capability portfolios, and independently assessing how a proposed capability fits into its corresponding joint capability area. The Functional Capabilities Boards previously had responsibility for identifying, assessing, and prioritizing (if required) joint capability needs proposals within assigned joint capability areas but, according to officials, have not always carried out these responsibilities. Previously, Functional Capabilities Boards acted primarily as technical reviewers of requirements documents, and program sponsors briefed the JROC on the attributes of the program. However, the new guidance does not specify how the independent assessment is to be conducted, and it is too soon to tell how the Functional Capabilities Boards will respond to the new requirement. Officials reported that Functional Capabilities Boards are expected to develop methodologies on a case-by-case basis. DOD officials said that the analytic approach and uses of analytic information will evolve over time.

DOD has previously attempted to manage capabilities departmentwide through a portfolio approach, but has never fully implemented the approach. In 2006, DOD established an effort to manage resources across capability areas by establishing capability portfolio managers to enable DOD to develop and manage capabilities across the department rather than by military service or individual program, and by doing so, to improve the interoperability of future capabilities, minimize capability redundancies and gaps, and maximize capability effectiveness. However, as we reported in 2008, capability portfolio managers make recommendations on capability development issues within their portfolios but do not have independent decision-making authority. In 2011, Joint Staff officials told us that they were unaware of capability portfolio managers’ active involvement in the JCIDS process.

Attendance at JROC meetings will be limited to key decision makers and stakeholders. Beginning in October 2011, the JROC Chairman began to limit attendance at JROC meetings to facilitate candid discussion among senior leaders about priorities for joint requirements and alternative solutions. Joint Staff officials told us that previously, meetings were open to a broad range of interested parties and service sponsors provided briefings on their proposals for new capabilities. Under the new approach being implemented, a representative of one of the Functional Capabilities Boards will provide a briefing on the proposal to the JROC. The representatives would then present the board’s independent assessment of benefits, costs, and risk for the JROC to discuss and decide upon. According to JCIDS officials, JROC members are expected to make decisions from the perspective of the joint force and avoid taking a service-centric approach.

Affordability of proposals will be a primary factor in validation decisions. JROC members have always been expected to consider the resource implications of validation decisions, but officials stated that until recently, these considerations have not been a focus because capabilities were not competing with each other for funding. According to officials, the Functional Capabilities Boards have been directed to take similar steps to ensure that capability proposals not only meet technical requirements but also represent the most efficient alternative for providing a capability within the joint capability area without creating duplication or overlap. According to JCIDS officials, the JROC is also reconsidering previous validation decisions and asking for changes to proposals to minimize costs. We reviewed recent Functional Capabilities Board briefings to the JROC, which provided information on how needs might be met with current capabilities and alternatives that might meet needs while minimizing costs. However, it is too soon to assess how the JROC will consider affordability of programs when making validation decisions.

New JCIDS Processes Are Not Fully Developed and Clearly Documented

The Vice Chairman of the Joint Chiefs of Staff has sought some policy changes and, according to officials, provided other direction to implement changes in the JCIDS process, but the new approach to managing requirements and considering affordability is still evolving and has not been fully developed and clearly documented. According to Joint Staff officials, the dynamic fiscal environment and the evolutionary method being used to develop the new approach and implementation processes make it important that decision makers maintain flexibility in decision making. We believe that the new approach has promise in positioning the JROC to more accurately identify capability gaps and trade-offs, but it has not been fully developed to include steps to ensure that the approach is fully implemented, that the intent is fully communicated to all stakeholders involved, and that the results of the new approach will be measurable.

We have previously reported that key practices for results-oriented management involve leadership from top officials as well as the involvement of stakeholders at all levels throughout a period of transition. We have also reported that in order to demonstrate a successful results-oriented framework, officials must include clearly defined measures to assess intended outcomes. As shown in table 1, key practices for supporting change should include, among other actions, obtaining and sustaining support from senior leadership to facilitate the transformation, establishing clear lines of communication between all affected parties, and demonstrating value and credibility of new processes through the use of metrics.

According to Joint Staff officials, the Chairman of the JROC has been a driving force in positioning the JROC to take on the responsibility of aligning needs and balancing risk and resources, which fulfills one of the key steps for results-oriented management, but the approach is still new and officials have not completed all of the steps that facilitate institutional acceptance and implementation of the new approach. The Joint Staff has begun the process of change by articulating a clear rationale for change—that the JROC can more effectively represent the warfighters’ requirements and make strategic trade-off decisions as budgets stay flat or decrease by taking a more active role in shaping an affordable joint force. However, best practices for managing a results-oriented change state that goals and procedures should be communicated to stakeholders throughout the organization so that they understand how they should implement the new approach and how the organization will measure progress. The Joint Staff issued guidance that outlines new procedures intended to establish an approach for prioritizing capabilities from a joint perspective and to increase the timeliness of JCIDS reviews by categorizing proposals according to the level of urgency of the need and streamlining procedures for urgent needs. However, the guidance does not clearly outline criteria and measures for demonstrating progress toward meeting the goal of aligning needs with available resources or clearly communicate the goals and the analytic approach envisioned to support JROC decision making. Further, the guidance does not describe how the proposed change will affect the services, combatant commands, and other stakeholders. Finally, the JCIDS guidance does not establish criteria and measures for demonstrating progress toward the goal of creating a balanced portfolio of programs that takes into account needs, risks, and available resources, nor do other documents provided to us by DOD. Measures such as the proportion of requirements that address joint priority needs versus service-specific needs, the savings obtained through the elimination of redundant capabilities, and the comparison of estimated costs of a proposed program with the actual costs of operating the program over its life cycle could be helpful in assessing whether the process is balancing requirements with available resources or whether further adjustments to the process are needed.

Considering new capabilities across the department in the context of joint capability areas can help DOD begin to identify priorities for future investment. However, unless the Joint Staff takes steps to define and institutionalize the new approach by adhering to the key principles of results-oriented management, it is not clear whether the current momentum in implementing an analytic approach through JCIDS will be sustained.

JROC Has Not Always Had Complete Information on Future Costs to Sustain Proposed Capabilities before Approving Documentation for Program Development

Even though sustainment costs make up a significant portion of the total ownership costs of a weapon system, the JROC has not always had complete information on such costs when validating documentation used in the decision to initiate program development. “Sustainment” as a category represents a range of activities intended to ensure the availability of weapon systems and to support their operations, including some logistics and personnel services. During the identification of capability gaps and consideration of selected alternatives, it is difficult for the sponsors to provide detailed information on program capabilities and cost estimates. As a major defense acquisition program moves toward the development stage, JCIDS requires that more complete and accurate sustainment information be presented in capability development documents. According to DOD officials, decision makers need more accurate cost information to assess whether the benefit of a proposed capability justifies the cost of sustaining the capability over its life. JCIDS guidance requires that sponsors of potential major defense acquisition programs include sustainment information in capability development documents, which detail proposed solutions to fulfill capability needs.

The JROC has generally relied on sponsor-provided assessments of sustainment information in the capability development documents to make its validation decisions, but these documents have not always included all the information suggested in JCIDS guidance or sufficient detail to enable the JROC to assess the quality of the information. A DOD regarding the development of sustainment information suggests manual that when sustainment requirements and underlying assumptions are not clearly documented, subsequent decisions about the project may be based on incorrect assumptions. Prior GAO workjoint warfighting capabilities and proposals to fulfill the gaps should be clearly identified to decision-making bodies, such as the JROC, to inform deliberations. Further, information should be complete so those making the important decisions may do so as effectively as possible. suggests that gaps in JCIDS guidance requires sponsors of major defense acquisition programs to address sustainment based on four metrics—materiel availability, operational availability, reliability, and ownership cost (renamed operation and support cost in January 2012). The guidance includes a series of criteria, which are listed as review criteria in the guidance. The criteria provide additional information on each metric. For example, JCIDS guidance lists as review criteria for the materiel availability metric whether there is a clear definition and accounting for the intended service life of the program, an identification of planned downtime, and a comparison of downtime value with experiences of other analogous systems, among other criteria. Table 2 outlines examples of key review criteria within each of the four sustainment metrics under the guidance in effect through December 2011.

Program sponsors provide initial information on the sustainment metrics for proposed capability solutions when they submit a capability development document, one of three capability documents the JROC considers in its review and potential validation of capability proposals. Officials from both the Joint Staff and the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics review the capability development documents and, according to officials, verify that all required sustainment elements have been included before the documents are validated by the JROC. Officials also provide their independent assessments of the quality of the cost estimates to the JROC.from the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics said they work with program sponsors to ensure that life cycle sustainment planning and costs are as accurate as possible. Officials from the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics and the Joint Staff told us that they consider reported sustainment information important to a program’s development and review all reported information.

The JCIDS Manual notes that listed criteria, information, and activities cannot necessarily be applied to all systems. Sponsors have a degree of latitude in determining which items are applicable for their specific concept, technology, system, or combination of these. For example, a program sponsor for a major defense acquisition program is required to report a measure for operational availability, but would not necessarily have to report on the respective criteria, such as addressing downtime associated with failure, including recovery time or movement of maintenance teams to the work site. Because the guidance does not specifically require program sponsors to report on the individual criteria, they generally include some, but not all, of the individual criteria.

Our analysis of six capability development documents found that all of the documents provided information on all of the required sustainment metrics. However, we found that the completeness of information reported for all of the metrics’ key criteria varied. Specifically, none of the documents included complete information for each of the four sustainment metrics’ review criteria elements. In addition, each of the documents had some common omissions; for example, none of the six capability documents we reviewed included information on all nine of the ownership cost metric’s criteria elements. Further, several of these documents only included information on one criteria element for a single metric, and none reported information on all of the elements for any of the metrics. Finally, when information on the metrics’ key criteria was provided, the level of detail varied among the documents. For example, for some criteria, some documents provided a paragraph of supporting information and analysis whereas others provided single-sentence responses that did not provide as substantial detail for the supporting criteria. According to officials, for the capability documents we reviewed, there were cases in which information on some of the suggested criteria should have been included but were not.

Joint Staff officials and officials from the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics noted that while criteria for each of the required sustainment metrics may not be applicable to every program, it would be beneficial to the JROC if the services reported on the criteria for each metric outlined in the guidance or indicated a reason why a specific criterion was not applicable. Officials from the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics told us that when they conduct their reviews of sponsor-reported sustainment information, not all of the supporting documentation they need for a thorough independent assessment is available in the capability development document. These officials said that they can generally find detailed documentation on sustainment planning and costs from sources outside of the JCIDS process, but that the information is not always readily available within the JCIDS database. Ultimately, review efficiency could be improved if all the information were available in the JCIDS database. Updated JCIDS guidance issued in January 2012 still does not clearly require program sponsors to report on the individual criteria for each of the sustainment metrics. The Joint Staff is developing a new reporting tool intended to provide a standard format for reporting sustainment information. The tool will require program sponsors to at least minimally address each of the four sustainment metrics in order to submit the capability development document for review through JCIDS, according to officials. However, officials stated that this tool will not require that sponsors address each of the individual criteria elements within the four sustainment metrics.

Without complete and detailed information on each of the individual criteria elements, the JROC may not be in the best position to weigh the costs and benefits of a proposal within a capability portfolio. The quality and completeness of the data that sponsors provide through the capability development document in the JCIDS process will become more important as the JROC increases its examination of the benefits of programs balanced against their associated costs. As we have previously reported, incomplete and inaccurate sustainment information has been a long-standing problem for DOD. In November 2009, a DOD team assessing weapon system acquisition reform reported that DOD lacked valid, measurable sustainment information to accurately assess how programmatic decisions affected life cycle costs and made recommendations to improve weapon system life cycle sustainment.Until the JROC requires program sponsors to report complete sustainment information, including both the overall metrics and the supporting criteria, the JROC may not always have the complete and detailed information it needs to make the most informed decisions.

Conclusions

The prospect of declining budgets has amplified the need for DOD to prioritize among capability gaps and to use its resources to maximize the capabilities of the joint force. The Chairman of the JROC has begun to take steps to better balance risks across the joint force by examining proposals for new capabilities within the context of existing joint capability areas and to consider affordability, including sustainment costs, as a factor in validating requirements proposals. The revised approach is new and evolving, but in order for it to achieve the intended results of prioritizing capability needs and aligning those needs with available resources, the Joint Staff needs to take steps to fully develop the approach and document it more explicitly. Specifically, DOD does not yet have a documented implementation plan with measures of success that support change. In addition, having good sustainment information is a key element needed to improve JCIDS’s success over the long term. Sustainment costs historically represent 70 percent of a system’s life cycle costs, but DOD has been making decisions with incomplete information on sustainment and does not require that sponsors address all of the criteria outlined in JCIDS guidance. Until the JROC has developed and fully documented an approach for prioritizing capability needs and aligning these needs with available resources and has complete sustainment information associated with the operation of new capabilities, it will not be in the best position to align resources with priorities or balance costs with benefits in affordable investment plans.

Recommendations for Executive Action

To help sustain momentum for efforts to bring a capability portfolio approach to the JCIDS process and to improve the quality of sustainment information reported in capability development documents, we recommend that the Vice Chairman of the Joint Chiefs of Staff, as the JROC Chairman, take the following two actions:

Revise and implement guidance to reflect changes to the JCIDS process as well as to establish criteria and measures for determining the relative importance of capability needs across capability areas and assessing progress.

Explicitly require that program sponsors address each of the criteria outlined for the individual sustainment metrics when submitting capability development documents.

Agency Comments and Our Evaluation

The Joint Staff provided written comments on a draft of this report. In its comments, the Joint Staff said our report represented a fair and objective assessment of the JCIDS process. It partially agreed with both of our recommendations, citing ongoing and planned changes to the joint requirements development process. However, the comments did not detail any specific steps that DOD plans to take to address our recommendations. The comments are reprinted in their entirety in appendix III. The Joint Staff also provided technical comments, which we have incorporated into the report as appropriate.

The Joint Staff partially concurred with our recommendation that the Vice Chairman of the Joint Chiefs of Staff revise and implement guidance to reflect changes to the JCIDS process as well as to establish criteria and measures for determining the relative importance of capability needs across capability areas and assessing progress. In its written response, the Joint Staff described recent initiatives to substantially change the joint requirements development process to require that capability requirements be evaluated within a capability portfolio by Functional Capability Boards, the Joint Capabilities Board, and the JROC. The Joint Staff also discussed its planned efforts to improve prioritization of capability needs and stated that JROC reviews will incorporate an evolving portfolio assessment tool. The Joint Staff expects that the departmentwide priorities outlined in DOD’s strategic guidance as well as a revised process for assessing capability gaps and combatant command priorities will enable the JROC to make more informed decisions about priorities. While we agree that the Joint Staff has taken important steps to enable prioritization of capabilities, such as addressing prioritization in a new enclosure in its revised JCIDS Manual, the enclosure does not explicitly outline implementation processes. We continue to believe that clear guidance that establishes criteria for determining priority levels and measures for demonstrating progress will be essential in sustaining momentum toward the goal of creating a balanced portfolio of programs that takes into account needs, risks, and available resources. Moreover, providing guidance that fully documents the new procedures for assigning priority levels to capability gaps is an essential step toward clarifying how the procedures will be implemented.

The Joint Staff also partially concurred with our recommendation that the Vice Chairman of the Joint Chiefs of Staff explicitly require that program sponsors address each of the criteria outlined for the individual sustainment metrics when submitting capability development documents. According to its written comments, the Joint Staff criteria for the sustainment metrics were designed to guide the development of requirements, but were not intended to be prescriptive because individual programs are unique and criteria applicable to one problem may not apply to another. We agree that each of the criteria may not be applicable to every program. However, if program sponsors addressed each criterion in some manner, including explaining that a criterion is not applicable to the program, the JROC would be assured that program sponsors considered all criteria when developing program proposals. Further, the Joint Staff commented that JCIDS reviews of capability development documents by Joint Staff and Office of the Secretary of Defense officials ensure that a document is thoroughly vetted for consideration by the JROC. It suggested that the inclusion of analyses and rationale for sustainment metrics development in capability development documents might be duplicative because this information is contained in acquisition documents that exist outside of JCIDS. However, as we noted in our report, the documents that contain the analysis and rationale for the required sustainment metrics are not necessarily reviewed by or available to the JROC members during their consideration of a capability development document. We continue to believe that the inclusion of a sponsor- provided rationale for each metric criterion would enhance the thoroughness and efficiency of the JROC’s review of sustainment information through JCIDS.

We are sending copies of this report to interested congressional committees, the Secretary of Defense, the Vice Chairman of the Joint Chiefs of Staff, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-3489 or pendletonj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV.

Appendix I: Observations on Reporting Provisions in the National Defense Authorization Act for Fiscal Year 2011

Section 862 of the Ike Skelton National Defense Authorization Act for Fiscal Year 2011 requires the Comptroller General to carry out a comprehensive review of the Joint Capabilities Integration and Development System (JCIDS) and to submit to the congressional defense committees a report on the review. This appendix, in conjunction with the letter, addresses each of the reporting provisions as described in the act. Specifically, section 862 requires the following contents for the review: Purpose. The purpose of the review is to evaluate the effectiveness of JCIDS in achieving the following objectives:

Timeliness in delivering capability to the warfighter.

Efficient use of the investment resources of the Department of Defense (DOD).

Control of requirements creep.

Responsiveness to changes occurring after the approval of a requirements document environment, the emergence of new capabilities, or changes in the resources estimated to procure or sustain a capability). (including changes to the threat

Development of the personnel skills, capacity, and training needed for an effective and efficient requirements process.

Matters considered. In performing the review, the Comptroller General is required to gather information on and consider the following matters:

The time that requirements documents take to receive approval through JCIDS.

The quality of cost information considered in JCIDS and the extent of its consideration.

The term requirements document is defined in the statute as “a document produced in JCIDS that is provided for an acquisition program to guide the subsequent development, production, and testing of the program and that—(A) justifies the need for a materiel approach, or an approach that is a combination of materiel and non-materiel, to satisfy one or more specific capability gaps; (B) details the information necessary to develop an increment of militarily useful, logistically supportable, and technically mature capability, including key performance parameters; or (C) identifies production attributes required for a single increment of a program.” See § 862(c)(2).

The extent to which JCIDS establishes a meaningful level of priority for requirements.

The extent to which JCIDS is considering trade-offs between cost, schedule, and performance objectives.

The quality of information on sustainment considered in JCIDS and the extent to which sustainment information is considered.

An evaluation of the advantages and disadvantages of designating a commander of a unified combatant command for each requirements document for which the Joint Requirements Oversight Council (JROC) is the validation authority to provide a joint evaluation task force to participate in a materiel solution and to provide input to the analysis of alternatives; participate in testing (including limited user tests and prototype testing);provide input on a concept of operations and doctrine; provide end user feedback to the resource sponsor; and participate, through the combatant commander concerned, in any alteration of the requirement for such solution.

Section 862 also provided definitions for JCIDS, requirements document, requirements creep, and materiel solution.

Tables 3 through 12 contain our response to each of the requirements mandated by the Ike Skelton National Defense Authorization Act for Fiscal Year 2011. The provision mandating our report defined the JCIDS process by referring to the JCIDS guidance in effect from March 2009 through December 2011. Accordingly, our response to the mandated elements as presented in this appendix generally focuses on JCIDS under that guidance. However, we also provide some information relating to JCIDS as described in the revised guidance issued in January 2012. In addition, our assessments generally focused on those programs that were determined to be JROC-interest, or those that were designated as major defense acquisition programs or major automated information systems and capabilities that have a potentially significant impact on interoperability in allied and coalition operations. Generally, these programs have greater costs or are more complex than smaller programs, and therefore provide an opportunity to assess the effectiveness of more aspects of the JCIDS process.

Appendix II: Scope and Methodology

To assess the extent to which the Joint Staff has developed and implemented an analytic approach to prioritize capability needs, we reviewed relevant legislation and Joint Staff guidance on the roles and requirements of the JCIDS process and the JROC as it pertains to assigning levels of priority to capability proposals. Specifically, we reviewed section 181 of the U.S. Code, Title 10; Chairman of the Joint Chiefs of Staff Instruction 3170.01G; Chairman of the Joint Chiefs of Staff Instruction 3170.01H (which we reviewed in draft form); and the JCIDS Manual. We then compared the current and prior versions of the instruction and manual to identify changes in the guidance with respect to prioritization of capability proposals. We met with officials from the Joint Staff; Department of the Air Force; Department of the Army; Department of the Navy; and Office of the Under Secretary of Defense for Acquisition, Technology and Logistics to discuss their perspectives on the implementation of changes to JCIDS with respect to prioritizing capability requirements. In order to understand how the JROC is implementing its new approach for prioritizing capabilities, we reviewed briefing materials presented at a JROC forum in November 2011. To corroborate our understanding of the documents we reviewed, we conducted interviews with Joint Staff and Office of the Under Secretary of Defense for Acquisition, Technology and Logistics officials. To understand the Joint Staff’s recent internal review of JCIDS, we reviewed the charter and recommendations, and met with Joint Staff officials to discuss how those recommendations from the review might affect the JROC’s prioritization of capability proposals. We also reviewed prior reports by GAO, the House Armed Services Committee, and the Defense Business Board that discussed prioritization of capability proposals through JCIDS, and compared the JROC’s current efforts to prioritize with what has been reported in the past. We assessed whether the guidance in the JCIDS Manual and JCIDS instruction (in draft form during our review and issued in January 2012) on prioritization meets the intent of recommendations contained in our prior reports.

To assess the extent to which the JROC has considered aspects of the availability and operational support requirements of weapon systems— called sustainment—when validating the requirements of proposed capability solutions, we reviewed relevant DOD and Joint Staff policy documents and related guidance outlining the requirement to develop and report sustainment metrics for capability documents. Specifically, we reviewed the reporting requirements for major defense acquisition programs processed through JCIDS.requirements for capability development documents and to understand the JCIDS process, we reviewed the JCIDS Manual enclosure pertaining to sustainment and instructions from the Chairman of the Joint Chiefs of Staff. We also reviewed prior GAO work on related topics. Further, we interviewed DOD and Joint Staff officials to discuss preparation, presentation, and consideration of sustainment data.

We also conducted a case study analysis of select capability development documents that included sustainment information. We sought a universe of all capability development documents subject to reporting the sustainment key performance parameter validated since 2007, when JCIDS began requiring program sponsors to include this information. We initially obtained a universe of 22 JCIDS capability development documents from the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics/Logistics and Materiel Readiness. We relied on this provided list of documents because the Joint Staff’s Knowledge Management/Decision Support database did not produce reliable results of all requirements documents containing sustainment information. We narrowed the list by eliminating programs that have been downgraded, truncated, or canceled; programs in which the sustainment data were not at the key performance parameter level or were more narrowly defined by only one type of platform; programs whose information was contained in capability production documents rather than capability development documents; and programs whose capability development documents were entered in JCIDS before the full implementation of the sustainment key performance parameter requirement. Additionally, several programs did not have supporting documentation that would allow a review; Joint Staff officials we met with stated that these documents were not available either because of a misidentification of the type of capability document that was being reviewed (a capability production document as opposed to a capability development document), or because the document was not included in the Joint Staff’s database for JROC review. These factors led to a refined universe of 12 requirements documents. We then randomly selected capability development documents for two programs per service—Army, Air Force, and Navy—resulting in a total of six programs to serve as case studies. Because this was a nonprobability sample of programs, the results are not generalizable to all programs; however, they are illustrative of the kinds of issues that are possible in such programs. In order to assess reported sustainment information in the six selected cases, we performed a content analysis of the documentation available for the six cases. Two GAO analysts independently reviewed each of the six capability development documents, assessing whether each of the individual elements of the JCIDS Manual sustainment metrics was included, coding the inclusion of each metric as “yes,” “no,” “partial,” and “don’t know.” The two analysts then discussed and reconciled all initial disagreements regarding the assigned codes. We then discussed the results of this content analysis with officials from the Joint Staff and the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics/Logistics and Materiel Readiness to verify that the results of our analysis were valid.

We conducted this performance audit from April 2011 through February 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Margaret Morgan, Assistant Director; Melissa Blanco; Mae Jones; Kate Lenane; Jennifer Madison; Ron Schwenn; Michael Shaughnessy; Michael Silver; Jennifer Spence; Amie Steele; and Kristy Williams made key contributions to this report.