Defense Management: Additional Actions Needed to Enhance DOD's Risk-Based Approach for Making Resource Decisions
Released: 2005-11-15T00:00:00

Letter

Background

In our report, High-Risk Series: An Update, we identified agencies’ lack of comprehensive risk management strategies as an emerging challenge for the federal government. Increasingly limited fiscal resources across the federal government, coupled with the emerging requirements from the changing security environment, emphasize the need for DOD to develop a risk-based strategic investment approach. For this reason, we have advocated that DOD adopt a comprehensive risk management approach for decision making. Furthermore, DOD and other federal agencies are required by statute to develop a results-oriented management approach to strategically allocate resources on the basis of performance. The balanced scorecard—a concept to balance an organization’s focus across financial, customer, internal business, and learning and growth management areas— is one approach for developing results-oriented management that government agencies have recently started to adopt. At the direction of the Secretary of Defense, DOD developed a risk management framework that DOD later aligned with its results-oriented management activities through a DOD balanced scorecard.

Risk Management Is an Emerging 21st Century Challenge

An emerging challenge for the federal government involves the need for the completion of comprehensive national threat and risk assessments in a variety of areas. For example, emerging requirements from the changing security environment, coupled with increasingly limited fiscal resources across the federal government, emphasize the need for agencies to adopt a sound approach to establishing resource decisions. We have advocated that the federal government, including DOD, adopt a comprehensive threat or risk management approach as a framework for decision making that fully links strategic goals to plans and budgets, assesses values and risks of various courses of actions as a tool for setting priorities and allocating resources, and provides for the use of performance measures to assess outcomes. Based on our review of the literature, as shown in figure 1, the goal of risk management is to integrate systematic concern for risk into the usual cycle of agency decision making and implementation.

A risk management cycle represents a series of analytical and managerial steps, basically sequential, that can be used to assess risk, evaluate alternatives for reducing risks, choose among those alternatives, implement the alternatives, monitor their implementation, and continually use new information to adjust and revise the assessments and actions, as needed. Adoption of a risk management cycle such as this can aid in assessing risk by determining which vulnerabilities should be addressed, and how they should be addressed, within available resources. For the purposes of this report, we focused on the stages of the risk management cycle that involve DOD’s actions to set strategic goals and objectives, establish investment priorities based on risk assessments, and implementation and monitoring.

Risk management’s objectives are essentially the same as those of good management, and they are consistent with the broad economy and efficiency objectives of good government—namely, to provide better outcomes for the same amount of money, or to provide the same outcomes with less money. Therefore, risk management’s objectives are also compatible with those of the federal government’s results-oriented management approach, which was enacted in the Government Performance and Results Act (GPRA) of 1993, and the balanced scorecard approach. Congress enacted GPRA to focus the federal government on achieving results through the creation of clear links between the process of allocating scarce resources and an agency’s strategic goals, or the expected results to be achieved with those resources. Building on GPRA’s foundation, the current administration has taken steps to strengthen the integration of budget, cost, and performance information by including budget and performance integration as one of its management initiatives under the umbrella of the President’s Management Agenda. The Budget and Performance Integration initiative includes efforts such as the Program Assessment Rating Tool (PART), improving outcome measures, and improving monitoring of program performance. The balanced scorecard approach is a management tool that some federal agencies have adopted to help them translate the strategy set forth in a results-oriented management approach into the operational objectives that drive both behavior and performance. The balanced scorecard consists of four management areas that organizations should focus on—financial, customer, internal business, and learning and growth.

DOD’s 2001 Strategic Plan Outlines a New Risk Management Framework

DOD introduced the risk management framework in its strategic plan, the 2001 QDR report. The 2001 strategic plan articulated the new administration’s emphasis on transforming military forces and defense business practices to meet the changing threats facing our nation. In his guidance to the department for the 2001 QDR strategic planning process, the Secretary of Defense stated the need for DOD to use a risk mitigation approach for balancing force, resource, and modernization requirements across defense planning timelines. This guidance also stated that DOD must include the identification of output-based measures to reduce inefficiencies through the department in any approach to risk management. Building on the guidance, the 2001 QDR outlined DOD’s risk management framework. According to the QDR, the framework would enable DOD to address the tension between preparing for future threats and meeting the demands of the present with finite resources. It was also intended to ensure that DOD was sized, shaped, postured, committed, and managed with a view toward accomplishing the strategic plan’s defense policy goals.

DOD adapted the balanced scorecard concept to the risk management framework by substituting the four dimensions of risk—force management, operational, future challenges, and institutional—for the scorecard’s four management areas. The risk management framework was to be a transformational tool that would provide a balanced perspective of the organization’s execution of strategy and ensure a top-down approach. The 2002 policy guidance also designated four preliminary performance goals for each of the four risk quadrants. In addition, the guidance required that performance goals and measures were to be cascaded to the services and defense agencies. Figure 2 shows a comparison, as provided by DOD.

Despite Positive Steps, Additional Actions Needed to Fully Implement the Risk Management Framework

Despite positive steps, DOD needs to take additional actions before the risk management framework is fully implemented and DOD can demonstrate real and sustainable progress in using a risk-based and results-oriented approach to strategically allocate resources across the spectrum of its investment priorities. For example, DOD is still in the process of developing department-level measures for the framework that address results-based management principles, such as linking performance information to strategic goals so that this information can be used to monitor performance results and determine how well the department is doing in achieving its strategy. Without more results-oriented performance measures, DOD may be unable to provide the services and other defense components with clear roadmaps of how their activities contribute to meeting DOD’s strategic goals. In addition, the framework’s performance goals and measures are not clearly linked to DOD’s current strategic plan and strategic goals. Furthermore, the extent to which the risk management framework is linked to the budget cycle is unclear. Without better measures, clear linkages, and greater transparency, DOD will be unable to fully measure progress in achieving strategic goals or demonstrate to Congress and others how it considered risks and made trade-offs in making investment decisions.

Developing a Set of Measures That Can Be Used to Monitor Performance Is a Work in Progress

DOD has taken positive steps toward developing measures for each of the performance goals under the framework’s four risk quadrants; however, developing a set of measures that can be used to monitor performance results is still a work in progress. Based on GAO’s prior work on results- based management principles, we found that leading organizations’ performance measures are: (1) designed to demonstrate results, or provide information on how well the organization is achieving its goals; (2) limited to a vital few, and balanced across priorities; and (3) used by management to improve performance. However, the set of measures DOD has developed for the risk management framework do not adequately address these principles. While DOD established four risk quadrants and developed performance goals and measures of two types—activity measures (measures to track initiatives) and performance measures—the majority of its measures do not provide sufficient information to monitor performance against the risk quadrants’ goals.

First, DOD officials acknowledge that establishing department-level measures for the framework that demonstrate results is still a work in progress, as the majority of the risk management framework’s measures require further development or refinement. In fact, as shown in table 1, 44 of the 77 department-level measures for all four quadrants, or over 50 percent, are activity measures. According to DOD sources, activity measures are to result in a new performance measure, a new baseline or benchmark, or define a new capability, rather than monitor a specific annual performance target. Once these activities are completed, DOD officials stated that the department will be better able to monitor department-level performance against strategic goals. However, our analysis found that the activity measures, as defined in DOD’s external reports, typically do not provide sufficient information to monitor the department’s progress in achieving the stated goal they are to measure, such as developing a new performance measure or baseline. The desired outcomes for activity measures generally state that a task was or will be completed by a certain date but they do not provide sufficient information on whether the activity is on schedule, the interdependencies among tasks, or the contribution toward enhancing the department’s performance. Therefore, Congress and other external stakeholders lack information and adequate assurances that DOD is making progress in implementing a risk-based and results-oriented management approach to making investment decisions.

Second, DOD’s department-level performance measures are still a work in progress in that these measures do not provide a well-rounded depiction of DOD’s performance. In our previous work, we have found that performance measurement efforts that are not balanced across priorities may skew an agency’s performance and keep its senior leadership from seeing the whole picture. For example, in developing department-level measures for the risk management framework, DOD appears to have overemphasized its force management priorities at the expense of operational risk. As illustrated in table 2, the operational risk quadrant has no performance measures, while the force management risk quadrant has a total of 36 measures, including 15 activity measures and 21 performance measures.

In providing technical comments to a draft of this report, DOD objected to our recoding of five department-level performance measures as activity measures. We recoded these measures because they tracked milestones and events, which corresponded to DOD’s definition of an activity measure. The measures we recoded addressed the following: a civilian human resources strategic plan, a military human resources strategic plan, monitor the status of defense technology objectives, strategic transformation appraisal, and support acquisition excellence goals.

Finally, DOD officials indicated that DOD is systematically using performance measures to monitor progress and improve performance for only one risk quadrant, although individual measures under the other three risk quadrants may be monitored. We have found that leading organizations use performance information to improve organizational performance and identify performance gaps, and to provide incentives that reinforce a results-oriented management approach. According to DOD officials, the force management quadrant is the only quadrant that is managed by one individual and one office—the Under Secretary of Defense for Personnel and Readiness and his office. These officials stated that this situation is a critical factor in the progress DOD has made in systematically monitoring performance across the force management quadrant on a routine basis. For example, officials stated that the Under Secretary of Defense personally leads quarterly monitoring sessions on the force management quadrant’s performance. DOD officials also told us that the Under Secretary of Defense for Personnel and Readiness has greatly facilitated this monitoring by developing a centralized database to capture the performance data used to track DOD’s performance in meeting the quadrant’s goals. Unless all of the risk management framework’s quadrants are systematically monitored, implementation of the framework may be hindered and the framework risks becoming a paper-driven, compliance exercise. Indeed, one DOD official told us that he views the risk management framework and its measures as a “reporting drill” and, in addition, his office would not change its processes if DOD was to no longer use the framework.

Cascading the Risk Management Framework’s Goals and Measures Is an Ongoing Effort

DOD is still in the process of cascading the risk management framework’s goals and measures to the services. We have found that leading organizations seek to establish clear hierarchies of goals and measures that cascade down so that subordinate units have straightforward roadmaps to demonstrate how their activities contribute to meeting the organization’s strategy. According to DOD officials, all of the services are attempting to align their existing performance measures with the department-level performance goals and measures. However, service officials said that it is challenging to cascade the department-level activity measures, because these measures represent very broad initiatives that may not be applicable at all DOD levels. Officials from one service said they have had to develop new measures to align with the department-level measures, because they had been assessing performance with fewer measures than the Office of the Secretary of Defense had developed.

Developing a Strategic Plan with Clear Linkages between the Risk Management Framework and Strategic Goals Is a Critical Next Step

The risk management framework’s performance goals and measures are not clearly linked—a key principle of results-oriented management—to a coherent strategic plan. The development of such a strategic plan is a critical next step in using a risk-based and results-oriented approach to making investment decisions. Without these linkages, DOD cannot easily demonstrate how achievement of a performance goal or measure contributes to the achievement of strategic goals and ultimately the organization’s mission. Our previous work indicated that DOD’s strategic plan, the 2001 QDR, did not provide a sound foundation for the risk management framework. We reported that the usefulness of the 2001 QDR was limited by the lack of focus on longer-term threats and requirements for critical support capabilities, and provided few insights into how future threats and planned technical advances could affect future force requirements. In turn, this lack of focus and insight limited the QDR’s usefulness as a foundation for fundamentally reassessing U.S. defense plans and programs and for balancing resources across near- and midterm risks.

DOD officials indicated that DOD has not yet defined the linkages between the risk management framework’s performance goals and the strategic goals in the 2001 QDR. Furthermore, the Defense Business Board’s official minutes for its July 28, 2005, meeting contained a recommendation that the Secretary of Defense define department-level objectives, which should then be cascaded down the department. In discussing the ongoing 2005 QDR, DOD stated that although the department would continue its efforts to do so, establishing these linkages was very challenging because of the size and scope of DOD’s operations. However, as suggested by the Defense Business Board and our previous work, if DOD’s strategic plan is to drive the department’s operations, a straightforward linkage is needed among strategic goals, annual performance goals, and day-to-day activities. The ongoing 2005 QDR offers DOD the opportunity to strengthen its strategic planning.

Although Risk Considered, Linkages Between the Risk Management Framework and Budget Are Unclear

According to DOD officials, the department has begun to consider risk in its investment decision making; however, the full extent to which the framework’s risk-based and results-oriented approach has been linked to the fiscal year 2006 budget cycle is unclear. Our work indicates that leading organizations link strategy to the budget process through results- oriented management to evaluate potential investments or initiatives.

DOD sources indicated that the department has begun to consider risk during its usual cycle of investment decision making. For example, according to DOD sources, the Secretary of Defense articulated broad areas for increasing or decreasing risk under each quadrant in the fiscal years 2006–2011 planning guidance, leaving it up to the defense components to decide how to structure their investment decisions within those broad areas consistent with the Secretary’s risk guidance. In addition, DOD officials stated that the framework has increased awareness within the department on the need to balance risk over time. For example, when DOD reduced the fiscal years 2006–2011 defense program by $30 billion, DOD officials stated that the department did not take the traditional budgetary approach of cutting each defense component’s budget by a certain percentage. Instead, DOD officials stated that the Secretary of Defense used a collaborative approach with service participation to discuss where to take the budget reductions and how these cuts would affect risk, although DOD officials offered various views on how extensively the framework was used to make those decisions.

Second, DOD required that the services and other defense components offset any funding increase in one area with a funding decrease in another area for the fiscal years 2006–2007 budget submission. According to DOD officials, risk—whether on the basis of “professional judgment” or analysis—was considered in these deliberations. For example, the Army’s plan for fiscal years 2006–2023 articulated areas for increasing risks so that it could decrease risk in the operational risk dimension by investing in current capacity.

However, the fiscal year 2006 budget submission does not include any specific information on how DOD systematically identified or assessed departmental risks to establish DOD-wide investment priorities. For example, the military services’ share of the Future Years Defense Program (FYDP) remained relatively unchanged from fiscal year 2005 to fiscal year 2006 (see table 3), providing one indication that the risk management framework may not yet be a useful tool for balancing departmental risks across the services.

DOD has reported on the risk management framework in the department’s GPRA and other reporting requirements. For example, the fiscal year 2004 Performance and Accountability Report describes what DOD is doing, or plans to do, to define, measure, and monitor performance goals in the four risk quadrants but does not discuss the implementation status of the risk management framework. Furthermore, the fiscal year 2004 report, the most recent available, provided insufficient information to assist Congress in overseeing how DOD plans to prioritize investment decisions within or across the risk quadrants. Without more detailed information, Congress may have insufficient transparency into how DOD has identified and assessed risks and made trade-offs in its investment decision making. In addition, we reported in May 2004 that congressional visibility over investment decision making also was limited by the absence of linkages between the risk management framework and military capabilities planning and the FYDP. Because the FYDP lacked these linkages, we concluded that decision makers could not use it to determine how a proposed increase in capability would affect the risk management framework.

Our work also has shown that the FYDP may understate the costs of weapon system programs; therefore, DOD may be starting more programs than it can afford. For example, our assessment of 54 major programs, representing an investment of over $800 billion, found that the majority of these programs were costing more and taking longer to develop than planned. Problems occurred because of DOD’s overly optimistic planning assumptions about the long-term costs of weapon system programs and its failure to capture early on the requisite knowledge that is needed to efficiently and effectively manage program risks. When DOD has too many programs competing for funding and approves programs with low levels of knowledge, it is accepting the attendant likely adverse cost and schedule risks. As a result, it will probably get fewer quantities for the same investment or face difficult choices on which investments it cannot afford to pursue. The findings of our work suggest that having a departmentwide investment strategy for weapon systems, to allocate resources across investment priorities, would help reduce these risks.

Cultural Resistance, Combined with the Lack of Leadership, Implementation Goals, and Process Integration, Affects DOD’s Implementation of the Risk Management Framework

Four key challenges impede DOD’s progress toward implementing the risk management framework. The first implementation challenge facing DOD is overcoming cultural resistance to change in a department as massive, complex, and decentralized as DOD. The second challenge is the lack of sustained leadership, and the third challenge is the absence of implementation goals and timelines. These challenges relate to DOD’s failure to follow crucial transformational steps. The fourth challenge— integrating the risk management framework with decision support processes and related reform initiatives, into a coherent, unified management approach for the department—relates to key results-oriented management practices. Unless DOD addresses these challenges and successfully implements the risk management framework, or a similar approach, it may continue to experience (1) a mismatch between programs and budgets, and (2) the proportional, rather than strategic, allocation of resources to the services.

Transforming DOD’s Organizational Culture Is a Significant Challenge

Transforming DOD’s organizational culture—from a focus on inputs and programs to strategically balancing investment risks and monitoring outcomes across the department—through the implementation of the risk management framework is a significant challenge for the department for several reasons. First, as we noted in our 21st Century Challenges report, to successfully transform, DOD needs to overcome the inertia of various organizations, policies, and practices that became rooted in the Cold War era. The department’s expense, size, and complexity, however, make overcoming this resistance and inertia difficult. In fiscal year 2004, DOD reported that its operations involved $1.2 trillion in assets, $1.7 trillion in liabilities, over $605 billion in net cost of operations, and over 3.3 million military and civilian personnel. For fiscal year 2005, DOD received appropriations of about $417 billion. Moreover, execution of its operations spans a wide range of defense organizations, including the military services and their respective major commands and functional activities, numerous large defense agencies and field activities, and various combatant and joint operation commands, which are responsible for military operations for specific geographic regions or theaters of operations.

Second, DOD’s highly decentralized management structure is another contributing factor that makes cultural change difficult. Although under the authority, direction, and control of the Secretary of Defense, the military services have the legislative authority to organize, equip, and train the nation’s armed forces for combat under Title 10 of the U.S. Code. Furthermore, Congress directly appropriates funds to the services for programs and activities that support these purposes. In the opinion of knowledgeable DOD officials, this legislative authority has resulted in a culture that makes it difficult to develop department-level, or joint, management approaches. For example, the allocation of budgets on a proportional, rather than a strategic basis, among the military services is a long-standing budgetary problem that we have identified as a major management challenge for the department. In addition, the Joint Defense Capabilities Study, chartered by the Secretary of Defense in March 2003, made the following observations on how DOD’s organizational culture does not reinforce a departmental or joint approach to investment decision making and results management: DOD’s bottom-up strategic planning process did not support early senior leadership involvement and did not provide integrated departmentwide objectives, priorities, and roles as a framework for planning joint capabilities.

Service-centric focus on programs and weapons platforms resulted in a process that did not provide an accurate picture of joint needs, nor did it provide a consistent view of priorities and acceptable risks across the department.

The resulting budget did not optimize capabilities at either the department or the service level.

Accountability and feedback focused on monetary input rather than output; therefore, much of the information provided did not support the senior leaders’ decision making as it did not tell how well the department was being resourced to meet current and future mission requirements.

Lack of Sustained Leadership and Appropriate Accountability Has Challenged DOD’s Implementation of the Risk Management Framework

The lack of sustained leadership attention and appropriate accountability has challenged DOD’s progress in implementing the risk management framework. Our work has indicated that sustained leadership is a key transformational, or change management, practice. However, knowledgeable DOD officials indicated that DOD’s senior leadership did not provide sustained attention to the framework’s implementation. For example, a DOD official actively involved in the framework’s implementation stated that meetings with senior leadership that were to provide oversight of the framework’s implementation have not been regularly scheduled. DOD officials indicated that as a result of this lack of sustained leadership, DOD has not placed much emphasis on implementing the risk management framework at the department level. In addition, other DOD officials stated that changes in leadership have made it difficult to implement the risk management framework or develop performance measures. For example, since October 2004, DOD has experienced turnover in the following senior level positions, including the Deputy Secretary of Defense; the Under Secretary of Defense for Acquisition, Technology and Logistics; and the Director of Program Analysis and Evaluation (PA&E). Lacking sustained leadership attention, DOD officials offered conflicting perspectives on the status of the risk management framework with some officials suggesting that the framework had been overtaken by other performance-based or risk-based management initiatives while another suggested that the framework was primarily a compliance exercise. DOD officials also held differing perspectives on the purpose of the framework, including the beliefs that it was developed to monitor the Secretary of Defense’s priority areas or that it was a programming and budgeting tool.

Implementation of the risk management framework has also been challenged by the lack of clear lines of authority and appropriate accountability. No single individual or organization has been given overarching leadership responsibilities, authority, or the accountability for achieving the framework’s implementation. Instead, the responsibility for various tasks and performance measures have been spread among several organizations, including the Director, PA&E; the Under Secretary of Defense for Personnel and Readiness (P&R); and the Under Secretary of Defense, Comptroller/Chief Financial Officer.

We testified in April 2005 that as DOD embarks on large-scale change initiatives, the complexity and long-term nature of these initiatives require the development of an executive position capable of providing strong and sustained leadership—over a number of years and various administrations. For this reason, we have supported legislation to create a CMO at DOD to provide such sustained leadership. A CMO could also provide the leadership needed to successfully develop a risk-based and results-oriented management approach at DOD, such as the risk management framework.

Lack of Implementation Goals and Timelines Further Challenges DOD’s Implementation of Risk Management Framework

Accountability for implementation of the risk management framework also has been hindered by the absence of implementation goals and timelines with which to gauge progress. As we have previously reported, successful change management efforts use implementation goals and timelines to pinpoint performance shortfalls and gaps, suggest midcourse corrections, and build momentum by demonstrating progress. However, DOD’s limited guidance on the risk management framework did not establish implementation goals and timelines, nor did it require that implementation goals and timelines be developed. According to knowledgeable DOD officials, DOD did not see the need for implementation goals or timelines because the framework was not meant to change processes or create new ones, but rather was a management tool to improve upon investment decision-making processes. Regardless of how DOD classifies the risk management framework, we have found that implementation goals and timelines are essential to any transformational change, such as that envisioned by the Secretary of Defense with the risk management framework, because of the number of years it can take to complete the change. Moreover, the absence of implementation goals and timelines makes it difficult to determine whether progress has been made in implementing the framework over the last 2 ½ years, and whether DOD’s revisiting of the framework during the 2005 QDR represents an evolutionary progression or implementation delays.

Integrating the Risk Management Framework with Decision Support Processes and Related Reform Initiatives Is a Significant Challenge

DOD faces a significant challenge integrating the risk management framework with decision support processes for planning, programming, and budgeting and with related reform initiatives into a coherent, unified management approach. The goal of both risk management and results- oriented management is to integrate the systematic concern for risk and performance into the usual cycle of agency decision making and implementation. DOD’s challenge in meeting these goals is demonstrated by the number of initiatives, as shown in table 4, that DOD has put in place to improve investment decision making and manage performance results. For example, both capabilities planning and the risk management framework are to define risks and develop performance measures but, according to DOD officials, the department is still determining how to align capabilities planning with the risk management framework. Other initiatives, including GPRA and PART, are also to develop performance measures and DOD is still working on integrating these initiatives with the risk management framework and individual performance monitoring approaches of the services and other defense components into a single, integrated system. In December 2002, the Deputy Secretary of Defense issued a memorandum to correct this situation by requiring the alignment of the risk management framework and the President’s Management Agenda with DOD’s results-oriented management activities, including those associated with GPRA.

We note that these reform initiatives address key business processes within the department and that we have placed DOD’s overall business transformation on our list of federal programs and activities at high risk of waste, fraud, abuse, and mismanagement.

The Under Secretary of Defense for Acquisition, Technology and Logistics indicated that DOD plans to address the challenge associated with the integration of DOD’s planning, resourcing, and execution processes and initiatives, including the risk management framework. The Under Secretary stated that one task of the ongoing 2005 QDR was “strategic process integration.” The Under Secretary also stated that the department is planning to provide a roadmap with performance goals and timelines on how it will implement initiatives to improve strategic process integration. This roadmap is to be submitted with the 2005 QDR report to Congress in early 2006 with the fiscal year 2007 budget.

Conclusions

DOD has made some progress in implementing the risk management framework, including establishing risk quadrants and performance goals. However, more work will be required for DOD to be able to put in place a management tool, such as the risk management framework, to strategically balance the allocation of resources across the spectrum of its investment priorities against risk over time and to monitor performance. The development of performance measures that clearly demonstrate results and that are cascaded down throughout the department would enable DOD to provide a clear roadmap of how its activities at all levels contribute to meeting its strategic goals and would assist the department in aligning the core processes and resources of its four military services and multiple defense agencies to better support a departmental or joint approach to national security. Furthermore, the risk management framework cannot be fully implemented until its performance goals are clearly linked to DOD’s strategic planning goals. Unless a cause and effect relationship can be demonstrated between the department’s performance measures and strategic goals, the framework’s usefulness as a tool for monitoring DOD’s execution of its strategic plan and identifying performance goals will be severely restricted, if not eliminated. Furthermore, the fiscal year 2006 budget submission does not provide sufficient information on how DOD identified or assessed departmental risks to establish DOD-wide investment priorities; thus, the linkages between the framework and the budget are unclear. Without better measures, clear linkages, and greater transparency, DOD will be unable to fully measure progress in achieving strategic goals or demonstrate to Congress and others how it considered risks and made trade-off decisions, balancing needs and costs for weapon programs and other investment priorities.

The efforts of DOD’s senior leadership to establish a risk-based and results-oriented management approach have been impeded by some key challenges. The lack of sustained leadership and clear lines of accountability has hampered implementation of the risk management framework and the establishment and achievement of implementation goals and timelines. Strong and sustained leadership could enable DOD to overcome resistance to change that exists in a department as massive and complex as DOD. In addition, the establishment of implementation goals and timelines could enable DOD to determine what progress has been made in implementing the risk management framework. Furthermore, the successful integration of the risk management framework into DOD’s investment decision-making processes, including recent reform initiatives, could assist DOD in its overall transformation efforts. Until DOD develops a risk-based and results-oriented management approach for making investment decisions, it will likely continue to experience a mismatch between programs and budgets, and the proportional, rather than strategic, allocation of resources to the services.

Recommendations for Executive Action

To address the challenges associated with implementing the risk management framework, or a similar risk-based management approach, we recommend that the Secretary of Defense take the following four actions: develop or refine department-level performance measures so that they clearly demonstrate performance results and cascade those measures down throughout the department, assign clear leadership with accountability and authority to implement and sustain the risk management framework, develop implementation goals and timelines, and demonstrate the integration of the risk management framework with DOD’s decision support processes and related reform initiatives to improve investment decision making and manage performance results.

Agency Comments and Our Evaluation

In written comments on a draft of this report, DOD partially concurred with our four recommendations. DOD’s written comments are reprinted in their entirety in appendix II. DOD also provided technical comments, which we incorporated as appropriate.

DOD partially concurred with our first recommendation. DOD stated that it concurred with our recommendation that the Secretary of Defense refine department-level performance measures so that they clearly demonstrate results, but that it did not concur with the notion that effectively cascading the risk management framework has been inhibited by the current suite of performance measures. DOD noted that that a number of defense components—including the Army, DOD Comptroller, the Defense Logistics Agency, and the Defense Information Systems Agency—have successfully cascaded departmentwide strategic goals and implemented frameworks to measure their organization’s performance. DOD also believes that empowering the leadership at the component level to develop measures, while ensuring strategic alignment, is the most effective way of encouraging performance management and increasing its utility. In our report, we acknowledge that DOD has taken positive steps toward developing a performance monitoring system and cascading the framework’s goals and measures to defense components. However, our recommendation addresses limitations in those measures that currently hinder DOD’s ability to use the risk management framework as a management tool for aligning the components’ performance goals and measures with the risk management framework, or for strategic balancing investment decisions across the risk quadrants. For example, the majority of the risk management framework’s measures are activity measures, or initiatives, that do not monitor a specific annual performance target, nor do these measures provide sufficient information to determine whether the activity is on schedule or contributes to enhancing the department’s overall performance. Finally, our recommendation is not intended to suggest that DOD not empower the components to develop performance measures, but rather that DOD establish a clear hierarchy of goals and measures that provide straightforward roadmaps to demonstrate how the components’ activities contribute to meeting DOD’s strategic goals.

DOD partially concurred with our second recommendation that the Secretary of Defense assign clear leadership with accountability and authority to implement and sustain the risk management framework. DOD stated that, although it agrees that such leadership is key to any successful performance management system, the department’s senior executives provide sufficient leadership and accountability for implementing and sustaining the risk management framework. DOD also stated that it did not agree that a new organization or bureaucratic structure is needed to ensure successful implementation and sustainment of the risk management framework. We agree that DOD has assigned specific roles and responsibilities for goals and measures associated with the risk management framework to various high-level DOD officials. However, we based our recommendation on the fact that no single individual, with appropriate authority, was held responsible for ensuring that the risk management framework was implemented across the department. Further, our recommendation does not propose that DOD set up a new organization or bureaucratic structure, but, as stated in this report, we continue to believe that one way to provide strong and sustained leadership for change initiatives, such as the risk management framework, over a number of years and various administrations is to legislatively establish a CMO.

In partially concurring with our third recommendation to develop implementation goals and timelines, DOD agreed that tracking progress in implementing the risk management framework is a good management practice. DOD stated that it has established goals and timelines for the risk management framework that are unique to the individual metrics, or measures, and that because the risk management framework continually evolves over time, new metrics will be developed while others may be retired. As we stated in the report, successful change management efforts use implementation goals—such as, for example, linking the risk management framework to the budget—and timelines for meeting those goals, to pinpoint shortfalls and gaps, suggest midcourse corrections, and build momentum by demonstrating progress. Therefore, while DOD may continually refine the individual goals and measures associated with the framework’s risk quadrants, we believe that goals and timelines for the overall implementation of the framework across the department are essential for keeping this reform initiative on track.

DOD partially concurred with our fourth recommendation that the Secretary of Defense demonstrate the integration of the risk management framework with DOD’s decision support processes and related reform initiatives to improve investment decision making and manage performance results. DOD stated that the department is currently studying ways to further integrate the risk management framework with other decision support processes, but no single framework or decision model can provide all the necessary information or flexibility needed by the Secretary of Defense and his senior leadership team. We recognize that DOD’s senior leadership needs reliable information from a variety of sources and flexibility to make decisions among alternative actions or solutions. However, if the risk management framework is to successfully serve as a management tool to assist decision makers in formulating top- down strategy, balancing investment priorities against risk over time, measuring near- and midterm outputs against strategic goals, and focusing on actual performance results—as intended by DOD’s senior leadership— it is crucial that it be successfully integrated with DOD’s investment decision-making processes, including recent reform initiatives.

We are sending copies of this report to interested congressional committees; the Secretaries of Defense, Army, Navy, and Air Force; the Commandant of the Marine Corps; and the Director, Office of Management and Budget. We will also make copies available to others upon request. In addition, the report will be available at no charge on GAO’s Web site at http://www/gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-9619 or pickups@gao.gov. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Scope and Methodology

To assess to what extent the Department of Defense (DOD) has implemented the risk management framework, we obtained and analyzed DOD directives, briefings, and other documents that described the risk management framework’s purpose, implementation status, and performance measures. We also obtained and analyzed DOD’s 2001 Quadrennial Defense Review and annual strategic planning and budget documents. Moreover, we interviewed knowledgeable DOD and service officials involved with the implementation of the risk management framework. Specifically, we obtained testimonial evidence from officials representing the Office of the Secretary of Defense (OSD) offices—such as Program Analysis and Evaluation; Comptroller; Policy; Acquisition, Technology and Logistics; and Personnel and Readiness—the Joint Staff, the military services, and the Defense Business Board. To identify key risk- based and results-oriented management principles, we reviewed our prior reports and other relevant literature, including information on the balanced scorecard concept. For example, we identified characteristics of results-oriented performance measures. These characteristics focused on performance measures that are (1) designed to demonstrate results by providing information on how well the organization is achieving its goals; (2) limited to a vital few, and balanced across priorities; and (3) used by management to improve performance. As another example, risk-based and results-oriented management principles indicate that leading organizations seek to establish clear hierarchies of goals and measures that cascade down so that subordinate units have straightforward roadmaps to demonstrate how their activities contribute to meeting the organization’s strategy. We systematically analyzed and compared the risk management framework’s department-level performance measures with these characteristics. However, we did not validate the procedures that DOD has in place to ascertain the reliability of the data used to support the performance measures. Regarding strategic planning, these principles focused on (1) establishing clear linkages among strategic planning goals, resources, performance goals and measures and (2) integrating the consideration of risk into the usual cycle of agency decision making and implementation. While these principles do not cover all attributes associated with risk-based and results-oriented management approaches, we believe that they are the most important ones for assessing DOD’s progress in implementing the risk management framework.

To identify the most significant challenges, we reviewed our previous work on change management principles. We then compared DOD’s implementation of the risk management framework to sound change management principles and interviewed knowledgeable DOD officials about the challenges that faced the department in implementing the risk management framework. In addition, we reviewed our previous work to determine to what extent deficiencies in DOD’s overall business transformation efforts might influence the implementation of the risk management framework.

Our work was performed from October 2004 through September 2005 in accordance with generally accepted government auditing standards.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, David Moser, Assistant Director; Donna Byers; Gina Flacco; and Renee S. Brown made key contributions to this report.