Defense Management: Additional Actions Needed to Enhance DOD's Risk-Based Approach for Making Resource Decisions
Published: 2005-11-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Defense (DOD) is simultaneously conducting costly military operations and transforming its forces and business practices while it is also competing for resources in an increasingly constrained fiscal environment. As a result, GAO has advocated that DOD adopt a comprehensive threat or risk management approach as a framework for decision making. In its 2001 strategic plan, the Quadrennial Defense Review (QDR), DOD stated its intent to establish an approach--the risk management framework--to balance priorities against risk over time and monitor results against its strategic goals. GAO was asked to (1) assess the extent to which DOD has implemented the framework, including using it to make investment decisions, and (2) identify the most significant challenges DOD faces in implementing the framework, or a similar approach.
section_title: What GAO Found
paragraphs: DOD has taken some positive steps to implement the framework, but additional actions are needed before DOD can show real and sustainable progress in using a risk-based and results-oriented approach to strategically allocate resources across the spectrum of its investment priorities. For example, DOD defined four risk areas, and developed performance goals and department-level measures, but it needs to, among other things, further develop and refine the measures so that they clearly demonstrate results and provide a well-rounded depiction of departmental performance. DOD's current strategic plan and goals also are not clearly linked to the framework's performance goals and measures, and linkages between the framework and budget are also unclear. While DOD officials stated that risk was considered during the fiscal year 2006 budget cycle, DOD's budget submission does not specifically discuss how DOD identified or assessed risks to establish DOD-wide investment priorities. Without better measures, clear linkages, and greater transparency, DOD will be unable to fully measure progress in achieving strategic goals or demonstrate to Congress and others how it considered risks, and made trade-off decisions, balancing needs and costs for weapon programs and other investment priorities. DOD faces four challenges that have affected the implementation of the framework. First, DOD's organizational culture resists department-level approaches to priority setting and investment decisions. Second, sustained leadership, adequate transparency, and appropriate accountability are lacking. Further, no one individual or office has been assigned overall responsibility or sufficient authority for the framework's implementation. DOD also has not developed implementation goals or timelines with which to establish accountability, or measure progress. Finally, integrating the risk management framework with decision support processes and related reform initiatives into a coherent, unified management approach for the department is a challenge that DOD plans to address during the 2005 QDR. However, GAO has concerns about DOD's ability to follow through on this integration, because of its limited success in implementing other management reforms. Unless DOD successfully addresses these challenges and effectively implements the framework, or a similar approach, it will likely continue to experience (1) a mismatch between programs and budgets, and (2) a proportional, rather than strategic, allocation of resources to the services.

section_title: Letter
paragraphs: Among the 21st century challenges facing the Department of Defense  (DOD) and the nation as a whole are difficult decisions concerning how to  strike an affordable balance between current and future national security  needs and between national security and domestic needs. For example,  DOD is simultaneously maintaining a high pace of military operations for  combating terrorism and transforming its military forces and business  operations for the 21st century while it is also competing for federal  resources in an increasingly fiscally constrained environment. We have  advocated that DOD—as well as the rest of the federal government—adopt  a comprehensive threat or risk management approach as a framework for  decision making. This approach would fully link strategic goals to plans  and budgets; assess the values and risks of various courses of actions as a  tool for reexamining defense programs, setting priorities, and allocating  resources; and use performance measures to assess outcomes.
To its credit, DOD introduced a balanced scorecard for risk management,  commonly known as the risk management framework, in its strategic plan,  the 2001 Quadrennial Defense Review (QDR) report. The 2001 strategic  plan articulated the new administration’s emphasis on transforming  military forces and defense business practices to meet the emerging  challenges facing our nation. DOD intended the framework to be used as a  management tool to focus DOD’s efforts on implementing the defense  program as outlined in the strategic plan. In particular, DOD’s senior  leadership intended the risk management framework to assist decision  makers in formulating top-down strategy, balancing investment priorities  against risk over time, measuring near- and midterm outputs against  strategic goals, and focusing on actual performance results. According to  DOD officials, the risk management framework also was intended to  increase transparency within the department over the decision-making  process. During the ongoing 2005 QDR, DOD plans to refine the risk  management framework.
You asked us to examine the status of DOD’s efforts to adopt a risk-based  approach to decision making, given the emphasis that DOD was placing on  the risk management framework. In response, we (1) assessed the extent  to which DOD has implemented its risk management framework, including  the extent to which DOD has used the framework to make investment  decisions; and (2) identified the most significant challenges DOD faced in  implementing the risk management framework or a similar risk-based and  results-oriented management approach.
To assess the extent to which DOD has implemented the risk management  framework, we analyzed key documents, policy guidance, data, and  interview results, and compared the analysis to the principles for  managing risk and results identified in prior GAO reports. In addition, we  conducted interviews with DOD and service officials, and members of the  Joint Staff. We discussed the department’s progress in implementing the  risk management framework with members of the Defense Business  Board. We also analyzed DOD’s department-level performance goals and  measures that are associated with the risk management framework and  assessed how DOD reported that information externally. We did not  validate the appropriateness of the risk management framework’s risk  quadrants or the procedures that DOD has in place to ascertain the  reliability of performance data and we also did not assess the basis for  DOD’s investment decisions. To identify the most significant challenges  DOD faced in implementing the risk management framework, we analyzed  documents, data, and interview results, and compared the results of this  analysis to the key practices to assist mergers and organizational  transformation identified in prior GAO reports. A more detailed  discussion of our scope and methodology is presented in appendix I.
Our work was performed from October 2004 through September 2005 in  accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: In our report, High-Risk Series: An Update, we identified agencies’ lack  of comprehensive risk management strategies as an emerging challenge  for the federal government. Increasingly limited fiscal resources across the  federal government, coupled with the emerging requirements from the  changing security environment, emphasize the need for DOD to develop a  risk-based strategic investment approach. For this reason, we have  advocated that DOD adopt a comprehensive risk management approach  for decision making. Furthermore, DOD and other federal agencies are  required by statute to develop a results-oriented management approach to  strategically allocate resources on the basis of performance. The balanced  scorecard—a concept to balance an organization’s focus across financial,  customer, internal business, and learning and growth management areas— is one approach for developing results-oriented management that  government agencies have recently started to adopt. At the direction of  the Secretary of Defense, DOD developed a risk management framework  that DOD later aligned with its results-oriented management activities  through a DOD balanced scorecard.
subsections: section_title: Risk Management Is an Emerging 21st Century Challenge
paragraphs: An emerging challenge for the federal government involves the need for  the completion of comprehensive national threat and risk assessments in a  variety of areas. For example, emerging requirements from the changing  security environment, coupled with increasingly limited fiscal resources  across the federal government, emphasize the need for agencies to adopt a  sound approach to establishing resource decisions. We have advocated  that the federal government, including DOD, adopt a comprehensive threat  or risk management approach as a framework for decision making that  fully links strategic goals to plans and budgets, assesses values and risks  of various courses of actions as a tool for setting priorities and allocating  resources, and provides for the use of performance measures to assess  outcomes. Based on our review of the literature, as shown in figure 1, the  goal of risk management is to integrate systematic concern for risk into  the usual cycle of agency decision making and implementation.
A risk management cycle represents a series of analytical and managerial  steps, basically sequential, that can be used to assess risk, evaluate  alternatives for reducing risks, choose among those alternatives,  implement the alternatives, monitor their implementation, and continually  use new information to adjust and revise the assessments and actions, as  needed. Adoption of a risk management cycle such as this can aid in  assessing risk by determining which vulnerabilities should be addressed,  and how they should be addressed, within available resources. For the  purposes of this report, we focused on the stages of the risk management  cycle that involve DOD’s actions to set strategic goals and objectives,  establish investment priorities based on risk assessments, and  implementation and monitoring.
Risk management’s objectives are essentially the same as those of good  management, and they are consistent with the broad economy and  efficiency objectives of good government—namely, to provide better  outcomes for the same amount of money, or to provide the same  outcomes with less money. Therefore, risk management’s objectives are  also compatible with those of the federal government’s results-oriented  management approach, which was enacted in the Government  Performance and Results Act (GPRA) of 1993, and the balanced  scorecard approach. Congress enacted GPRA to focus the federal  government on achieving results through the creation of clear links  between the process of allocating scarce resources and an agency’s  strategic goals, or the expected results to be achieved with those  resources. Building on GPRA’s foundation, the current administration has  taken steps to strengthen the integration of budget, cost, and performance  information by including budget and performance integration as one of its  management initiatives under the umbrella of the President’s Management  Agenda. The Budget and Performance Integration initiative includes  efforts such as the Program Assessment Rating Tool (PART), improving  outcome measures, and improving monitoring of program performance.  The balanced scorecard approach is a management tool that some federal  agencies have adopted to help them translate the strategy set forth in a  results-oriented management approach into the operational objectives that  drive both behavior and performance. The balanced scorecard consists of  four management areas that organizations should focus on—financial,  customer, internal business, and learning and growth.
section_title: DOD’s 2001 Strategic Plan Outlines a New Risk Management Framework
paragraphs: DOD introduced the risk management framework in its strategic plan, the  2001 QDR report. The 2001 strategic plan articulated the new  administration’s emphasis on transforming military forces and defense  business practices to meet the changing threats facing our nation. In his  guidance to the department for the 2001 QDR strategic planning process,  the Secretary of Defense stated the need for DOD to use a risk mitigation  approach for balancing force, resource, and modernization requirements  across defense planning timelines. This guidance also stated that DOD  must include the identification of output-based measures to reduce  inefficiencies through the department in any approach to risk  management. Building on the guidance, the 2001 QDR outlined DOD’s risk  management framework. According to the QDR, the framework would  enable DOD to address the tension between preparing for future threats  and meeting the demands of the present with finite resources. It was also  intended to ensure that DOD was sized, shaped, postured, committed, and  managed with a view toward accomplishing the strategic plan’s defense  policy goals.
DOD adapted the balanced scorecard concept to the risk management  framework by substituting the four dimensions of risk—force  management, operational, future challenges, and institutional—for the  scorecard’s four management areas. The risk management framework was  to be a transformational tool that would provide a balanced perspective of  the organization’s execution of strategy and ensure a top-down approach.  The 2002 policy guidance also designated four preliminary performance  goals for each of the four risk quadrants. In addition, the guidance  required that performance goals and measures were to be cascaded to the  services and defense agencies. Figure 2 shows a comparison, as provided  by DOD.
section_title: Despite Positive Steps, Additional Actions Needed to Fully Implement the Risk Management Framework
paragraphs: Despite positive steps, DOD needs to take additional actions before the  risk management framework is fully implemented and DOD can  demonstrate real and sustainable progress in using a risk-based and  results-oriented approach to strategically allocate resources across the  spectrum of its investment priorities. For example, DOD is still in the  process of developing department-level measures for the framework that  address results-based management principles, such as linking performance  information to strategic goals so that this information can be used to  monitor performance results and determine how well the department is  doing in achieving its strategy. Without more results-oriented performance  measures, DOD may be unable to provide the services and other defense  components with clear roadmaps of how their activities contribute to  meeting DOD’s strategic goals. In addition, the framework’s performance  goals and measures are not clearly linked to DOD’s current strategic plan  and strategic goals. Furthermore, the extent to which the risk management  framework is linked to the budget cycle is unclear. Without better  measures, clear linkages, and greater transparency, DOD will be unable to  fully measure progress in achieving strategic goals or demonstrate to  Congress and others how it considered risks and made trade-offs in  making investment decisions.
subsections: section_title: Developing a Set of Measures That Can Be Used to Monitor Performance Is a Work in Progress
paragraphs: DOD has taken positive steps toward developing measures for each of the  performance goals under the framework’s four risk quadrants; however,  developing a set of measures that can be used to monitor performance  results is still a work in progress. Based on GAO’s prior work on results- based management principles, we found that leading organizations’  performance measures are: (1) designed to demonstrate results, or provide  information on how well the organization is achieving its goals; (2) limited  to a vital few, and balanced across priorities; and (3) used by management  to improve performance. However, the set of measures DOD has  developed for the risk management framework do not adequately address  these principles. While DOD established four risk quadrants and  developed performance goals and measures of two types—activity  measures (measures to track initiatives) and performance measures—the  majority of its measures do not provide sufficient information to monitor  performance against the risk quadrants’ goals.
First, DOD officials acknowledge that establishing department-level  measures for the framework that demonstrate results is still a work in  progress, as the majority of the risk management framework’s measures  require further development or refinement. In fact, as shown in table 1, 44  of the 77 department-level measures for all four quadrants, or over 50  percent, are activity measures. According to DOD sources, activity  measures are to result in a new performance measure, a new baseline or  benchmark, or define a new capability, rather than monitor a specific  annual performance target. Once these activities are completed, DOD  officials stated that the department will be better able to monitor  department-level performance against strategic goals. However, our  analysis found that the activity measures, as defined in DOD’s external  reports, typically do not provide sufficient information to monitor the  department’s progress in achieving the stated goal they are to measure,  such as developing a new performance measure or baseline. The desired  outcomes for activity measures generally state that a task was or will be  completed by a certain date but they do not provide sufficient information  on whether the activity is on schedule, the interdependencies among  tasks, or the contribution toward enhancing the department’s  performance. Therefore, Congress and other external stakeholders lack  information and adequate assurances that DOD is making progress in  implementing a risk-based and results-oriented management approach to  making investment decisions.
Second, DOD’s department-level performance measures are still a work in  progress in that these measures do not provide a well-rounded depiction  of DOD’s performance. In our previous work, we have found that  performance measurement efforts that are not balanced across priorities  may skew an agency’s performance and keep its senior leadership from  seeing the whole picture. For example, in developing department-level  measures for the risk management framework, DOD appears to have  overemphasized its force management priorities at the expense of  operational risk. As illustrated in table 2, the operational risk quadrant has  no performance measures, while the force management risk quadrant has  a total of 36 measures, including 15 activity measures and 21 performance  measures.
In providing technical comments to a draft of this report, DOD objected to  our recoding of five department-level performance measures as activity  measures. We recoded these measures because they tracked milestones  and events, which corresponded to DOD’s definition of an activity  measure. The measures we recoded addressed the following: a civilian human resources strategic plan, a military human resources strategic plan, monitor the status of defense technology objectives, strategic transformation appraisal, and   support acquisition excellence goals.
Finally, DOD officials indicated that DOD is systematically using  performance measures to monitor progress and improve performance for  only one risk quadrant, although individual measures under the other three  risk quadrants may be monitored. We have found that leading  organizations use performance information to improve organizational  performance and identify performance gaps, and to provide incentives that  reinforce a results-oriented management approach. According to DOD  officials, the force management quadrant is the only quadrant that is  managed by one individual and one office—the Under Secretary of  Defense for Personnel and Readiness and his office. These officials stated  that this situation is a critical factor in the progress DOD has made in  systematically monitoring performance across the force management  quadrant on a routine basis. For example, officials stated that the Under  Secretary of Defense personally leads quarterly monitoring sessions on the  force management quadrant’s performance. DOD officials also told us that  the Under Secretary of Defense for Personnel and Readiness has greatly  facilitated this monitoring by developing a centralized database to capture  the performance data used to track DOD’s performance in meeting the  quadrant’s goals. Unless all of the risk management framework’s quadrants  are systematically monitored, implementation of the framework may be  hindered and the framework risks becoming a paper-driven, compliance  exercise. Indeed, one DOD official told us that he views the risk  management framework and its measures as a “reporting drill” and, in  addition, his office would not change its processes if DOD was to no  longer use the framework.
section_title: Cascading the Risk Management Framework’s Goals and Measures Is an Ongoing Effort
paragraphs: DOD is still in the process of cascading the risk management framework’s  goals and measures to the services. We have found that leading  organizations seek to establish clear hierarchies of goals and measures  that cascade down so that subordinate units have straightforward  roadmaps to demonstrate how their activities contribute to meeting the  organization’s strategy. According to DOD officials, all of the services are  attempting to align their existing performance measures with the  department-level performance goals and measures. However, service  officials said that it is challenging to cascade the department-level activity  measures, because these measures represent very broad initiatives that  may not be applicable at all DOD levels. Officials from one service said  they have had to develop new measures to align with the department-level  measures, because they had been assessing performance with fewer  measures than the Office of the Secretary of Defense had developed.
section_title: Developing a Strategic Plan with Clear Linkages between the Risk Management Framework and Strategic Goals Is a Critical Next Step
paragraphs: The risk management framework’s performance goals and measures are  not clearly linked—a key principle of results-oriented management—to a  coherent strategic plan. The development of such a strategic plan is a  critical next step in using a risk-based and results-oriented approach to  making investment decisions. Without these linkages, DOD cannot easily  demonstrate how achievement of a performance goal or measure  contributes to the achievement of strategic goals and ultimately the  organization’s mission. Our previous work indicated that DOD’s strategic  plan, the 2001 QDR, did not provide a sound foundation for the risk  management framework. We reported that the usefulness of the 2001  QDR was limited by the lack of focus on longer-term threats and  requirements for critical support capabilities, and provided few insights  into how future threats and planned technical advances could affect future  force requirements. In turn, this lack of focus and insight limited the  QDR’s usefulness as a foundation for fundamentally reassessing U.S.  defense plans and programs and for balancing resources across near- and  midterm risks.
DOD officials indicated that DOD has not yet defined the linkages between  the risk management framework’s performance goals and the strategic  goals in the 2001 QDR. Furthermore, the Defense Business Board’s official  minutes for its July 28, 2005, meeting contained a recommendation that  the Secretary of Defense define department-level objectives, which should  then be cascaded down the department. In discussing the ongoing 2005  QDR, DOD stated that although the department would continue its efforts  to do so, establishing these linkages was very challenging because of the  size and scope of DOD’s operations. However, as suggested by the Defense  Business Board and our previous work, if DOD’s strategic plan is to drive  the department’s operations, a straightforward linkage is needed among  strategic goals, annual performance goals, and day-to-day activities. The  ongoing 2005 QDR offers DOD the opportunity to strengthen its strategic  planning.
section_title: Although Risk Considered, Linkages Between the Risk Management Framework and Budget Are Unclear
paragraphs: According to DOD officials, the department has begun to consider risk in  its investment decision making; however, the full extent to which the  framework’s risk-based and results-oriented approach has been linked to  the fiscal year 2006 budget cycle is unclear. Our work indicates that  leading organizations link strategy to the budget process through results- oriented management to evaluate potential investments or initiatives.
DOD sources indicated that the department has begun to consider risk  during its usual cycle of investment decision making. For example,  according to DOD sources, the Secretary of Defense articulated broad  areas for increasing or decreasing risk under each quadrant in the fiscal  years 2006–2011 planning guidance, leaving it up to the defense  components to decide how to structure their investment decisions within  those broad areas consistent with the Secretary’s risk guidance. In  addition, DOD officials stated that the framework has increased awareness  within the department on the need to balance risk over time. For example,  when DOD reduced the fiscal years 2006–2011 defense program by  $30 billion, DOD officials stated that the department did not take the  traditional budgetary approach of cutting each defense component’s  budget by a certain percentage. Instead, DOD officials stated that the  Secretary of Defense used a collaborative approach with service  participation to discuss where to take the budget reductions and how  these cuts would affect risk, although DOD officials offered various views  on how extensively the framework was used to make those decisions.
Second, DOD required that the services and other defense components  offset any funding increase in one area with a funding decrease in another  area for the fiscal years 2006–2007 budget submission. According to DOD  officials, risk—whether on the basis of “professional judgment” or  analysis—was considered in these deliberations. For example, the Army’s  plan for fiscal years 2006–2023 articulated areas for increasing risks so  that it could decrease risk in the operational risk dimension by investing in  current capacity.
However, the fiscal year 2006 budget submission does not include any  specific information on how DOD systematically identified or assessed  departmental risks to establish DOD-wide investment priorities. For  example, the military services’ share of the Future Years Defense Program  (FYDP) remained relatively unchanged from fiscal year 2005 to fiscal year  2006 (see table 3), providing one indication that the risk management  framework may not yet be a useful tool for balancing departmental risks  across the services.
DOD has reported on the risk management framework in the department’s  GPRA and other reporting requirements. For example, the fiscal year 2004  Performance and Accountability Report describes what DOD is doing, or  plans to do, to define, measure, and monitor performance goals in the four  risk quadrants but does not discuss the implementation status of the risk  management framework. Furthermore, the fiscal year 2004 report, the  most recent available, provided insufficient information to assist Congress  in overseeing how DOD plans to prioritize investment decisions within or  across the risk quadrants. Without more detailed information, Congress  may have insufficient transparency into how DOD has identified and  assessed risks and made trade-offs in its investment decision making. In  addition, we reported in May 2004 that congressional visibility over  investment decision making also was limited by the absence of linkages  between the risk management framework and military capabilities  planning and the FYDP. Because the FYDP lacked these linkages, we  concluded that decision makers could not use it to determine how a  proposed increase in capability would affect the risk management  framework.
Our work also has shown that the FYDP may understate the costs of  weapon system programs; therefore, DOD may be starting more programs  than it can afford. For example, our assessment of 54 major programs,  representing an investment of over $800 billion, found that the majority of  these programs were costing more and taking longer to develop than  planned. Problems occurred because of DOD’s overly optimistic planning  assumptions about the long-term costs of weapon system programs and its  failure to capture early on the requisite knowledge that is needed to  efficiently and effectively manage program risks. When DOD has too many  programs competing for funding and approves programs with low levels of  knowledge, it is accepting the attendant likely adverse cost and schedule  risks. As a result, it will probably get fewer quantities for the same  investment or face difficult choices on which investments it cannot afford  to pursue. The findings of our work suggest that having a departmentwide  investment strategy for weapon systems, to allocate resources across  investment priorities, would help reduce these risks.
section_title: Cultural Resistance, Combined with the Lack of Leadership, Implementation Goals, and Process Integration, Affects DOD’s Implementation of the Risk Management Framework
paragraphs: Four key challenges impede DOD’s progress toward implementing the risk  management framework. The first implementation challenge facing DOD  is overcoming cultural resistance to change in a department as massive,  complex, and decentralized as DOD. The second challenge is the lack of  sustained leadership, and the third challenge is the absence of  implementation goals and timelines. These challenges relate to DOD’s  failure to follow crucial transformational steps. The fourth challenge— integrating the risk management framework with decision support  processes and related reform initiatives, into a coherent, unified  management approach for the department—relates to key results-oriented  management practices. Unless DOD addresses these challenges and  successfully implements the risk management framework, or a similar  approach, it may continue to experience (1) a mismatch between  programs and budgets, and (2) the proportional, rather than strategic,  allocation of resources to the services.
subsections: section_title: Transforming DOD’s Organizational Culture Is a Significant Challenge
paragraphs: Transforming DOD’s organizational culture—from a focus on inputs and  programs to strategically balancing investment risks and monitoring  outcomes across the department—through the implementation of the risk  management framework is a significant challenge for the department for  several reasons. First, as we noted in our 21st Century Challenges report,  to successfully transform, DOD needs to overcome the inertia of various  organizations, policies, and practices that became rooted in the Cold War  era. The department’s expense, size, and complexity, however, make  overcoming this resistance and inertia difficult. In fiscal year 2004, DOD  reported that its operations involved $1.2 trillion in assets, $1.7 trillion in  liabilities, over $605 billion in net cost of operations, and over 3.3 million  military and civilian personnel. For fiscal year 2005, DOD received  appropriations of about $417 billion. Moreover, execution of its operations  spans a wide range of defense organizations, including the military  services and their respective major commands and functional activities,  numerous large defense agencies and field activities, and various  combatant and joint operation commands, which are responsible for  military operations for specific geographic regions or theaters of  operations.
Second, DOD’s highly decentralized management structure is another  contributing factor that makes cultural change difficult. Although under  the authority, direction, and control of the Secretary of Defense, the  military services have the legislative authority to organize, equip, and train  the nation’s armed forces for combat under Title 10 of the U.S. Code.  Furthermore, Congress directly appropriates funds to the services for  programs and activities that support these purposes. In the opinion of  knowledgeable DOD officials, this legislative authority has resulted in a  culture that makes it difficult to develop department-level, or joint,  management approaches. For example, the allocation of budgets on a  proportional, rather than a strategic basis, among the military services is a  long-standing budgetary problem that we have identified as a major  management challenge for the department. In addition, the Joint Defense  Capabilities Study, chartered by the Secretary of Defense in March 2003,  made the following observations on how DOD’s organizational culture  does not reinforce a departmental or joint approach to investment  decision making and results management:  DOD’s bottom-up strategic planning process did not support early senior  leadership involvement and did not provide integrated departmentwide  objectives, priorities, and roles as a framework for planning joint  capabilities.
Service-centric focus on programs and weapons platforms resulted in a  process that did not provide an accurate picture of joint needs, nor did it  provide a consistent view of priorities and acceptable risks across the  department.
The resulting budget did not optimize capabilities at either the department  or the service level.
Accountability and feedback focused on monetary input rather than  output; therefore, much of the information provided did not support the  senior leaders’ decision making as it did not tell how well the department  was being resourced to meet current and future mission requirements.
section_title: Lack of Sustained Leadership and Appropriate Accountability Has Challenged DOD’s Implementation of the Risk Management Framework
paragraphs: The lack of sustained leadership attention and appropriate accountability  has challenged DOD’s progress in implementing the risk management  framework. Our work has indicated that sustained leadership is a key  transformational, or change management, practice. However,  knowledgeable DOD officials indicated that DOD’s senior leadership did  not provide sustained attention to the framework’s implementation. For  example, a DOD official actively involved in the framework’s  implementation stated that meetings with senior leadership that were to  provide oversight of the framework’s implementation have not been  regularly scheduled. DOD officials indicated that as a result of this lack of  sustained leadership, DOD has not placed much emphasis on  implementing the risk management framework at the department level. In  addition, other DOD officials stated that changes in leadership have made  it difficult to implement the risk management framework or develop  performance measures. For example, since October 2004, DOD has  experienced turnover in the following senior level positions, including the  Deputy Secretary of Defense; the Under Secretary of Defense for  Acquisition, Technology and Logistics; and the Director of Program  Analysis and Evaluation (PA&E). Lacking sustained leadership attention,  DOD officials offered conflicting perspectives on the status of the risk  management framework with some officials suggesting that the  framework had been overtaken by other performance-based or risk-based  management initiatives while another suggested that the framework was  primarily a compliance exercise. DOD officials also held differing  perspectives on the purpose of the framework, including the beliefs that it  was developed to monitor the Secretary of Defense’s priority areas or that  it was a programming and budgeting tool.
Implementation of the risk management framework has also been  challenged by the lack of clear lines of authority and appropriate  accountability. No single individual or organization has been given  overarching leadership responsibilities, authority, or the accountability for  achieving the framework’s implementation. Instead, the responsibility for  various tasks and performance measures have been spread among several  organizations, including the Director, PA&E; the Under Secretary of  Defense for Personnel and Readiness (P&R); and the Under Secretary of  Defense, Comptroller/Chief Financial Officer.
We testified in April 2005 that as DOD embarks on large-scale change  initiatives, the complexity and long-term nature of these initiatives require  the development of an executive position capable of providing strong and  sustained leadership—over a number of years and various  administrations. For this reason, we have supported legislation to create  a CMO at DOD to provide such sustained leadership. A CMO could also  provide the leadership needed to successfully develop a risk-based and  results-oriented management approach at DOD, such as the risk  management framework.
section_title: Lack of Implementation Goals and Timelines Further Challenges DOD’s Implementation of Risk Management Framework
paragraphs: Accountability for implementation of the risk management framework also  has been hindered by the absence of implementation goals and timelines  with which to gauge progress. As we have previously reported, successful  change management efforts use implementation goals and timelines to  pinpoint performance shortfalls and gaps, suggest midcourse corrections,  and build momentum by demonstrating progress. However, DOD’s  limited guidance on the risk management framework did not establish  implementation goals and timelines, nor did it require that implementation  goals and timelines be developed. According to knowledgeable DOD  officials, DOD did not see the need for implementation goals or timelines  because the framework was not meant to change processes or create new  ones, but rather was a management tool to improve upon investment  decision-making processes. Regardless of how DOD classifies the risk  management framework, we have found that implementation goals and  timelines are essential to any transformational change, such as that  envisioned by the Secretary of Defense with the risk management  framework, because of the number of years it can take to complete the  change. Moreover, the absence of implementation goals and timelines  makes it difficult to determine whether progress has been made in  implementing the framework over the last 2 ½ years, and whether DOD’s  revisiting of the framework during the 2005 QDR represents an  evolutionary progression or implementation delays.
section_title: Integrating the Risk Management Framework with Decision Support Processes and Related Reform Initiatives Is a Significant Challenge
paragraphs: DOD faces a significant challenge integrating the risk management  framework with decision support processes for planning, programming,  and budgeting and with related reform initiatives into a coherent, unified  management approach. The goal of both risk management and results- oriented management is to integrate the systematic concern for risk and  performance into the usual cycle of agency decision making and  implementation. DOD’s challenge in meeting these goals is demonstrated  by the number of initiatives, as shown in table 4, that DOD has put in place  to improve investment decision making and manage performance results.  For example, both capabilities planning and the risk management  framework are to define risks and develop performance measures but,  according to DOD officials, the department is still determining how to  align capabilities planning with the risk management framework. Other  initiatives, including GPRA and PART, are also to develop performance  measures and DOD is still working on integrating these initiatives with the  risk management framework and individual performance monitoring  approaches of the services and other defense components into a single,  integrated system. In December 2002, the Deputy Secretary of Defense  issued a memorandum to correct this situation by requiring the alignment  of the risk management framework and the President’s Management  Agenda with DOD’s results-oriented management activities, including  those associated with GPRA.
We note that these reform initiatives address key business processes  within the department and that we have placed DOD’s overall business  transformation on our list of federal programs and activities at high risk of  waste, fraud, abuse, and mismanagement.
The Under Secretary of Defense for Acquisition, Technology and Logistics  indicated that DOD plans to address the challenge associated with the  integration of DOD’s planning, resourcing, and execution processes and  initiatives, including the risk management framework. The Under  Secretary stated that one task of the ongoing 2005 QDR was “strategic  process integration.” The Under Secretary also stated that the department  is planning to provide a roadmap with performance goals and timelines on  how it will implement initiatives to improve strategic process integration.  This roadmap is to be submitted with the 2005 QDR report to Congress in  early 2006 with the fiscal year 2007 budget.
section_title: Conclusions
paragraphs: DOD has made some progress in implementing the risk management  framework, including establishing risk quadrants and performance goals.  However, more work will be required for DOD to be able to put in place a  management tool, such as the risk management framework, to  strategically balance the allocation of resources across the spectrum of its  investment priorities against risk over time and to monitor performance.  The development of performance measures that clearly demonstrate  results and that are cascaded down throughout the department would  enable DOD to provide a clear roadmap of how its activities at all levels  contribute to meeting its strategic goals and would assist the department  in aligning the core processes and resources of its four military services  and multiple defense agencies to better support a departmental or joint  approach to national security. Furthermore, the risk management  framework cannot be fully implemented until its performance goals are  clearly linked to DOD’s strategic planning goals. Unless a cause and effect  relationship can be demonstrated between the department’s performance  measures and strategic goals, the framework’s usefulness as a tool for  monitoring DOD’s execution of its strategic plan and identifying  performance goals will be severely restricted, if not eliminated.  Furthermore, the fiscal year 2006 budget submission does not provide  sufficient information on how DOD identified or assessed departmental  risks to establish DOD-wide investment priorities; thus, the linkages  between the framework and the budget are unclear. Without better  measures, clear linkages, and greater transparency, DOD will be unable to  fully measure progress in achieving strategic goals or demonstrate to  Congress and others how it considered risks and made trade-off decisions,  balancing needs and costs for weapon programs and other investment  priorities.
The efforts of DOD’s senior leadership to establish a risk-based and  results-oriented management approach have been impeded by some key  challenges. The lack of sustained leadership and clear lines of  accountability has hampered implementation of the risk management  framework and the establishment and achievement of implementation  goals and timelines. Strong and sustained leadership could enable DOD to  overcome resistance to change that exists in a department as massive and  complex as DOD. In addition, the establishment of implementation goals  and timelines could enable DOD to determine what progress has been  made in implementing the risk management framework. Furthermore, the  successful integration of the risk management framework into DOD’s  investment decision-making processes, including recent reform initiatives,  could assist DOD in its overall transformation efforts. Until DOD develops  a risk-based and results-oriented management approach for making  investment decisions, it will likely continue to experience a mismatch  between programs and budgets, and the proportional, rather than  strategic, allocation of resources to the services.
section_title: Recommendations for Executive Action
paragraphs: To address the challenges associated with implementing the risk  management framework, or a similar risk-based management approach,  we recommend that the Secretary of Defense take the following four  actions:    develop or refine department-level performance measures so that they  clearly demonstrate performance results and cascade those measures  down throughout the department,    assign clear leadership with accountability and authority to implement and  sustain the risk management framework,    develop implementation goals and timelines, and    demonstrate the integration of the risk management framework with  DOD’s decision support processes and related reform initiatives to  improve investment decision making and manage performance results.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, DOD partially concurred  with our four recommendations. DOD’s written comments are reprinted in  their entirety in appendix II. DOD also provided technical comments,  which we incorporated as appropriate.
DOD partially concurred with our first recommendation. DOD stated that  it concurred with our recommendation that the Secretary of Defense  refine department-level performance measures so that they clearly  demonstrate results, but that it did not concur with the notion that  effectively cascading the risk management framework has been inhibited  by the current suite of performance measures. DOD noted that that a  number of defense components—including the Army, DOD Comptroller,  the Defense Logistics Agency, and the Defense Information Systems  Agency—have successfully cascaded departmentwide strategic goals and  implemented frameworks to measure their organization’s performance.  DOD also believes that empowering the leadership at the component level  to develop measures, while ensuring strategic alignment, is the most  effective way of encouraging performance management and increasing its  utility. In our report, we acknowledge that DOD has taken positive steps  toward developing a performance monitoring system and cascading the  framework’s goals and measures to defense components. However, our  recommendation addresses limitations in those measures that currently  hinder DOD’s ability to use the risk management framework as a  management tool for aligning the components’ performance goals and  measures with the risk management framework, or for strategic balancing  investment decisions across the risk quadrants. For example, the majority  of the risk management framework’s measures are activity measures, or  initiatives, that do not monitor a specific annual performance target, nor  do these measures provide sufficient information to determine whether  the activity is on schedule or contributes to enhancing the department’s  overall performance. Finally, our recommendation is not intended to  suggest that DOD not empower the components to develop performance  measures, but rather that DOD establish a clear hierarchy of goals and  measures that provide straightforward roadmaps to demonstrate how the  components’ activities contribute to meeting DOD’s strategic goals.
DOD partially concurred with our second recommendation that the  Secretary of Defense assign clear leadership with accountability and  authority to implement and sustain the risk management framework. DOD  stated that, although it agrees that such leadership is key to any successful  performance management system, the department’s senior executives  provide sufficient leadership and accountability for implementing and  sustaining the risk management framework. DOD also stated that it did not  agree that a new organization or bureaucratic structure is needed to  ensure successful implementation and sustainment of the risk  management framework. We agree that DOD has assigned specific roles  and responsibilities for goals and measures associated with the risk  management framework to various high-level DOD officials. However, we  based our recommendation on the fact that no single individual, with  appropriate authority, was held responsible for ensuring that the risk  management framework was implemented across the department.  Further, our recommendation does not propose that DOD set up a new  organization or bureaucratic structure, but, as stated in this report, we  continue to believe that one way to provide strong and sustained  leadership for change initiatives, such as the risk management framework,  over a number of years and various administrations is to legislatively  establish a CMO.
In partially concurring with our third recommendation to develop  implementation goals and timelines, DOD agreed that tracking progress in  implementing the risk management framework is a good management  practice. DOD stated that it has established goals and timelines for the risk  management framework that are unique to the individual metrics, or  measures, and that because the risk management framework continually  evolves over time, new metrics will be developed while others may be  retired. As we stated in the report, successful change management efforts  use implementation goals—such as, for example, linking the risk  management framework to the budget—and timelines for meeting those  goals, to pinpoint shortfalls and gaps, suggest midcourse corrections, and  build momentum by demonstrating progress. Therefore, while DOD may  continually refine the individual goals and measures associated with the  framework’s risk quadrants, we believe that goals and timelines for the  overall implementation of the framework across the department are  essential for keeping this reform initiative on track.
DOD partially concurred with our fourth recommendation that the  Secretary of Defense demonstrate the integration of the risk management  framework with DOD’s decision support processes and related reform  initiatives to improve investment decision making and manage  performance results. DOD stated that the department is currently studying  ways to further integrate the risk management framework with other  decision support processes, but no single framework or decision model  can provide all the necessary information or flexibility needed by the  Secretary of Defense and his senior leadership team. We recognize that  DOD’s senior leadership needs reliable information from a variety of  sources and flexibility to make decisions among alternative actions or  solutions. However, if the risk management framework is to successfully  serve as a management tool to assist decision makers in formulating top- down strategy, balancing investment priorities against risk over time,  measuring near- and midterm outputs against strategic goals, and focusing  on actual performance results—as intended by DOD’s senior leadership— it is crucial that it be successfully integrated with DOD’s investment  decision-making processes, including recent reform initiatives.
We are sending copies of this report to interested congressional  committees; the Secretaries of Defense, Army, Navy, and Air Force; the  Commandant of the Marine Corps; and the Director, Office of Management  and Budget. We will also make copies available to others upon request. In  addition, the report will be available at no charge on GAO’s Web site at  http://www/gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-9619 or pickups@gao.gov. Contact points for our offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix III.
section_title: Appendix I: Scope and Methodology
paragraphs: To assess to what extent the Department of Defense (DOD) has  implemented the risk management framework, we obtained and analyzed  DOD directives, briefings, and other documents that described the risk  management framework’s purpose, implementation status, and  performance measures. We also obtained and analyzed DOD’s 2001  Quadrennial Defense Review and annual strategic planning and budget  documents. Moreover, we interviewed knowledgeable DOD and service  officials involved with the implementation of the risk management  framework. Specifically, we obtained testimonial evidence from officials  representing the Office of the Secretary of Defense (OSD) offices—such as  Program Analysis and Evaluation; Comptroller; Policy; Acquisition,  Technology and Logistics; and Personnel and Readiness—the Joint Staff,  the military services, and the Defense Business Board. To identify key risk- based and results-oriented management principles, we reviewed our prior  reports and other relevant literature, including information on the  balanced scorecard concept. For example, we identified characteristics of  results-oriented performance measures. These characteristics focused on  performance measures that are (1) designed to demonstrate results by  providing information on how well the organization is achieving its goals;  (2) limited to a vital few, and balanced across priorities; and (3) used by  management to improve performance. As another example, risk-based and  results-oriented management principles indicate that leading organizations  seek to establish clear hierarchies of goals and measures that cascade  down so that subordinate units have straightforward roadmaps to  demonstrate how their activities contribute to meeting the organization’s  strategy. We systematically analyzed and compared the risk management  framework’s department-level performance measures with these  characteristics. However, we did not validate the procedures that DOD has  in place to ascertain the reliability of the data used to support the  performance measures. Regarding strategic planning, these principles  focused on (1) establishing clear linkages among strategic planning goals,  resources, performance goals and measures and (2) integrating the  consideration of risk into the usual cycle of agency decision making and  implementation. While these principles do not cover all attributes  associated with risk-based and results-oriented management approaches,  we believe that they are the most important ones for assessing DOD’s  progress in implementing the risk management framework.
To identify the most significant challenges, we reviewed our previous  work on change management principles. We then compared DOD’s  implementation of the risk management framework to sound change  management principles and interviewed knowledgeable DOD officials  about the challenges that faced the department in implementing the risk  management framework. In addition, we reviewed our previous work to  determine to what extent deficiencies in DOD’s overall business  transformation efforts might influence the implementation of the risk  management framework.
Our work was performed from October 2004 through September 2005 in  accordance with generally accepted government auditing standards.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, David Moser, Assistant Director;  Donna Byers; Gina Flacco; and Renee S. Brown made key contributions to  this report.