Business Systems Modernization: DOD Needs to Fully Define Policies and Procedures for Institutionally Managing Investments
Published: 2007-05-11T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In 1995, GAO first designated the Department of Defense's (DOD) business systems modernization program as "high-risk," and continues to do so today. In 2004, Congress passed legislation reflecting prior GAO recommendations for DOD to adopt a corporate approach to information technology (IT) business system investment management. To support GAO's legislative mandate to review DOD's efforts, GAO assessed whether the department's corporate investment management approach comports with relevant federal guidance. In doing so, GAO applied its IT Investment Management framework and associated methodology, focusing on the framework's stages related to the investment management provisions of the Clinger-Cohen Act of 1996.
section_title: What GAO Found
paragraphs: DOD has established the management structures needed to effectively manage its business system investments, but it has not fully defined many of the related policies and procedures that GAO's IT Investment Management framework defines. Specifically, the department has defined four of nine practices that call for project-level policies and procedures, and one of the five practices that call for portfolio-level policies and procedures. For example, DOD has established an enterprisewide IT investment board responsible for defining and implementing its business system investment governance process, documented policies and procedures for ensuring that systems support ongoing and future business needs, developed procedures for identifying and collecting information about these systems to support investment selection and control, and assigned responsibility to an individual or a group for managing the development and modification of the business system portfolio selection criteria. However, DOD has not fully documented business system investment policies and procedures for directing investment board operations, selecting new investments, reselecting ongoing investments, integrating the investment funding and the investment selection processes, and developing and maintaining a complete business system investment portfolio(s). Regarding project-level investment management practices, DOD officials said that these are performed at the component level, and that departmental policies and procedures established for overseeing components' execution of these practices are sufficient. For portfolio-level practices, however, these officials stated that they intend to improve departmental policies and procedures for business system investments by, for example, establishing a single governance structure, but plans or time frames for doing so have not been established. Until DOD fully defines departmentwide policies and procedures for both individual projects and portfolios of projects, it risks selecting and controlling these business system investments in an inconsistent, incomplete, and ad hoc manner, which in turn reduces the chances that these investments will meet mission needs in the most cost-effective manner.

section_title: Letter
paragraphs: For decades, the Department of Defense (DOD) has been challenged in  modernizing its timeworn business systems. In 1995, we designated DOD’s  business systems modernization program as high risk, and we continue to  designate it as such today. As our research on public and private sector  organizations shows, one essential ingredient to a successful systems  modernization program is having an effective institutional approach to  managing information technology (IT) investments.
In May 2001, we recommended that the department establish a corporate  approach to investment control and decision making. Between 2001 and  2005, we reported that the department’s business systems modernization  program was still not being effectively managed, and we made additional  investment-related recommendations. Congress subsequently included  provisions in the Ronald W. Reagan National Defense Authorization Act  for Fiscal Year 2005 that reflected our recommendations, including those  for establishing and implementing effective business system investment  management structures and processes.
Between 2005 and 2006, we reported that DOD had made important  progress in establishing and implementing these structures and processes,  but that much remained to be accomplished relative to the act’s  requirements. For example, we reported that the department’s business  system investment approach was not institutionalized at all levels of the  department.
To support GAO’s legislative mandate to review DOD’s annual report on  its business systems modernization program, and as agreed with your  offices, the objective of this review was to determine whether DOD’s  corporate investment management approach comports with relevant  federal guidance. To accomplish our objective, we analyzed documents  and interviewed agency officials to determine whether DOD has developed  the structures, policies, and procedures associated with executing those  key practices in our IT Investment Management (ITIM) framework that  assist organizations in complying with the investment management  provisions of the Clinger-Cohen Act of 1996. This framework provides a  hierarchical maturity model for IT investment management and a method  for evaluating and assessing the maturity of an agency’s investment  management. We performed our work at DOD headquarters in Arlington,  Virginia, from August 2006 through April 2007 in accordance with  generally accepted government auditing standards. Details on our  objective, scope, and methodology are contained in appendix I.
subsections: section_title: Background
paragraphs: DOD is a massive and complex organization. To illustrate, the department  reported that its fiscal year 2006 operations involved approximately $1.4  trillion in assets and $2.0 trillion in liabilities, more than 2.9 million  military and civilian personnel, and $581 billion in net cost of operations.  To date, for fiscal year 2007, the department received appropriations of  about $501 billion. Organizationally, the department includes the Office of  the Secretary of Defense (OSD), the Chairman of the Joint Chiefs of Staff,  the military departments, numerous defense agencies and field activities,  and various unified combatant commands that are responsible for either  specific geographic regions or specific functions. (See fig. 1 for a  simplified depiction of DOD’s organizational structure.)
In support of its military operations, the department performs an  assortment of interrelated and interdependent business functions,  including logistics management, procurement, health care management,  and financial management. As we have previously reported, the systems  environment that supports these business functions is overly complex and  error-prone, and is characterized by (1) little standardization across the  department, (2) multiple systems performing the same tasks, (3) the same  data stored in multiple systems, and (4) the need for data to be entered  manually into multiple systems. Moreover, according to DOD, this systems  environment is comprised of approximately 3,100 separate business  systems. For fiscal year 2007, Congress appropriated approximately $15.7  billion to DOD, and for fiscal year 2008, DOD has requested about $15.9  billion in appropriated funds to operate, maintain, and modernize these  business systems and the associated infrastructures.
As we have previously reported, the department’s nonintegrated and  duplicative systems impair DOD’s ability to combat fraud, waste, and  abuse. In fact, DOD currently bears responsibility, in whole or in part, for  15 of our 27 high-risk areas. Eight of these areas are specific to DOD,  and the department shares responsibility for 7 other governmentwide high- risk areas. DOD’s business systems modernization is one of the high-risk  areas, and it is an essential enabler to addressing many of the department’s  other high-risk areas. For example, modernized business systems are  integral to the department’s efforts to address its financial, supply chain,  and information security management high-risk areas.
subsections: section_title: IT Investment Management Is Critical to Achieving Successful Systems Modernization
paragraphs: A corporate approach to IT investment management is characteristic of  successful public and private organizations. Recognizing this, Congress  enacted the Clinger-Cohen Act of 1996, which requires the Office of  Management and Budget (OMB) to establish processes to analyze, track,  and evaluate the risks and results of major capital investments in IT  systems made by executive agencies. In response to the Clinger-Cohen  Act and other statutes, OMB has developed policy and issued guidance for  the planning, budgeting, acquisition, and management of federal capital  assets. We have also issued guidance in this area, which defines  institutional structures, such as the IRBs; processes for developing  information on investments (such as costs and benefits); and practices to  inform management decisions (such as whether a given investment is  aligned with an enterprise architecture).
subsections: section_title: IT Investment Management: A Brief Description
paragraphs: IT investment management is a process for linking IT investment decisions  to an organization’s strategic objectives and business plans. Consistent  with this, the federal approach to IT investment management focuses on  selecting, controlling, and evaluating investments in a manner that  minimize risks while maximizing the return of investment.
During the selection phase, the organization (1) identifies and analyzes  each project’s risks and returns before committing significant funds to any  project and (2) selects those IT projects that will best support its mission  needs.
During the control phase, the organization ensures that projects, as they  develop and investment expenditures continue, meet mission needs at the  expected levels of cost and risk. If the project is not meeting expectations  or if problems arise, steps are quickly taken to address the deficiencies.
During the evaluation phase, expected results are compared with actual  results after a project has been fully implemented. This comparison is  done to (1) assess the project’s impact on mission performance,   (2) identify any changes or modifications to the project that may be  needed, and (3) revise the investment management process based on  lessons learned.
section_title: Overview of GAO’s ITIM Maturity Framework
paragraphs: Our ITIM framework consists of five progressive stages of maturity for any  given agency relative to selecting, controlling, and evaluating its  investment management capabilities. (See fig. 2 for the five ITIM stages  of maturity.) This framework is grounded in our research of IT investment  management practices of leading private and public sector organizations.  The maturity stages are cumulative; that is, to attain a higher stage, an  agency must institutionalize all of the critical processes at the lower  stages, in addition to the higher stage critical processes.
The framework can be used to assess the maturity of an agency’s  investment management processes and as a tool for organizational  improvement. The overriding purpose of the framework is to encourage  investment selection and control and to evaluate processes that promote  business value and mission performance, reduce risk, and increase  accountability and transparency. We have used the framework in several  of our evaluations, and a number of agencies have adopted it.
With the exception of the first stage, each maturity stage is composed of  “critical processes” that must be implemented and institutionalized for the  organization to achieve that stage. Each ITIM critical process consists of  “key practices”—to include organizational structures, policies, and  procedures—that must be executed to implement the critical process. It is  not unusual for an organization to perform key practices from more than  one maturity stage at the same time. However, our research shows that  agency efforts to improve investment management capabilities should  focus on implementing all lower-stage practices before addressing higher- stage practices.
In the ITIM framework, Stage 2 critical processes lay the foundation by  establishing successful, predictable, and repeatable investment control  processes at the project level. At this stage, the emphasis is on establishing  basic capabilities for selecting new IT projects; controlling projects so that  they finish predictably within the established cost, schedule, and  performance expectations; and identifying and mitigating exposure to risk.
Stage 3 is where the agency moves from project-centric processes to  portfolio-based processes and evaluates potential investments according  to how well they support the agency’s missions, strategies, and goals. This  stage focuses on continually assessing both proposed and ongoing  projects as part of complete investment portfolios—integrated and  competing sets of investment options. It also focuses on maintaining  mature, integrated selection (and reselection); control; and  postimplementation evaluation processes. This portfolio perspective  allows decision makers to consider the interaction among investments and  the contributions to organizational mission goals and strategies that could  be made by alternative portfolio selections, rather than to focus  exclusively on the balance between the costs and benefits of individual  investments. Organizations implementing Stages 2 and 3 practices have in  place capabilities that assist in establishing selection, control, and  evaluation structures, policies, procedures, and practices that are required  by the investment management provisions of the Clinger-Cohen Act.
Stages 4 and 5 require the use of evaluation techniques to continuously  improve both investment processes and portfolios to better achieve  strategic outcomes. At Stage 4, an organization has the capacity to conduct  IT succession activities and, therefore, can plan and implement the  deselection of obsolete, high-risk, or low-value IT investments. An  organization with Stage 5 maturity conducts proactive monitoring for  breakthrough technologies that will enable it to change and improve its  business performance.
section_title: Overview of DOD’s Corporate Approach for Identifying, Funding, and Acquiring All System Investments
paragraphs: DOD’s major system investments (i.e., weapon and business systems) are  governed by three management systems—the Joint Capabilities  Integration and Development System (JCIDS); the Planning, Programming,  Budgeting, and Execution (PPBE) system; and the Defense Acquisition  System (DAS).
JCIDS is a need-driven, capabilities-based approach to identify warfighting  needs and meet future joint forces challenges. It is intended to identify  future capabilities for DOD; address capability gaps and mission needs  recognized by the Joint Chiefs of Staff or derived from strategic guidance,  such as the National Security Strategy Report or Quadrennial Defense  Review; and identify alternative solutions by considering a range of  doctrine, organization, training, materiel, leadership and education,  personnel, and facilities solutions. According to DOD, the Joint Chiefs of  Staff, through the Joint Requirements Oversight Council, has primary  responsibility for defining and implementing JCIDS.
PPBE is a calendar-driven approach that is composed of four phases that  occur over a moving 2-year cycle. The four phases—planning,  programming, budgeting, and executing—define how budgets for each  DOD component and the department as a whole are created, vetted, and  executed. As recently reported, the components start programming and  budgeting for addressing a JCIDS-identified capability gap or mission need  several years before actual product development under DAS begins, and  before OSD formally reviews the components’ programming and  budgeting proposals (i.e., Program Objective Memorandums). Once  reviewed and approved, the financial details in the Program Objective  Memorandums become part of the President’s budget request to Congress.  During budget execution, components may submit program change  proposals or budget change proposals, or both (e.g., program cost  increases or schedule delays). According to DOD, the OSD Under  Secretary of Defense (Policy), the Director for Program Analysis and  Evaluation, and the Under Secretary of Defense (Comptroller) have  primary responsibility for defining and implementing the PPBE system.
DAS is described in the DOD Directive 5000.1 and the DOD Instruction  5000.2 and establishes the procedures for the Defense Acquisition  Management Framework, which consists of three event-based milestones  associated with five key program life-cycle phases. These five phases are  as follows: 1.  Concept Refinement: Intended to refine the initial JCIDS-validated  system solution (concept) and create a strategy for acquiring the  investment solution. A decision is made at the end of this phase  (milestone A decision) regarding whether to move to the next phase  (Technology Development).  2.  Technology Development: Intended to determine the appropriate set of  technologies to be integrated into the investment solution by  iteratively assessing the viability of various technologies while  simultaneously refining user requirements. Once the technology has  been demonstrated in a relevant environment, a decision is made at  the end of this phase (milestone B decision) regarding whether to  move to the next phase (System Development and Demonstration).  3.  System Development and Demonstration: Intended to develop a  system or a system increment and demonstrate through developer  testing that the system/system increment can function in its target  environment. A decision is made at the end of this phase (milestone C  decision) regarding whether to move to the next phase (Production  and Deployment).  4.  Production and Deployment: Intended to achieve an operational  capability that satisfies the mission needs, as verified through  independent operational test and evaluation, and ensures that the  system is implemented at all applicable locations.  5.  Operations and Support: Intended to operationally sustain the system  in the most cost-effective manner over its life cycle.
A key principle of DAS is that investments are assigned a category, where  programs of increasing dollar value and management interest are subject  to more stringent oversight. For example, Major Defense Acquisition  Programs (MDAP) and Major Automated Information Systems (MAIS)  are large, expensive programs subject to the most extensive statutory and  regulatory reporting requirements and, unless delegated, are reviewed by  acquisition boards at the DOD corporate level. Smaller and less risky  acquisitions are generally reviewed at the component executive or lower  levels. Another key principle is that DAS requires acquisition management  under the direction of a milestone decision authority. The milestone  decision authority—with support from the program manager and advisory  boards, such as the Defense Acquisition Board and the IT Acquisition  Board—determines the project’s baseline cost, schedule, and  performance commitments. The Under Secretary of Defense for  Acquisition, Technology, and Logistics (USD(AT&L)) has primary  responsibility for defining and implementing DAS.
section_title: DOD Business System Investments Are Subject to a Fourth Management System
paragraphs: DOD’s business system investments are also governed by a fourth  management system that addresses how these investments are reviewed,  certified, and approved for compliance with the business enterprise  priorities and activities outlined by the business enterprise architecture  (BEA). For the purposes of this report, we refer to this fourth management  system as the Business Investment Management System. This fourth  management system is described in the following text in terms of  governance entities, tiered accountability, and business system investment  certification reviews and approvals. According to DOD, these four  management systems are the means by which DOD selects, controls, and  evaluates its business system investments.
subsections: section_title: Business System Investment Roles and Responsibilities
paragraphs: In 2005, the department reassigned responsibility for providing executive  leadership for the direction, oversight, and execution of its business  systems modernization efforts to several entities. These entities and their  responsibilities include the following:    The Defense Business Systems Management Committee (DBSMC) serves  as the highest-ranking governance body for business systems  modernization activities.
The Principal Staff Assistants serve as the certification authorities for  business system modernizations in their respective core business  missions.
The IRBs are chartered by the Principal Staff Assistants and are the review  and decision-making bodies for business system investments in their  respective areas of responsibility.
The component pre-certification authority (PCA) is accountable for the  component’s business system investments and acts as the component’s  principal point of contact for communication with the IRBs.
The Business Transformation Agency (BTA) is responsible for leading and  coordinating business transformation efforts across the department. The  BTA is organized into seven directorates, one of which is the Defense  Business Systems Acquisition Executive (DBSAE)—the component  acquisition executive for DOD enterprise-level (DOD-wide) business  systems and initiatives. This directorate is responsible for developing,  coordinating, and integrating enterprise-level projects, programs, systems,  and initiatives—including managing resources such as fiscal, personnel,  and contracts for assigned systems and programs.
Table 1 lists these entities and provides greater detail on their roles,  responsibilities, and composition. Figure 3 provides a simplified  illustration of the relationships among these entities.
According to DOD, in 2005 it adopted a tiered accountability approach to  business transformation. Under this approach, responsibility and  accountability for business investment management is allocated between  the DOD corporate (i.e., OSD) and the components on the basis of the  amount of development/modernization funding involved and the  investment’s “tier.” DOD corporate is responsible for ensuring that all  business systems with a development/modernization investment in excess  of $1 million are reviewed by the IRBs for compliance with the BEA,  certified by the Principal Staff Assistants, and approved by the DBSMC.  Components are responsible for certifying development/modernization  investments with total costs of $1 million or less. All DOD development  and modernization efforts are also assigned a tier on the basis of the  acquisition category or the size of the financial investment, or both.  According to DOD, a system is given a tier designation when it passes  through the certification process. Table 2 describes the four investment  tiers and identifies the associated reviewing and approving entities.
DOD’s business investment management system includes two types of  reviews for business systems: certification and annual reviews.  Certification reviews apply to new modernization projects with total cost  over $1 million. This review focuses on program alignment with the BEA  and must be completed before components obligate funds for programs.  The annual review applies to all business programs. The focus for the  annual review is to determine whether the system development effort is  meeting its milestones and addressing its IRB certification conditions.
Certification reviews and approvals: Tiers 1 through 3 business system  investments are certified at two levels—component-level precertification  and corporate-level certification and approval. At the component level,  program managers prepare, enter, maintain, and update information about  their investments in the DOD IT Portfolio Repository (DITPR), such as  regulatory compliance reporting, an architectural profile, and  requirements for investment certification and annual reviews. The  component PCA validates that the system information is complete and  accessible on the IRB Portal, reviews system compliance with the BEA  and enterprise transition plan, and verifies the economic viability analysis.  The PCA asserts the status and validity of the investment information by  submitting a component precertification letter to the appropriate IRB for  its review.
At the corporate level, the IRB reviews the system information and  precertification letter submitted by the PCA to determine whether to  recommend investment certification. On completion of its review, a  certification memorandum is prepared and signed by the designated  certification authority that documents the IRB’s system certification  decisions and any related conditions. The memorandum is then forwarded  to the DBSMC, which either approves or disapproves the IRB’s decisions  and issues a memorandum containing its decisions. If the DBSMC  disapproves a system investment, it is up to the component PCA to decide  whether to resubmit the investment after it has resolved the relevant  issues. Figure 4 provides a simplified overview of the process flow of  certification reviews and approvals.
Annual reviews: Tiers 1 through 4 business system investments are  annually reviewed at two levels—the component level and the corporate  level. At the component level, program managers review and update  information on all tiers of investments, both in modernization and  operations and maintenance, on an annual basis in DITPR. The updates for  Tiers 1 through 3 with system development/modernization include cost,  milestone, and risk variances and actions or issues related to certification  conditions. The PCA then verifies and submits the information for Tiers 1  through 3 systems in development/modernization for IRB review in an  annual review assertion letter. The letter addresses system compliance  with the BEA and the enterprise transition plan, and includes investment  cost, schedule, and performance information.
At the corporate level, the IRBs annually review certified Tiers 1 through 3  investments in development/modernization. These reviews focus on  program compliance with the BEA, program performance against cost and  milestone baselines, and progress in meeting certification conditions. The  IRBs can revoke an investment’s certification when the system has  significantly failed to achieve performance commitments (i.e., capabilities  and costs). When this occurs, the component must address the IRB’s  concerns and resubmit the investment for certification. Figure 5 shows a  simplified overview of the process flow of annual reviews.
section_title: DOD Has Established the Structures Needed to Effectively Manage Business System Investments, but Has Not Fully Defined Many of the Related Policies and Procedures
paragraphs: According to our ITIM framework, organizations should establish the  management structures needed to manage their investments and build an  investment foundation by having defined policies and procedures for  selecting and controlling individual projects (Stage 2 capabilities), and  organizations also should manage projects as a portfolio of investments  according to defined policies and procedures, treating them as an  integrated package of competing investment options and pursuing those  that best meet the strategic goals, objectives, and mission of the agency  (Stage 3 capabilities). These Stages 2 and 3 capabilities assist agencies in  complying with the investment management provisions of the Clinger- Cohen Act.
The department has defined four of nine practices that call for project- level policies and procedures (see table 4) and one of the five practices  that call for portfolio-level policies and procedures (see table 6).  Specifically, it has established the management structures contained in our  ITIM framework, but it has not fully defined many of the related policies  and procedures.
With respect to project-level investment management practices, DOD  officials stated that these are performed at the component level, and that  departmental policies and procedures established for overseeing  components’ execution of these practices are sufficient. With respect to  portfolio-level practices, however, these officials stated that they intend to  improve departmental policies and procedures for business system  investments by, for example, establishing a single governance structure,  but plans or time frames for doing so have not been established. According  to our ITIM framework, adequately documenting both the policies and the  associated procedures that govern how an organization manages its IT  investment portfolio(s) is important because doing so provides the basis  for having rigor, discipline, and repeatability in how investments are  selected and controlled across the entire organization. Until DOD fully  defines departmentwide policies and procedures for both individual  projects and the portfolios of projects, it risks selecting and controlling  these business system investments in an inconsistent, incomplete, and ad  hoc manner, which in turn reduces the chances that these investments will  meet mission needs in the most cost-effective manner.
subsections: section_title: DOD Has Begun to Build a Foundation for Project- Level Investment Management, but Key Policies and Procedures Are Not Fully Defined
paragraphs: At ITIM Stage 2, an organization has attained repeatable and successful IT  project-level investment control and basic selection processes. Through  these processes, the organization can identify project expectation gaps  early and take the appropriate steps to address them. ITIM Stage 2 critical  processes include (1) defining investment board operations,   (2) identifying the business needs for each investment, (3) developing a  basic process for selecting new proposals and reselecting ongoing  investments, (4) developing project-level investment control processes,  and (5) collecting information about existing investments to inform  investment management decisions. Table 3 describes the purpose of each  of these Stage 2 critical processes.
Within these five critical processes are nine key practices that call for  policies and procedures associated with effective project-level  management. DOD has fully defined the policies and procedures needed to  ensure that four of these nine practices are performed in a consistent and  repeatable manner. Specifically, DOD has established the management  structures by instituting an enterprisewide investment board—the  DBSMC—composed of senior executives, including the Deputy Secretary  of Defense, with final approval authority over associated subsidiary  investment boards. These lower-level investment boards include  representatives from combatant commands, components, and the Joint  Chiefs of Staff. In addition, DOD’s business transformation and IRB  guidance define a process for ensuring that programs support the  department’s ongoing and future business needs. DOD also has policies  and procedures for submitting, updating, and maintaining investment  information in DITPR and the IRB Portal. Furthermore, the department  has assigned the component’s PCA the responsibility to ensure that  specific investment information contained in the portfolio repository and  the IRB Portal is accurate and complete.
However, the policies and procedures associated with the remaining five  project-level management practices are missing critical elements needed  to effectively carry out essential investment management activities. For  example:    Policies and procedures for instituting the investment board do not  address how investments that are past the development/modernization  stage (i.e., in operations and maintenance) are to be governed. Given that  DOD invests billions of dollars annually in operating and maintaining  business systems, this is significant. While DOD officials stated that  component-level policies and procedures address systems outside of  development/modernization, our ITIM framework emphasizes that the  corporate investment boards should continue to review important  information about an investment, such as cost and performance baselines,  throughout the investment’s life cycle. In addition, the IRB Concept of  Operations and other IRB documentation do not explicitly outline how the  business investment management system is coordinated with JCIDS,  PPBE, and DAS. Without clearly defined visibility into all investments with  an understanding of decisions reached through other management  systems, inconsistent decisions may result.
Procedures do not specify how the full range of cost, schedule, and benefit  data is used by the IRBs in making selection (i.e., certification) decisions.  According to BTA officials, each IRB decides how to ensure compliance  and determines additional factors to consider when making certification  decisions. However, DOD did not provide us with any supplemental  policies or procedures for any of the four IRBs. Without documenting how  IRBs consider factors such as cost, schedule, and benefits when making  selection decisions, the department cannot ensure that the IRBs and the  DBSMC consistently and objectively select proposals that best meet the  department’s needs and priorities. Furthermore, while the procedures  specify decision criteria that address statutory requirements for alignment  to the BEA, the criteria allow programs to postpone demonstrating full  compliance with several BEA artifacts until the final phases of the  acquisition process. As a result, programs risk beginning production and  deployment before ensuring that a business system is fully aligned to the  BEA.
Policies and procedures do not specify how reselection decisions at the  corporate level (i.e., annual review decisions) consider investments that  are in operations and maintenance. Without an understanding of how the  IRBs are to consider these investments when making reselection  decisions, their ability to make informed and consistent reselection and  termination decisions is limited.
Policies and procedures do not specify how funding decisions are  integrated with the process of selecting an investment at the corporate  level. Without considering component and corporate budget constraints  and opportunities, the IRBs risk making investment decisions that do not  effectively consider the relative merits of various projects and systems  when funding limitations exist.
Policies and procedures do not exist that provide for sufficient oversight  and visibility into component-level investment management activities,  including component reviews of systems in operations and maintenance  and Tier 4 investments. According to DOD officials, investment oversight  is implemented through tiered accountability, which, among other things,  allocates responsibility and accountability for business system  investments with total costs of $1 million or less and those in operations  and maintenance to the components. However, the department did not  provide policies and procedures defining how the DBSMC and the IRBs  ensure visibility into these component processes. This is particularly  important because, according to DOD’s March 15, 2007, annual report to  Congress, only 285 of approximately 3,100 total business systems have  completed the IRB certification process and have been approved by the  DBSMC. DOD officials also stated that the remaining business systems  have not been through the certification process and have not been given a  tier designation. Without policies and procedures defining how the  DBSMC and the IRBs have visibility into and oversight of all business  system investments, DOD risks components continuing to invest in  systems that are duplicative, stovepiped, nonintegrated, and unnecessarily  costly to manage, maintain, and operate.
Table 4 summarizes our findings relative to DOD’s execution of the nine  practices that call for the policies and procedures needed to manage IT  investments at the project level.
According to BTA officials, the IRB Concept of Operations and the  Investment Certification and Annual Review Process User Guidance are  not intended to describe the detailed approach that each IRB will use  when making certification decisions, adding that the components are  responsible for selection, annual review, budgeting, and acquisition. While  the ITIM framework does allow for multiple entities to carry out  investment selection, control, and evaluation, building a sound investment  foundation requires that the enterprisewide investment review board has  documented criteria and decision-making procedures, clear integration  among investment decision-support systems, and policies to ensure board  access to system information throughout the life cycle for all investments.  Until DOD’s documented IT investment management policies and  procedures include fully defined policies and procedures for Stage 2  activities, specify the linkages between the various related processes, and  describe how investments are to be governed in the operations and  maintenance phase, DOD risks that investment management activities will  not be carried out consistently and in a disciplined manner. Moreover,  DOD also risks selecting investments that will not cost-effectively meet its  mission needs.
section_title: DOD Has Assigned Responsibility, but Has Not Defined the Policies and Procedures Associated with Effective Portfolio- Level Management
paragraphs: At Stage 3, an organization has defined critical processes for managing its  investments as a portfolio or set of portfolios. Portfolio management is a  conscious, continuous, and proactive approach to allocating limited  resources among competing initiatives in light of the investments’ relative  benefits. Taking an agencywide perspective enables an organization to  consider its investments comprehensively, so that collectively the  investments optimally address the organization’s missions, strategic goals,  and objectives. Managing IT investments as portfolios also allows an  organization to determine its priorities and make decisions about which  projects to fund on the basis of analyses of the relative organizational  value and risks of all projects, including projects that are proposed, under  development, and in operation. Although investments may initially be  organized into subordinate portfolios—on the basis of, for example,  business lines or life-cycle stages—and managed by subordinate  investment boards, they should ultimately be aggregated into enterprise- level portfolios.
According to ITIM, Stage 3 involves (1) defining the portfolio criteria;  (2) creating the portfolio; (3) evaluating (i.e., overseeing) the portfolio; and  (4) conducting postimplementation reviews. Table 5 summarizes the  purpose of each of these activities.
DOD is executing one of the five practices within these four critical  processes that call for policies and procedures associated with effective  portfolio-level management. Specifically, DOD has issued departmentwide  guidance that assigns responsibilities to the USD(AT&L) for managing  and establishing business system investment portfolios, including  leveraging or establishing a governance forum to oversee these business  system investment portfolio activities.
However, DOD has not fully defined the policies and procedures needed to  effectively execute the remaining four portfolio management practices  relative to business system investments. Specifically, DOD does not have  policies and procedures for defining the portfolio criteria or for creating  and evaluating the portfolio. In addition, while DOD has policies and  procedures for conducting postimplementation reviews as part of DAS,  these reviews do not address systems at all tier levels. Furthermore, there  are no procedures detailing how lessons learned from these reviews are  used during investment review as the basis for management and process  improvements.
Table 6 summarizes the rating for each critical process required to manage  investment as a portfolio and summarizes the evidence that supports these  ratings.
According to BTA officials, while portfolio management is primarily a  component responsibility, they are working toward developing more  effective departmentwide portfolio management processes, but plans or  time frames for doing so have not been established. Without defining  corporate policies and procedures for managing business system  investment portfolios, DOD is at risk of not consistently selecting the mix  of investments that best supports the departmentwide mission needs and  ensuring that investment-related lessons learned are shared and applied  departmentwide.
section_title: Conclusions
paragraphs: Given the importance of business systems modernization to DOD’s  mission, performance, and outcomes, it is vital for the department to adopt  and employ an effective institutional approach to managing business  system investments. While the department has established aspects of such  an approach and, thus, has a foundation on which to build, it is lacking  other important elements, such as specific policies and procedures needed  for project-level and portfolio-level investment management, including  integration with DOD’s other key management systems and sufficient  oversight and visibility into operations and maintenance investments and  Tier 4 investments. This means that DOD lacks an institutional capability  to ensure that it is investing in business systems that best support its  strategic needs, and that ongoing projects meet cost, schedule, and  performance expectations. Until DOD develops this capability, the  department will be impaired in its ability to optimize business mission area  performance and accountability.
section_title: Recommendations for Executive Action
paragraphs: To strengthen DOD’s business system investment management capability  and address the weaknesses discussed in this report, we recommend that  the Secretary of Defense direct the Deputy Secretary of Defense, as the  chair of the DBSMC, to ensure that well-defined and disciplined business  system investment management policies and procedures are developed  and issued. At a minimum, this should include project-level management  policies and procedures that address the following five areas:  instituting the investment boards, including assigning the investment  boards responsibility, authority, and accountability for programs  throughout the investment life cycle and specifying how the business  investment management system is coordinated with JCIDS, PPBE, and  DAS;   selecting new investments, including specifying how cost, schedule, and  benefit data are to be used in making certification decisions; defining the  criteria used to select investments as enterprisewide; and establishing  consistent and effective guidance for BEA compliance;  reselecting ongoing investments, including specifying how cost, schedule,  and performance data are to be used in the annual review process and  providing for the reselection of investments that are in operations and  maintenance;   integrating funding with the process of selecting an investment, including  specifying how the DBSMC and the IRBs use funding information in  carrying out decisions on system certification and approvals; and    overseeing IT projects and systems, including providing sufficient  oversight and visibility into component-level investment management  activities.
These well-defined and disciplined business system investment  management policies and procedures should also include portfolio-level  management policies and procedures that address the following four  areas:    creating and modifying IT portfolio selection criteria for business system    analyzing, selecting, and maintaining business system investment  reviewing, evaluating, and improving the performance of its portfolio(s) by  using project indicators, such as cost, schedule, and risk; and   conducting postimplementation reviews for all investment tiers and  directing the investment boards, which are accountable for corporate  business system investments, to consider the information gathered and to  develop lessons learned from these reviews.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, signed by the Deputy Under  Secretary of Defense (Business Transformation) and reprinted in appendix  II, the department stated that it agreed with the report’s overall  conclusions, and it described efforts under way and planned that it said  would address many of the gaps identified in the report. In this regard, the  department partially concurred with five of the report’s recommendations,  adding that our recommendations and feedback are helpful in guiding  DOD’s business transformation and related improvement efforts.  Nevertheless, the department disagreed with the remaining four  recommendations on the grounds that their intent had already been met  through DOD’s existing business system investment management  structure and processes, or that they contradicted the tiered accountability  concept embedded in this structure and processes. The department’s  comments relative to each of our project-level and portfolio-level  recommendations, along with our responses to its comments, are provided  below.
With respect to our five project-level recommendations, the department  stated that it partially agreed with two and disagreed with three.
DOD partially agreed with our recommendation to define and implement  policies and procedures that assign the investment boards responsibility  for programs throughout the investment life cycle and specify how the  business investment management system is coordinated with JCIDS,  PPBE, and DAS. In particular, it stated that under its tiered accountability  approach to business systems investment management, the components  are currently required to review all programs throughout their investment  life cycles. We do not question this requirement, and we recognize it in our  report. However, consistent with our ITIM framework, the corporate  investment boards should continue to review investments that meet the  defined threshold criteria throughout their life cycles (i.e., when they are  in operations and maintenance). In contrast, DOD’s corporate boards  focus only on those investments that are in the  development/modernization stage. The department also stated that a  linkage is currently depicted in existing guidance among its investment  selection, acquisition, and funding processes. While we do not question  that this guidance contains an illustration depicting such a link, neither  this guidance nor supporting procedures define how this linkage is  executed (e.g., how investment funding decisions are in fact integrated  with investment selection decisions). DOD’s comments appear to  acknowledge this point by stating that the department has begun to define  and implement a Business Capability Lifecycle concept, which is intended  to integrate the investment selection and acquisition management  processes for Tier 1 and enterprise systems into a single oversight process  that leverages the existing IRB and DBSMC oversight framework.
DOD partially agreed with our recommendation to define and implement  policies and procedures that specify how cost, schedule, and benefit data  are to be used in making certification and annual review decisions; define  the criteria used to select investments as enterprisewide; and establish  consistent and effective guidance for BEA compliance. In particular, the  department agreed that additional criteria are required for selecting  enterprisewide investments, noting that initial criteria have been defined  and will be incorporated in the investment management process. However,  the department did not agree that cost, schedule, and BEA compliance  information are not sufficiently used for certification and annual review  decisions, adding that such information is required in its current policies.  We do not agree. Specifically, while we do not question whether  investment data are provided to the DBSMC and the IRBs, the  department’s policies and procedures do not include specific decision  criteria that explain how these data are to be used to make consistent,  repeatable selection and reselection decisions across all investments. In  addition, while BEA compliance policies have been developed and are  being used, the guidance is not fully defined. For example, the guidance  allows programs to defer demonstrating full compliance with important  BEA artifacts until the final phases of the acquisition process, at which  time addressing instances of noncompliance would be more expensive and  difficult. Furthermore, the compliance criteria are not consistently  described in different guidance documentation. As a result, DOD risks  beginning system production and deployment before ensuring that a  system is sufficiently aligned to the BEA.
DOD did not agree with our recommendation to define and implement  policies and procedures that provide for the reselection of investments  that are in operations and maintenance. According to DOD, components  are required by policy to annually review all business systems, including  investments for which there is no planned development or modernization  spending. We agree that the annual review process does require this.  However, consistent with our ITIM framework, the corporate investment  boards should continue to reselect investments that meet the defined  threshold criteria throughout their life cycles (i.e., when they are in  operations and maintenance). In contrast, DOD’s corporate boards focus  only on reselecting those investments that are in the  development/modernization stage.
DOD did not agree with our recommendation to define and implement  policies and procedures that specify how the corporate boards use funding  information in carrying out decisions on system certification and  approvals. In this regard, it stated that such information is required in its  current policies and considered during board deliberations. We do not  agree. Our recommendation does not address whether existing policies or  guidance provide for the collection of this information; our  recommendation addresses the definition of policy, guidance, and  supporting procedures that fall short of satisfying the best practices  embodied in our ITIM framework. Specifically, while we do not question  whether funding data are provided to investment decision-making bodies,  the department’s policies and procedures do not include specific decision  criteria that explain how these data are to be used to make consistent,  repeatable selection and reselection decisions across all investments.
DOD did not agree with our recommendation to define and implement  policies and procedures that provide for sufficient oversight and visibility  into component-level investment management activities. In particular, it  stated that this recommendation contradicts the department’s “tiered  accountability” approach to investment management. We do not agree.  Under the department’s current policies and guidance, most DOD  investments are not subject to corporate visibility and oversight, either  because they do not involve development/modernization (i.e., they are in  operations and maintenance) or because they do not exceed a certain  dollar threshold. Our framework recognizes that effective implementation  of a tiered accountability concept should include appropriate corporate  visibility into and oversight of investments, either through review and  approval of those investments that meet certain criteria or through  awareness of a subordinate board’s investment management activities.  Moreover, this visibility and oversight should extend to the entire portfolio  of investments, including those that are in operations and maintenance. To  ensure that this occurs, applicable policies and procedures need to  explicitly cover all such investments and need to define how this is to be  accomplished.
With respect to our four portfolio-level recommendations, the department  stated that it partially agreed with three and disagreed with one.
DOD partially agreed with our recommendation to define and implement  policies and procedures for creating and modifying portfolio selection  criteria for business system investments. In particular, it stated that while  components are responsible for developing and managing their own  portfolio management processes, upcoming initiatives, such as the  Business Capability Lifecycle concept, will lead to revisions in the  department’s investment review policies and procedures, such as  including portfolio selection criteria for enterprise systems that span  components. However, while these are important steps, the concept, as  defined by the department, does not apply to the thousands of investments  that are not enterprisewide.
DOD partially agreed with our recommendation to define and implement  policies and procedures that address analyzing, selecting, and maintaining  business system investment portfolios. In particular, it stated that the  implementation of the Business Capability Lifecyle concept will provide  the corporate boards with improved visibility into all investments in a  given portfolio and a broader set of criteria for analyzing, selecting, and  maintaining business system investment portfolios.
DOD partially agreed with our recommendation to define and implement  policies and procedures that address reviewing, evaluating, and improving  the performance of its portfolio(s) by using cost, schedule, and risk  indicators. In particular, it stated that while such indicators are part of the  investment certification and review processes, efforts are now under way  to better understand the nature and impact of program risks through  application of an Enterprise Risk Assessment Methodology. While we  recognize the role and value of such tools in understanding and addressing  program risks, this tool is program-specific and not portfolio-focused.
DOD did not agree with our recommendation to define and implement  policies and procedures that address conducting postimplementation  reviews and having the corporate investment boards consider the review  results and develop lessons learned from them. In particular, it stated that  this process should not be managed by the Deputy Secretary of Defense,  and also stated that our recommendation is redundant with  postimplementation reviews currently required under OMB Circular A- 130. We do not agree with DOD’s statements. Our recommendation does  not call for the Deputy Secretary to manage the postimplementation  review process. Rather, it provides for developing policies and procedures  for performing postimplementation reviews for all tiers of business  systems and having the DBSMC and IRBs, which are the corporate  investment boards, consider the information gathered from these reviews  and develop lessons learned.
We are sending copies of this report to interested congressional  committees; the Director, Office of Management and Budget; the Secretary  of Defense; the Deputy Secretary of Defense; the Under Secretary of  Defense for Acquisition, Technology, and Logistics; the Under Secretary of  Defense (Comptroller); the Assistant Secretary of Defense (Networks and  Information Integration)/Chief Information Officer; the Under Secretary of  Defense (Personnel and Readiness); and the Director, Defense Finance  and Accounting Service. Copies of this report will be made available to  other interested parties upon request. This report will also be available at  no charge on our Web site at http://www.gao.gov.
If you or your staffs have any questions on matters discussed in this  report, please contact me at (202) 512-3439 or hiter@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix III.
section_title: Appendix I: Objective, Scope, and Methodology
paragraphs: Our objective was to determine whether the Department of Defense’s  (DOD) corporate investment management approach comports with  relevant federal guidance. Our analysis was based on the best practices  contained in GAO’s Information Technology Investment Management  (ITIM) framework, and the framework’s associated evaluation  methodology, and focused on DOD’s establishment of departmental-level  policies and procedures for business system investments needed to assist  organizations in complying with the investment management provisions of  the Clinger-Cohen Act of 1996 (Stages 2 and 3). It did not include case  studies to verify the implementation of established policies and  procedures.
To address our objective, we asked DOD to complete a self-assessment of  its corporate investment management process and provide the supporting  documentation. We then reviewed the results of the department’s self- assessment of Stages 2 and 3 organizational commitment practices— meaning those practices related to structures, policies, and procedures— and compared them against our ITIM framework. We also validated and  updated the results of the self-assessment through document reviews and  interviews with officials, such as the Director of Investment Management  and the Defense Business Systems Acquisition Executive. In doing so, we  reviewed written policies, procedures, and guidance and other  documentation providing evidence of executed practices, including the  Defense Acquisition System guidance, the Investment Review Board (IRB)  Concept of Operations and Guidance, the Business Enterprise  Architecture Compliance Guidance, IRB charters and meeting minutes,  and the Business Transformation Guidance.
We compared the evidence collected from our document reviews and  interviews with the key practices in ITIM. We rated the key practices as  “executed” on the basis of whether the agency demonstrated (by providing  evidence of performance) that it had met all of the criteria of the key  practice. A key practice was rated as “not executed” when we found  insufficient evidence of all elements of a practice being fully performed or  when we determined that there were significant weaknesses in DOD’s  execution of the key practice. In addition, we provided DOD with the  opportunity to produce evidence for the key practices rated as “not  executed.”
We conducted our work at DOD headquarters offices in Arlington,  Virginia, from August 2006 through April 2007 in accordance with  generally accepted government auditing standards.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact person named above, key contributors to this  report were Neil Doherty, Nalani Fraser, Nancy Glover, Michael Holland,  Neelaxi Lakhmani (Assistant Director), Jacqueline Mai, Sabine Paul, Niti  Tandon, and Jennifer Stavros-Turner.