Business Systems Modernization: Air Force Needs to Fully Define Policies and Procedures for Institutionally Managing Investments
Published: 2007-10-31T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In 1995, GAO first designated the Department of Defense's (DOD) business systems modernization program as "high-risk" and continues to do so today. In 2004, Congress passed legislation reflecting prior GAO recommendations that DOD adopt a corporate approach to information technology (IT) business systems investment management including tiered accountability for business systems at the department and component levels. To support GAO's legislative mandate to review DOD's efforts, GAO assessed whether the investment management approach of one of DOD's components--the Department of the Air Force (Air Force)--is consistent with leading investment management best practices. In doing so, GAO applied its IT Investment Management (ITIM) framework and associated methodology, focusing on the stages related to the investment management provisions of the Clinger-Cohen Act of 1996.
section_title: What GAO Found
paragraphs: The Air Force has established the basic management structures needed to effectively manage its IT projects as investments, but has not fully implemented many of the related policies and procedures outlined in GAO's ITIM framework. Air Force has fully implemented three of the nine key practices that call for project-level management structures, policies, and procedures, and has not implemented any of the five practices that call for portfolio-level policies and procedures. Regarding project-level practices, it has established an IT investment board that is responsible for defining and implementing the department's business systems investment governance process, has developed procedures for identifying and collecting information about its business systems to support investment selection and control, and has assigned responsibility for ensuring that the information collected during project identification meets the needs of the investment management process. However, Air Force has not fully documented business systems investment policies and procedures for directing investment board operations, selecting new investments, reselecting ongoing investments, or integrating the investment funding and investment selection processes. In addition, it has not implemented any of the policies and procedures for developing and maintaining a complete business system investment portfolio. Air Force officials stated that they are aware of the absence of documented policies and procedures in certain areas of project-level and portfolio-level management and that they are currently working on guidance to address these areas. For example, officials stated that they had begun drafting portfolio-level policies and procedures. According to Air Force officials, the policies and procedures are expected to be completed and approved by December 2007. Until Air Force fully defines policies and procedures for both individual projects and portfolios of projects, it risks not being able to select and control these business system investments in a way that is consistent and complete, which in turn increases the chances that these investments will not meet mission needs in the most effective manner.

section_title: Letter
paragraphs: For decades, the Department of Defense (DOD) has been challenged in  modernizing its timeworn business systems. In 1995, we designated DOD’s  business systems modernization program as high risk, and we continue to  designate it as such today. Our research on public and private sector  organizations shows that an essential ingredient to a successful systems  modernization program is having an effective institutional approach to  managing information technology (IT) investments.
In May 2001, we recommended that the department establish a corporate  approach to investment control and decision making. Between 2001 and  2005, we reported that the department’s business systems modernization  program was still not being effectively managed, and we made additional  investment-related recommendations. Congress subsequently included  provisions in the Ronald W. Reagan National Defense Authorization Act  for Fiscal Year 2005 that reflected our recommendations, including those  for establishing and implementing effective business system investment  management structures and processes.
Between 2005 and 2007, we reported that DOD had made important  progress in establishing and implementing these structures and processes;  however, much remained to be accomplished. Most recently, we reported  that, according to DOD officials, investment management practices are  performed at the component level, and that departmental policies and  procedures established for overseeing components’ execution of these  practices are sufficient. However, DOD had not fully defined many of the  related policies and procedures outlined in GAO’s IT Investment  Management framework.
The Fiscal Year 2005 National Defense Authorization Act directs DOD to,  among other things, establish and implement effective IT business system  investment management structures and processes. As agreed with your  offices and to support the legislative mandate that GAO assess DOD’s  actions to comply with this requirement, the objective of our review was  to determine whether the investment management approach of the  Department of the Air Force (Air Force) is consistent with leading  investment management best practices. To accomplish our objective, we  analyzed documents and interviewed department officials to determine  whether Air Force has developed the structures, policies, and procedures  associated with executing those key practices in our IT Investment  Management (ITIM) framework that assist departments and agencies in  complying with the investment management provisions of the Clinger- Cohen Act of 1996.
We performed our work at Air Force offices in Arlington, Virginia, from  February 2007 through September 2007 in accordance with generally  accepted government auditing standards. Details on our objective, scope,  and methodology are contained in appendix I.
subsections: section_title: Background
paragraphs: DOD is a massive and complex organization. To illustrate, the department  reported that its fiscal year 2006 operations involved approximately $1.4  trillion in assets and $2.0 trillion in liabilities, more than 2.9 million  military and civilian personnel, and $581 billion in net cost of operations.  Organizationally, the department includes the Office of the Secretary of  Defense, the Chairman of the Joint Chiefs of Staff, the military  departments, numerous defense agencies and field activities, and various  unified combatant commands that are responsible for either specific  geographic regions or specific functions. Figure 1 provides a simplified  depiction of DOD’s organizational structure.
In support of its military operations, DOD performs an assortment of  interrelated and interdependent business functions, including logistics  management, procurement, health care management, and financial  management. As we have previously reported, the systems environment  that supports these business functions is overly complex and error prone,  and is characterized by (1) little standardization across the department,   (2) multiple systems performing the same tasks, (3) the same data stored  in multiple systems, and (4) the need for data to be entered manually into  multiple systems.
subsections: section_title: Air Force’s Mission, Organizational Structure, and Use of IT
paragraphs: Air Force is a major component of DOD. Its mission is to deliver options  for the defense of the United States and its global interests in air, space,  and cyberspace. Air Force relies extensively on IT to fulfill these  competencies effectively and to meet its organizational mission. It has 909  business systems; of these systems, 832 (91 percent) are in operations and  maintenance. In fiscal year 2006, Air Force was allocated approximately  $651 million for its business systems, of which about $406 million (62  percent) was allocated to systems in operations and maintenance and $245  million (38 percent) was allocated to systems in development and/or  modernization.
Air Force has created the Office of Warfighting Integration and Chief  Information Office to provide the IT and supporting infrastructure to fulfill  its mission. Among the goals of this organization are to    deliver the ability to direct forces while anticipating situations,  capabilities, and limitations;    develop adaptive, trained airmen;    provide policy, standards, oversight, and training to enable airmen to share  and exploit accurate information any place and, anytime; and  transform the communications and information career field to lead Air  Force in leveraging information for its competitive advantage.
The Office of Warfighting Integration and Chief Information Office  consists of several organizations, as depicted in figure 2.
section_title: IT Investment Management Is Critical to Achieving Successful Systems Modernization
paragraphs: Successful public and private organizations use a corporate approach to IT  investment management. Recognizing this, Congress enacted the Clinger- Cohen Act of 1996, which requires the Office of Management and Budget  (OMB) to establish processes to analyze, track, and evaluate the risks and  results of major capital investments in IT systems made by executive  agencies. In response to the Clinger-Cohen Act and other statutes, OMB  has developed policy and issued guidance for the planning, budgeting,  acquisition, and management of federal capital assets. We have also  issued guidance in this area that defines institutional structures, such as  investment review boards; processes for developing information on  investments (such as costs and benefits); and practices to inform  management decisions (such as whether a given investment is aligned with  an enterprise architecture).
section_title: IT Investment Management: A Brief Description
paragraphs: IT investment management is a process for linking IT investment decisions  to an organization’s strategic objectives and business plans. Consistent  with this, the federal approach to IT investment management focuses on  selecting, controlling, and evaluating investments in a manner that  minimize risks while maximizing the return on investment.
During the selection phase, the organization (1) identifies and analyzes  each project’s risks and returns before committing significant funds to any  project and (2) selects those IT projects that will best support its mission  needs.
During the control phase, the organization ensures that projects, as they  develop and investment expenditures continue, meet mission needs at the  expected levels of cost and risk. If the project is not meeting expectations  or if problems arise, steps are quickly taken to address the deficiencies.
During the evaluation phase, expected results are compared with actual  results after a project has been fully implemented. This comparison is  done to (1) assess the project’s impact on mission performance,   (2) identify any changes or modifications to the project that may be  needed, and (3) revise the investment management process based on  lessons learned.
section_title: Overview of GAO’s ITIM Maturity Framework
paragraphs: Our ITIM framework consists of five progressive stages of maturity for any  given agency relative to selecting, controlling, and evaluating its  investment management capabilities. (See fig. 3 for the five ITIM stages  of maturity.) This framework is grounded in our research of IT investment  management practices of leading private and public sector organizations.  The framework can be used to assess the maturity of an agency’s  investment management processes and as a tool for organizational  improvement. The overriding purpose of the framework is to encourage  investment selection and control and to evaluate processes that promote  business value and mission performance, reduce risk, and increase  accountability and transparency. We have used the framework in several  of our evaluations, and a number of agencies have adopted it.
ITIM’s five maturity stages represent the steps toward achieving stable and  mature processes for managing IT investments. Each stage builds on the  lower stages; the successful attainment of each stage leads to  improvement in the organization’s ability to manage its investments. With  the exception of the first stage, each maturity stage is composed of  “critical processes” that must be implemented and institutionalized in  order for the organization to achieve that stage. These critical processes  are further broken down into key practices that describe the types of  activities that an organization should be performing to successfully  implement each critical process. It is not unusual for an organization to be  performing key practices from more than one maturity stage at the same  time. However, our research has shown that agency efforts to improve  investment management capabilities should focus on implementing all  lower stage practices before addressing the higher stage practices.
In the ITIM framework, Stage 2 critical processes lay the foundation for  sound IT investment management by helping the agency to attain  successful, predictable, and repeatable investment management processes  at the project level. Specifically, Stage 2 encompasses building a sound  investment management foundation by establishing basic capabilities for  selecting new IT projects. This stage also involves developing the  capability to control projects so that they finish predictably within  established cost and schedule expectations and developing the capability  to identify potential exposures to risk and put in place strategies to  mitigate that risk. Further, it involves evaluating completed projects to  ensure they meet business needs and collecting lessons learned to improve  the IT investment management process. The basic management processes  established in Stage 2 lay the foundation for more mature management  capabilities in Stage 3, which represents a major step forward in maturity,  in which the agency moves from project-centric processes to a portfolio  approach, evaluating potential investments by how well they support the  agency’s missions, strategies, and goals.
Stage 3 requires that an organization continually assess both proposed and  ongoing projects as parts of a complete investment portfolio—an  integrated and competing set of investment options. It focuses on  establishing a consistent, well-defined perspective on the IT investment  portfolio and maintaining mature, integrated selection (and reselection),  control, and post-implementation evaluation processes. This portfolio  perspective allows decision makers to consider the interaction among  investments and the contributions to organizational mission goals and  strategies that could be made by alternative portfolio selections, rather  than focusing exclusively on the balance between the costs and benefits of  individual investments. Organizations that have implemented Stages 2 and  3 practices have capabilities in place that assist in establishing selection;  control; and evaluation structures, policies, procedures, and practices that  are required by the investment management provisions of the Clinger- Cohen Act.
Stages 4 and 5 require the use of evaluation techniques to continuously  improve both the investment portfolio and the investment processes in  order to better achieve strategic outcomes. At Stage 4, an organization has  the capacity to conduct IT succession activities and, therefore, can plan  and implement the deselection of obsolete, high-risk, or low-value IT  investments. An organization with Stage 5 maturity conducts proactive  monitoring for breakthrough information technologies that will enable it  to change and improve its business performance.
section_title: DOD and Air Force Approach for Identifying, Funding, and Acquiring System Investments
paragraphs: DOD’s major system investments (i.e., weapons and business systems) are  governed by three management systems that focus on defining needs,  budgeting for, and acquiring investments to support the mission—the Joint  Capabilities Integration and Development System (JCIDS); the Planning,  Programming, Budgeting, and Execution (PPBE) system; and the Defense  Acquisition System (DAS). In addition, DOD’s business systems are  subject to a fourth management system, which, for purposes of this report,  we refer to as the Business Investment Management System. For each of  these systems, DOD relies on its component agencies to execute the  underlying policies and procedures. According to DOD, the four  management systems, collectively, are the means by which the  department—and its components—selects, controls, and evaluates its  business systems investments.
section_title: Joint Capabilities Integration and Development System
paragraphs: JCIDS is a needs-driven, capabilities-based approach to identify mission  needs and meet future joint forces challenges. It is intended to identify  future capabilities for DOD; address capability gaps and mission needs  recognized by the Joint Chiefs of Staff or derived from strategic guidance,  such as the National Security Strategy Report or Quadrennial Defense  Review; and identify alternative solutions by considering a range of  doctrine, organization, training, materiel, leadership and education,  personnel, and facilities solutions. According to DOD, the Joint Chiefs of  Staff—through the Joint Requirements Oversight Council—has primary  responsibility for defining and implementing JCIDS. All JCIDS documents  are submitted to the Joint Chiefs of Staff, which determines whether the  proposed system has joint implications or is component-unique. If it is  designated as joint interest, then the Joint Requirements Oversight Council  is responsible for approving and validating the documents. If it is not  designated as having joint interests, the sponsoring component is  responsible for validation and approval.
section_title: Planning, Programming, Budgeting, and Execution
paragraphs: PPBE is a calendar-driven approach that is composed of four phases that  occur over a moving 2-year cycle. The four phases—planning,  programming, budgeting, and executing—define how budgets for each  DOD component and the department as a whole are created, vetted, and  executed. As recently reported, the components start programming and  budgeting for addressing a JCIDS-identified capability gap or mission need  several years before actual product development begins and before the  Office of the Secretary of Defense formally reviews the components’  programming and budgeting proposals (i.e., Program Objective  Memorandums). Once reviewed and approved, the financial details in the  Program Objective Memorandums become part of the President’s budget  request to Congress. During budget execution, components may submit  program change proposals or budget change proposals, or both (e.g.,  program cost increases or schedule delays). According to DOD, the Under  Secretary of Defense (Policy), the Director for Program Analysis and  Evaluation, and the Under Secretary of Defense (Comptroller) have  primary responsibility for defining and implementing the PPBE system.
section_title: Defense Acquisition System
paragraphs: DAS is a framework-based approach that is intended to translate mission  needs and requirements into stable, affordable, and well-managed  acquisition programs. It consists of five key program life-cycle phases.  These five phases are as follows:  Concept Refinement: Intended to refine the initial JCIDS-validated  system solution (concept) and create a strategy for acquiring the  investment solution. A decision is made at the end of this phase (Milestone  A decision) regarding whether to move to the next phase (Technology  Development).
Technology Development: Intended to determine the appropriate set of  technologies to be integrated into the investment solution by iteratively  assessing the viability of various technologies while simultaneously  refining user requirements. Once the technology has been demonstrated in  a relevant environment, a decision is made (Milestone B decision)  regarding whether to move to the next phase (System Development and  Demonstration).
System Development and Demonstration: Intended to develop a  system or a system increment and demonstrate through developer testing  that the system or system increment can function in its target  environment. A decision is made at the end of this phase (Milestone C  decision) regarding whether to move to the next phase (Production and  Deployment).
Production and Deployment: Intended to achieve an operational  capability that satisfies the mission needs, as verified through independent  operational test and evaluation, and ensures that the system is  implemented at all applicable locations.
Operations and Support: Intended to operationally sustain the system in  the most effective manner over its life cycle.
A key principle of DAS is that investments are assigned a category, where  programs of increasing dollar value and management interest are subject  to more stringent oversight. For example, Major Defense Acquisition  Programs and Major Automated Information Systems are large,  expensive programs subject to the most extensive statutory and regulatory  reporting requirements and unless delegated, are reviewed by acquisition  boards at the DOD corporate level. Smaller and less risky acquisitions are  generally reviewed at the component executive or lower levels. Another  key principle is that DAS requires acquisition management under the  direction of a milestone decision authority. The milestone decision  authority—with support from the program manager and advisory boards,  such as the Defense Acquisition Board and the IT Acquisition Board— determines the project’s baseline cost, schedule, and performance  commitments. The Under Secretary of Defense for Acquisition,  Technology, and Logistics has primary responsibility for defining and  implementing DAS.
DOD relies on its components to execute these investment management  policies and procedures. To implement DOD’s JCIDS process, the Air  Force has designated the Joint Staff Functional Capabilities Board to  review and approve operational capabilities. The Joint Staff Functional  Capabilities Board seeks to establish a common understanding of how a  capability will be used, who will use it, when it is needed, and why it is  needed to achieve a desired effect. Each capability is assessed based on  the effects it seeks to generate and the associated operational risk of not  having it. In addition, the Capabilities Review and Risk Assessment  process is being used to analyze concepts of operation and assess their  associated capabilities. This process uses a phased approach to produce a  prioritized list of capabilities, capability gaps or shortfalls, and possible  capability solutions. To implement the PPBE process, Air Force officials  stated that they use their Annual Planning and Programming Guidance  manual. Finally, to implement DAS, Air Force has developed guidance that  outlines a systematic acquisition framework that mirrors the framework  defined by DOD and includes the same three event-based milestones and  associated five program life-cycle phases.
section_title: Business Investment Management System
paragraphs: The Business Investment Management System is a calendar-driven  approach that is described in terms of governance entities, tiered  accountability, and certification reviews and approvals. This system was  initiated in 2005, when DOD reassigned responsibility for providing  executive leadership for the direction, oversight, and execution of its  business systems modernization efforts to several entities. These entities  and their responsibilities include the following:    The Defense Business Systems Management Committee serves as the  highest-ranking governance body for business systems modernization  activities.
The Principal Staff Assistants serve as the certification authorities for  business system modernizations in their respective core business  missions.
The Investment Review Boards are chartered by the principal staff  assistants and are the review and decision-making bodies for business  system investments in their respective areas of responsibility. The boards  are also responsible for recommending certification for all business  system investments costing more than $1 million.
The component precertification authority is accountable for the  component’s business system investments and acts as the component’s  principal point of contact for communication with the Investment Review  Boards. The Air Force has designated its CIO to be the precertification  authority.
The Business Transformation Agency is responsible for leading and  coordinating business transformation efforts across the department. The  agency is organized into seven directorates, one of which is the Defense  Business Systems Acquisition Executive—the component acquisition  executive for DOD enterprise-level (DOD-wide) business systems and  initiatives. This directorate is responsible for developing, coordinating,  and integrating enterprise-level projects, programs, systems, and  initiatives—including managing resources such as fiscal, personnel, and  contracts for assigned systems and programs.
Figure 4 provides a simplified illustration of the relationships among these  entities.
According to DOD, in 2005 it also adopted a tiered accountability  approach to business transformation. Under this approach, responsibility  and accountability for business system investment management is  allocated among DOD (i.e., Office of the Secretary of Defense) and the  component agencies, based on the amount of development/modernization  funding involved and the investment’s “tier.” DOD is responsible for  ensuring that all business systems with a development/modernization  investment in excess of $1 million are reviewed by the Investment Review  Boards for compliance with the business enterprise architecture, certified  by the principal staff assistants, and approved by the Defense Business  Systems Management Committee. Components are responsible for  certifying development/modernization investments with total costs of $1  million or less. All DOD development and modernization efforts are  assigned a tier on the basis of the acquisition category or the size of the  financial investment, or both. According to DOD, a system is given a tier  designation when it passes through the certification process. Table 1  describes the five investment tiers and identifies the associated reviewing  and approving entities for DOD and the Air Force.
DOD’s business investment management system includes two types of  reviews for business systems: certification and annual reviews.  Certification reviews apply to new modernization projects with total costs  over $1 million. These reviews focus on program alignment with the  business enterprise architecture and must be completed before  components obligate funds for programs. The annual reviews apply to all  business programs and are undertaken to determine whether the system  development effort is meeting its milestones and addressing its Investment  Review Board certification conditions.
Certification reviews and approvals: Tier 1 through 3 business system  investments in development and modernization are certified at two  levels—component-level precertification and DOD-level certification and  approval. At the component level, program managers prepare, enter,  maintain, and update information about their investments in the Air Force  data repository. The component precertification authority validates that  the system information is complete and accessible on the Air Force data  repository, reviews system compliance with the business enterprise  architecture and enterprise transition plan, and verifies the economic  viability analysis. This information is then transferred to DOD’s IT  Portfolio Repository. The precertification authority asserts the status and  validity of the investment information by submitting a component  precertification letter to the appropriate Investment Review Board for its  review.
Annual reviews: Tier 1 through 4 business system investments are  reviewed annually at the component and DOD-levels. At the component  level, program managers annually review and update information on all  tiers of system investments that are identified in their data repository. For  Tier 1 through 3 systems that are in development or being modernized,  information is updated on cost, milestone, and risk variances and actions  or issues related to certification conditions. The precertification authority  then verifies and submits the information for these business system  investments for DOD’s Investment Review Board review in an annual  review assertion letter. The letter addresses system compliance with the  DOD business enterprise architecture and the enterprise transition plan  and includes investment cost, schedule, and performance information.
At the DOD level, the Investment Review Boards annually review  investments for certified Tier 1 through 3 business systems that are in  development or are being modernized. These reviews focus on program  compliance with the business enterprise architecture, program cost and  performance milestones, and progress in meeting certification conditions.  The Investment Review Boards can revoke an investment’s certification  when the system has significantly failed to achieve performance  commitments (i.e., capabilities and costs). When this occurs, the  component must address the Investment Review Board’s concerns and  resubmit the investment for certification.
section_title: Air Force’s Precertification Process
paragraphs: As stated earlier, DOD relies on its components to execute investment  management policies and procedures. Air Force has developed a  precertification process, which is intended to ensure that new or existing  systems undergo proper scrutiny prior to being precertified by the Air  Force CIO. First, the certification package is to be prepared by the  Program Manager and reviewed by the Major Command (MAJCOM) and  functional portfolio managers.
The package is then to be provided to the Air Force Certification Process  Manager, who is to review the package for completeness based on  certification requirements and transmit the package to the Certification  Review Team—which is composed of subject matter experts—to assess  compliance with their relevant areas of expertise, such as the business  enterprise architecture and information assurance. Once the certification  review is complete, the package is to be sent to the Senior Working Group.  This group is responsible for reviewing the package and approving or  disapproving the package. Finally, the Precertification Authority is to  precertify the system for submission to the relevant Investment Review  Board.
Table 2 lists decision-making personnel involved in Air Force’s investment  management process and provides a description of their key  responsibilities.
Figure 5 shows the relationship among the key players in Air Force’s  precertification review and approval process.
Reviews and validates certification packages.  Pre-certifies Tier 1-3 and approves Tier 4 systems.
Technically reviews and endorses program information  and certification packages for Tier 1-4.
Reviews and validates certification packages for Tier 1-4  and coordinates review with other subject matter experts.
Functionally reviews and validates program information  and certification packages for Tier 1-4.
Enters and maintains business system investment  information in Air Force data repository for Tier 1-4,  completes certification package requirements for  Tier 1-3.
section_title: Air Force Has Established the Structures Needed to Effectively Manage Business System Investments but Has Not Fully Defined Many of the Related Policies and Procedures
paragraphs: DOD relies on its components to execute investment management policies  and procedures. However, while Air Force has established the basic  management structures needed to effectively manage its IT projects as  investments, it has not fully implemented many of the related policies and  procedures outlined in our ITIM framework. Relative to its business  system investments, Air Force has implemented three of nine practices  that call for project-level structures, policies, and procedures, and has not  defined any of the five practices that call for portfolio-level policies and  procedures. Air Force officials stated that they are aware of the absence of  documented policies and procedures, and they are currently working on  guidance to address these areas. For example, these officials stated that  they have drafted policies and procedures to establish portfolio-level  practices and are currently obtaining the necessary approvals. Air Force  plans to complete and approve these policies and procedures by  December 2007. According to our framework, adequately documenting  both the policies and the associated procedures that govern how an  organization manages its IT investment portfolio is important because  doing so provides the basis for having rigor, discipline, and repeatability in  how investments are selected and controlled across the entire  organization.
Until Air Force has fully defined policies and procedures for both  individual projects and the portfolio of projects, it risks selecting and  controlling these business system investments in an inconsistent,  incomplete, and ad hoc manner, which in turn could reduce the chances  that these investments will meet mission needs in the most effective  manner.
subsections: section_title: Air Force Has Begun to Build a Foundation for Project-Level Investment Management but Has Not Yet Fully Defined Key Policies and Procedures
paragraphs: At ITIM Stage 2, an organization has attained repeatable and successful IT  project-level investment control processes and basic selection processes.  Through these processes, the organization can identify project expectation  gaps early and take the appropriate steps to address them. ITIM Stage 2  critical processes include (1) defining investment board operations, (2)  identifying the business needs for each investment, (3) developing a basic  process for selecting new proposals and reselecting ongoing investments,  (4) developing project-level investment control processes, and (5)  collecting information about existing investments to inform investment  management decisions. Table 3 describes the purpose of each of these  Stage 2 critical processes.
Within these five critical processes are nine key practices required for  effective project-level management. Air Force has fully defined the policies  and procedures for three of these nine practices. Specifically, Air Force  has established a management structure by instituting a business system  Investment Review Board, called the Senior Working Group. This group is  composed of senior executives from the functional business units,  including the Office of the Air Force Chief Information Officer, and the  members are responsible for establishing and implementing investment  policies. In addition, Air Force has established policies and procedures for  capturing information about its IT projects and systems and submitting,  updating, and maintaining this information in its data repository. Finally, it  has assigned the Certification Process Manager the responsibility of  ensuring that specific investment information contained in the Air Force  data repository is accurate and complete.
However, the Air Force’s policies and procedures associated with the  remaining six project-level management practices are missing critical  elements needed to effectively carry out essential investment management  activities. For example:    Policies and procedures for directing the Investment Review Board’s  operations do not define how investments that are in operations and  maintenance are to be governed by the Investment Review Board. In  addition, procedures do not specify how the business investment  management process is coordinated with other DOD management  systems. Without clearly defined guidance and visibility into all  investments with an understanding of decisions reached through other  management systems, Air Force cannot be assured that consistent  investment management decisions are being made.
Policies and procedures do not define how systems in operations and  maintenance will support ongoing and future business needs. This  increases the risk that Air Force will continue to maintain legacy  investments that no longer support current organizational objectives.
Policies and procedures for selecting new systems do not specify how the  full range of cost, schedule, and performance data are being considered in  making selection (i.e., precertification) decisions. Without documenting  how factors such as cost, schedule, and performance are considered when  making precertification decisions, Air Force cannot ensure that it  consistently and objectively selects system investments that best meet the  department’s needs and priorities.
Policies and procedures do not include a structured method that defines  how the criteria will be evaluated when the precertification authority  makes reselection decisions. In addition, policies and procedures do not  define an approach to annually reviewing systems in operations and  maintenance. Given that Air Force spends millions of dollars annually in  operating and maintaining business systems, this is significant. Without an  understanding of how the precertification authority is to consider these  investments when making reselection decisions, Air Force’s ability to  make informed and consistent reselection and termination decisions is  limited.
Policies and procedures do not specify how funding decisions are  integrated with the process of selecting an investment. Without  considering budget constraints and opportunities, Air Force risks making  investment decisions that do not effectively consider the relative merits of  various projects and systems when funding limitations exist.
Policies and procedures do not provide for sufficient oversight and  visibility into investment management activities. Air Force has predefined  criteria for adherence to cost, schedule, and performance milestones, but  does not have policies and procedures that guide the implementation of  corrective actions when program expectations are not met. Without such  policies and procedures, the agency risks investing in systems that are  duplicative, stovepiped, nonintegrated, and unnecessarily costly to  manage, maintain, and operate.
Table 4 summarizes our findings relative to Air Force’s execution of the  nine key practices that call for the policies and procedures needed to  manage IT investments at the project level.
Air Force officials stated that they are aware of the absence of  documented procedures in certain areas of project-level management, and  plan to issue new policies and procedures addressing these areas by  December 2007. However, until Air Force fully documents IT investment  management policies and procedures for Stage 2 activities and specifies  the linkages between the various related processes, and describes how  system investments in operations and maintenance are to be governed, it  risks not being able to carry out investment management activities in a  consistent and disciplined manner. Moreover, the Air Force risks selecting  investments that will not effectively meet its mission needs.
section_title: Air Force Has Assigned Responsibility but Has Not Defined the Policies and Procedures Associated with Effective Portfolio- Level Management
paragraphs: At Stage 3, an organization has defined critical processes for managing its  investments as a portfolio or a set of portfolios. Portfolio management is  a conscious, continuous, and proactive approach to allocating limited  resources among competing initiatives in light of the investments’ relative  benefits. Taking a departmentwide perspective enables an organization to  consider its investments comprehensively, so that collectively the  investments optimally address the organization’s missions, strategic goals,  and objectives. Managing IT investments as portfolios also allows an  organization to determine its priorities and make decisions about which  projects to fund on the basis of analyses of the relative organizational  value and risks of all projects, including projects that are proposed, under  development, and in operation. Although investments may initially be  organized into subordinate portfolios—on the basis of, for example,  business lines or life-cycle stages—and managed by subordinate  investment boards, they should ultimately be aggregated into enterprise- level portfolios.
According to ITIM, Stage 3 involves four critical processes (1) defining the  portfolio criteria; (2) creating the portfolio; (3) evaluating (i.e., overseeing)  the portfolio; and (4) conducting post-implementation reviews. Within  these critical processes are five key practices that call for policies and  procedures to ensure effective portfolio management. Table 5 summarizes  the purpose of each of the critical processes.
Air Force has begun to establish a governance structure for portfolio-level  management, but it has not executed any of the five practices within the  Stage 3 critical processes that call for policies and procedures associated  with effective portfolio-level management. Specifically, Air Force has  assigned the Senior Working Group the responsibility of establishing a  governance forum to oversee business system portfolio activities.  However, the Senior Working Group has not developed and approved  charters outlining the roles and responsibilities to be assigned to  subordinate Investment Review Boards that are intended to establish and  manage the portfolios.
In addition, Air Force has not fully defined policies and procedures needed  to effectively execute portfolio management practices. Specifically, Air  Force does not have policies and procedures for defining the portfolio  criteria or creating and evaluating the portfolio. In addition, while DOD  has policies and procedures for conducting post-implementation reviews  for Tier 1 systems as part of the Defense Acquisition System, Air Force has  not established policies or procedures for conducting post-implementation  reviews for systems in the remaining tiers. Finally, Air Force has not  established procedures detailing how lessons learned from these reviews  are to be used during investment reviews as the basis for management and  process improvements. Table 6 summarizes the rating for each critical  process required to manage investments as a portfolio and summarizes the  evidence that supports these ratings.
Air Force officials are aware that they need to develop the appropriate  portfolio management processes, and in this regard, have drafted some  portfolio management guidance, such as the Air Force Operational  Support Portfolio Investment Review Process. According to Air Force  officials, this guidance is expected to be completed and approved by  December 2007. Until policies and procedures for managing business  systems investment portfolios are defined and implemented, Air Force is  at risk of not consistently selecting the mix of investments that best  supports the department mission needs and ensuring that investment- related lessons learned are shared and applied departmentwide.
section_title: Conclusions
paragraphs: Given the importance of business systems modernization to Air Force’s  mission, performance, and outcomes, it is vital for the department to adopt  and employ an effective institutional approach to managing business  system investments. However, while the department acknowledges these  shortcomings and the importance of addressing them and has established  aspects of such an approach, it is lacking important elements, such as  policies and procedures needed for project-level and portfolio-level  investment management. This means that Air Force lacks an institutional  capability to ensure that it is investing in business systems that best  support its strategic needs and that ongoing projects meet cost, schedule,  and performance expectations. Until Air Force develops this capability,  the department will be impaired in its ability to optimize business mission  area performance and accountability.
section_title: Recommendations for Executive Action
paragraphs: To strengthen Air Force’s business system investment management  capability and address the weaknesses discussed in this report, we  recommend that the Secretary of Defense direct the Secretary of the Air  Force to ensure that well-defined and disciplined business system  investment management policies and procedures are developed and  issued. At a minimum, this should include project-level management  policies and procedures that address the six key practices areas:    Specifying how systems that are in operations and maintenance will be  reviewed and specifying how Air Force’s business investment management  system is coordinated with JCIDS, PPBE, and DAS.
Ensuring that systems in operations and maintenance are aligned with  ongoing and future business needs.
Selecting investments, including specifying how factors, such as cost,  schedule, and performance data are to be used in making certification  decisions.
Reselecting ongoing investments, including specifying how factors, such  as cost, schedule, and performance data are to be used in making  reselection decisions during the annual review process and providing for  the reselection of investments that are in operations and maintenance.
Integrating funding with the process of selecting an investment, including  specifying how the precertification authority is using funding information  in carrying out decisions on system certification and approvals.
Overseeing IT projects and systems, including specifying policies and  procedures that guide the implementation of corrective actions when  program expectations are not met.
These well-defined and disciplined business system investment  management policies and procedures should also include portfolio-level  management policies and procedures that address the following five areas:    Creating and modifying IT portfolio selection criteria for business system  investments.
Defining the roles and responsibilities for the development and  modification of the IT portfolio selection criteria.
Analyzing, selecting, and maintaining business system investment  portfolios.
Reviewing, evaluating, and improving the performance of the portfolio(s)  by using project indicators, such as cost, schedule, and risk.
Conducting post-implementation reviews for all investment tiers and  directing the investment boards to consider the information gathered to  develop lessons learned from these reviews.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, signed by the Deputy Under  Secretary of Defense (Business Transformation) and reprinted in appendix  II, DOD partially concurred with the report’s recommendations. The  department further stated that our recommendations and feedback were  helpful in guiding its business transformation and related improvement  efforts.
Regarding our recommendation that DOD direct Air Force to ensure that  well-defined and disciplined business system investment management  policies and procedures are developed and issued, DOD stated that Air  Force has a well-defined business system investment management process  that, while not codified into formal policy, has been documented in  investment review guides. Further, it stated that Air Force is committed to  formalizing its policies as the processes expand and mature. Our report  recognizes that Air Force has drafted a business system investment  management process. However, as our report states, the draft review  guide lacks critical elements needed to effectively carry out essential  investment management activities. Until Air Force completes and issues IT  investment management policies and procedures that fully address these  elements, it risks not being able to carry out investment management  activities in a consistent and disciplined manner.
Regarding our recommendation that DOD direct Air Force to ensure that  the well-defined and disciplined business system investment management  policies and procedures also include portfolio-level management policies  and procedures, the department stated that DOD Instruction 8115.02  defines the DOD IT portfolio management process, which Air Force is  applying in its decisionmaking. However, as our report notes, Air Force  has not implemented a process for managing its business system  portfolios. Until Air Force defines and implements such a process, it is at  risk of not consistently selecting the mix of investments that best supports  the Air Force’s mission needs and ensuring that investment-related lessons  learned are shared and applied departmentwide.
We are sending copies of this report to interested congressional  committees; the Director, Office of Management and Budget; the Secretary  of Defense; the Deputy Secretary of Defense; the Secretary of Air Force;  the Air Force Chief Information Officer, and the Under Secretary of  Defense for Acquisition, Technology, and Logistics. Copies of this report  will be made available to other interested parties on request. This report  will also be available at no charge on our Web site at http://www.gao.gov.
Should you or your staffs have any questions on matters discussed in this  report, please contact me at (202) 512-6304 or melvinv@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix III.
section_title: Appendix I: Objective, Scope, and Methodology
paragraphs: Our objective was to determine whether the investment management  approach of the Department of the Air Force (a major Department of  Defense (DOD) component) is consistent with leading investment  management best practices. Our analysis was based on the best practices  contained in GAO’s Information Technology Investment Management  (ITIM) framework and the framework’s associated evaluation  methodology, and focused on Air Force’s establishment of policies and  procedures for business system investments needed to assist organizations  in complying with the investment management provisions of the Clinger- Cohen Act of 1996 (Stages 2 and 3).
To address our objective, we asked Air Force to complete a self- assessment of its investment management process and provide the  supporting documentation. We then reviewed the results of the  department’s self-assessment of Stages 2 and 3 organizational commitment  practices—meaning those practices related to structures, policies, and  procedures—and compared them with our ITIM framework. We focused  on Stages 2 and 3 because these stages represent the processes needed to  meet the standards of the Clinger-Cohen Act, and they establish the  foundation for effective acquisition management. We also validated and  updated the results of the self-assessment through document reviews and  interviews with officials, such as the Air Force Chief Information Officer  and Certification Process Manager. In doing so, we reviewed written  policies, procedures, and guidance and other documentation providing  evidence of executed practices, including the Air Force Information  Technology Investment Review Guide, Air Force Operations of  Capabilities Based Acquisition System Instruction, Air Force IT  Investment Architecture Compliance Guide, Senior Working Group  charter and meeting minutes, the Investment Review Board Concept of  Operations and Guidance, and the Business Transformation Guidance.
We compared the evidence collected from our document reviews and  interviews with the key practices in ITIM. We rated the key practices as  “executed” on the basis of whether the department demonstrated (by  providing evidence of performance) that it had met all of the criteria of the  key practice. A key practice was rated as “not executed” when we found  insufficient evidence of all elements of a practice being fully performed or  when we determined that there were significant weaknesses in Air Force’s  execution of the key practice. In addition, we provided Air Force with the  opportunity to produce evidence for the key practices rated as “not  executed.”
We conducted our work at the Air Force offices in Arlington, Virginia,  from February 2007 through September 2007 in accordance with generally  accepted government auditing standards.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact person named above, key contributors to this  report were Tonia Johnson, Assistant Director; Idris Adjerid; Sharhonda  Deloach; Nancy Glover; and Freda Paintsil.