Business Systems Modernization: Department of the Navy Needs to Establish Management Structure and Fully Define Policies and Procedures for Institutionally Managing Investments
Published: 2007-10-31T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In 1995, GAO first designated the Department of Defense's (DOD) business systems modernization program as "high-risk," and continues to do so today. In 2004, Congress passed legislation reflecting prior GAO recommendations that DOD adopt a corporate approach to information technology (IT) business systems investment management, including tiered accountability for business systems at the department and component levels. To support GAO's legislative mandate to review DOD's efforts, GAO assessed whether the investment management approach of one of DOD's components--the Department of the Navy--is consistent with leading investment management best practices. In doing so, GAO applied its IT Investment Management (ITIM) framework and associated methodology, focusing on the stages related to the investment management provisions of the Clinger-Cohen Act of 1996.
section_title: What GAO Found
paragraphs: The Department of the Navy has yet to establish the management structures needed to effectively manage its business systems investments or to fully develop many of the related policies and procedures outlined in GAO's ITIM framework. The department has implemented two of the nine key practices that call for project-level management structures, policies, and procedures, and none of the five practices that call for portfolio-level policies and procedures. Specifically, it has developed procedures for identifying and collecting information about its business systems to support investment selection and control, and assigned responsibility for ensuring that the information collected during project identification meets the needs of the investment management process. However, the department has not established the management structures needed to support effective investment oversight. It also has not fully documented business system investment policies and procedures for directing Investment Review Board operations, selecting new investments, reselecting ongoing investments, integrating the investment funding and investment selection processes, and developing and maintaining complete business system investment portfolio(s). Department officials stated that they are aware of the lack of an Investment Review Board and the absence of documented policies and procedures in certain areas of project and portfolio-level management, and are currently working on new guidance to address these areas. According to these officials, the new policies and procedures are expected to be approved by March 2008. However, until the department assigns responsibility for overseeing project-level management and portfolio management to a departmentwide review board and fully defines policies and procedures for both individual projects and portfolios of projects, it risks selecting and controlling these business system investments in a way that is inconsistent, incomplete, and ad hoc, which in turn reduces the chances that these investments will meet mission needs in the most effective manner.

section_title: Letter
paragraphs: For decades, the Department of Defense (DOD) has been challenged in  modernizing its timeworn business systems. In 1995, we designated DOD’s  business systems modernization program as high risk, and we continue to  designate it as such today. Our research on public and private sector  organizations shows that an essential ingredient to a successful systems  modernization program is having an effective institutional approach to  managing information technology (IT) investments.
In May 2001, we recommended that DOD establish a corporate approach  to investment control and decision making. Between 2001 and 2005, we  reported that DOD’s business systems modernization program was still not  being effectively managed, and we made additional investment-related  recommendations. Congress subsequently included provisions in the  Ronald W. Reagan National Defense Authorization Act for Fiscal Year  2005 that reflected our recommendations, including those for establishing  and implementing effective business system investment management  structures and processes.
Between 2005 and 2007, we reported that DOD had made important  progress in establishing and implementing these structures and processes;  however, much remained to be accomplished. Most recently, we reported  that, according to DOD officials, investment management practices are  performed at the component level, and policies and procedures  established for overseeing components’ execution of these practices are  sufficient. However, DOD had not fully defined many of the related  policies and procedures outlined in GAO’s IT Investment Management  framework.
The Fiscal Year 2005 National Defense Authorization Act directs DOD to,  among other things, establish and implement effective IT business system  investment management structures and processes. As agreed with your  offices and to support the legislative mandate that GAO assess DOD’s  actions to comply with this requirement, the objective of our review was  to determine whether the investment management approach of the  Department of the Navy is consistent with leading investment  management best practices. To accomplish our objective, we analyzed  documents and interviewed agency officials to determine whether the  department has developed the structures, policies, and procedures  associated with executing those key practices in our IT Investment  Management (ITIM) framework that assist departments and agencies in  complying with the investment management provisions of the Clinger- Cohen Act of 1996.
We performed our work at Department of the Navy offices in Arlington,  Virginia, from February 2007 through September 2007 in accordance with  generally accepted government auditing standards. Details on our  objective, scope, and methodology are contained in appendix I.
subsections: section_title: Background
paragraphs: DOD is a massive and complex organization. To illustrate, it reported that  its fiscal year 2006 operations involved approximately $1.4 trillion in assets  and $2.0 trillion in liabilities, more than 2.9 million military and civilian  personnel, and $581 billion in net cost of operations. Organizationally,  DOD includes the Office of the Secretary of Defense, the Chairman of the  Joint Chiefs of Staff, the military departments, numerous defense agencies  and field activities, and various unified combatant commands that are  responsible for either specific geographic regions or specific functions.  Figure 1 provides a simplified depiction of DOD’s organizational structure.
In support of its military operations, DOD performs an assortment of  interrelated and interdependent business functions, including logistics  management, procurement, health care management, and financial  management. As we have previously reported, the systems environment  that supports these business functions is overly complex and error prone,  and is characterized by (1) little standardization across DOD, (2) multiple  systems performing the same tasks, (3) the same data stored in multiple  systems, and (4) the need for data to be entered manually into multiple  systems.
subsections: section_title: Department of the Navy’s Mission, Organizational Structure, and Use of IT
paragraphs: The Department of the Navy is a major component of DOD, consisting of  two uniformed services: the Navy and the Marine Corps. The department’s  mission is to maintain, train, and equip combat-ready naval forces capable  of winning wars, deterring aggression, and maintaining freedom of the  seas. To support this mission, the department performs a variety of  interrelated and interdependent business functions, such as logistics and  financial management, relying extensively on IT to carry out its operations.  In fiscal year 2006, the department’s budget for IT was $4.3 billion, of  which $3.9 billion (90.3 percent) was allocated to operations and  maintenance of existing systems and $424 million (9.7 percent) was  allocated to systems in development and modernization. The department  was appropriated about $4.2 billion in fiscal year 2007 and requested about  $4 billion in fiscal year 2008 to operate, maintain, and modernize business  systems and associated infrastructures.
The Chief Information Officer (CIO) for the department is accountable for  all IT business system investments for both the Navy and Marine Corps.  The CIO’s office is organized to align and integrate information  management and IT programs across the two services and focus  departmentwide efforts in support of warfighter priorities. The CIO is  supported by Deputy CIOs for the Navy and Marine Corps and a Deputy  CIO for Policy and Integration, who directs the operations of the CIO  functional teams. The functional teams are led by team leaders who are  subject matter experts in their areas of responsibility and are responsible  for implementing the goals and objectives outlined in the department’s  information management and IT strategic plan, which includes, among  other things, ensuring that investments are effectively selected, resourced,  and acquired. Figure 2 outlines the department CIO organizational  structure.
section_title: IT Investment Management Is Critical to Achieving Successful Systems Modernization
paragraphs: A corporate approach to IT investment management is characteristic of  successful public and private organizations. Recognizing this, Congress  enacted the Clinger-Cohen Act of 1996, which requires the Office of  Management and Budget (OMB) to establish processes to analyze, track,  and evaluate the risks and results of major capital investments in IT  systems made by executive agencies. In response to the Clinger-Cohen  Act and other statutes, OMB has developed policy and issued guidance for  the planning, budgeting, acquisition, and management of federal capital  assets. We have also issued guidance in this area that defines  institutional structures, such as Investment Review Boards; processes for  developing information on investments (such as costs and benefits); and  practices to inform management decisions (such as whether a given  investment is aligned with an enterprise architecture).
section_title: IT Investment Management: A Brief Description
paragraphs: IT investment management is a process for linking IT investment decisions  to an organization’s strategic objectives and business plans. Consistent  with this, the federal approach to IT investment management focuses on  selecting, controlling, and evaluating investments in a manner that  minimizes risks while maximizing the return on investment.
During the selection phase, the organization (1) identifies and analyzes  each project’s risks and returns before committing significant funds to any  project and (2) selects those IT projects that will best support its mission  needs.
During the control phase, the organization ensures that projects, as they  develop and investment expenditures continue, meet mission needs at the  expected levels of cost and risk. If the project is not meeting expectations  or if problems arise, steps are quickly taken to address the deficiencies.
During the evaluation phase, expected results are compared with actual  results after a project has been fully implemented. This comparison is  done to (1) assess the project’s impact on mission performance, (2)  identify any changes or modifications to the project that may be needed,  and (3) revise the investment management process based on lessons  learned.
section_title: Overview of GAO’s ITIM Maturity Framework
paragraphs: Our ITIM framework consists of five progressive stages of maturity for any  given agency relative to selecting, controlling, and evaluating its  investment management capabilities. (See fig. 3 for the five ITIM stages  of maturity.) This framework is grounded in our research of IT investment  management practices of leading private and public sector organizations.  The framework can be used to assess the maturity of an agency’s  investment management processes and as a tool for organizational  improvement. The overriding purpose of the framework is to encourage  investment processes that increase business value and mission  performance, reduce risk, and increase accountability and transparency in  the decision process. We have used the framework in several of our  evaluations, and a number of agencies have adopted it.
ITIM’s five maturity stages represent steps toward achieving stable and  mature processes for managing IT investments. Each stage builds on the  lower stages; the successful attainment of each stage leads to  improvement in the organization’s ability to manage its investments. With  the exception of the first stage, each maturity stage is composed of  “critical processes” that must be implemented and institutionalized in  order for the organization to achieve that stage. These critical processes  are further broken down into key practices that describe the types of  activities that an organization should be performing to successfully  implement each critical process. It is not unusual for an organization to be  performing key practices from more than one maturity stage at the same  time. However, our research has shown that agency efforts to improve  investment management capabilities should focus on implementing all  lower stage practices before addressing the higher stage practices.
In the ITIM framework, Stage 2 critical processes lay the foundation for  sound IT investment management by helping the agency to attain  successful, predictable, and repeatable investment management processes  at the project level. Specifically, Stage 2 encompasses building a sound  investment management foundation by establishing basic capabilities for  selecting new IT projects. This stage also involves developing the  capability to control projects so that they finish predictably within  established cost and schedule expectations and developing the capability  to identify potential exposures to risk and put in place strategies to  mitigate that risk. Further, it involves evaluating completed projects to  ensure they meet business needs and collecting lessons learned to improve  the IT investment management process. The basic management processes  established in Stage 2 lay the foundation for more mature management  capabilities in Stage 3, which represents a major step forward in maturity,  in which the agency moves from project-centric processes to a portfolio  approach, evaluating potential investments by how well they support the  agency’s missions, strategies, and goals.
Stage 3 requires that an organization continually assess both proposed and  ongoing projects as parts of a complete investment portfolio—an  integrated and competing set of investment options. It focuses on  establishing a consistent, well-defined perspective on the IT investment  portfolio and maintaining mature, integrated selection (and reselection),  control, and post-implementation evaluation processes. This portfolio  perspective allows decision makers to consider the interaction among  investments and the contributions to organizational mission goals and  strategies that could be made by alternative portfolio selections, rather  than focusing exclusively on the balance between the costs and benefits of  individual investments. Organizations that have implemented Stages 2 and  3 practices have capabilities in place that assist in establishing selection;  control; and evaluation structures, policies, procedures, and practices that  are required by the investment management provisions of the Clinger- Cohen Act.
Stages 4 and 5 require the use of evaluation techniques to continuously  improve both the investment portfolio and the investment processes in  order to better achieve strategic outcomes. At Stage 4, an organization has  the capacity to conduct IT succession activities and, therefore, can plan  and implement the deselection of obsolete, high-risk, or low-value IT  investments. An organization with Stage 5 maturity conducts proactive  monitoring for breakthrough information technologies that will enable it  to change and improve its business performance.
section_title: DOD and Department of the Navy Approach for Identifying, Funding, and Acquiring System Investments
paragraphs: DOD’s major system investments (i.e., weapons and business systems) are  governed by three management systems that focus on defining needs,  budgeting for, and acquiring investments to support the mission—the Joint  Capabilities Integration and Development System (JCIDS); the Planning,  Programming, Budgeting, and Execution (PPBE) system; and the Defense  Acquisition System (DAS). In addition, DOD’s business systems are  subject to a fourth management system, which, for purposes of this report,  we refer to as the Business Investment Management System. For each of  these systems, DOD relies on its components to execute the underlying  policies and procedures. According to DOD, the four management  systems, collectively, are the means by which DOD—and its  components—selects, controls, and evaluates its business systems  investments.
section_title: Joint Capabilities Integration and Development System
paragraphs: JCIDS is a needs-driven, capabilities-based approach to identify mission  needs and meet future joint forces challenges. It is intended to identify  future capabilities for DOD; address capability gaps and mission needs  recognized by the Joint Chiefs of Staff or derived from strategic guidance,  such as the National Security Strategy Report or Quadrennial Defense  Review; and identify alternative solutions by considering a range of  doctrine, organization, training, materiel, leadership and education,  personnel, and facilities solutions. According to DOD, the Joint Chiefs of  Staff—through the Joint Requirements Oversight Council—has primary  responsibility for defining and implementing JCIDS. All JCIDS documents  are submitted to the Joint Chiefs of Staff, which determines whether the  proposed system has joint implications or is component-unique. If it is  designated as joint interest, then the Joint Requirements Oversight Council  is responsible for approving and validating the documents. If it is not  designated as having joint interests, the sponsoring component is  responsible for validation and approval.
section_title: Planning, Programming, Budgeting, and Execution System
paragraphs: PPBE is a calendar-driven approach that is composed of four phases that  occur over a moving 2-year cycle. The four phases—planning,  programming, budgeting, and executing—define how budgets for each  component and DOD as a whole are created, vetted, and executed. As  recently reported, the components start programming and budgeting for  addressing a JCIDS-identified capability gap or mission need several years  before actual product development begins and before the Office of the  Secretary of Defense formally reviews the components’ programming and  budgeting proposals (i.e., Program Objective Memorandums). Once  reviewed and approved, the financial details in the Program Objective  Memorandums become part of the President’s budget request to Congress.  During budget execution, components may submit program change  proposals or budget change proposals, or both (e.g., program cost  increases or schedule delays). According to DOD, the Under Secretary of  Defense (Policy), the Director for Program Analysis and Evaluation, and  the Under Secretary of Defense (Comptroller) have primary responsibility  for defining and implementing the PPBE system.
section_title: Defense Acquisition System
paragraphs: DAS is a framework-based approach that is intended to translate mission  needs and requirements into stable, affordable, and well-managed  acquisition programs, and it consists of five key program life-cycle phases.  These five phases are as follows:  Concept Refinement: Intended to refine the initial JCIDS-validated  system solution (concept) and create a strategy for acquiring the  investment solution. A decision is made at the end of this phase (Milestone  A decision) regarding whether to move to the next phase (Technology  Development).
Technology Development: Intended to determine the appropriate set of  technologies to be integrated into the investment solution by iteratively  assessing the viability of various technologies while simultaneously  refining user requirements. Once the technology has been demonstrated in  a relevant environment, a decision is made (Milestone B decision)  regarding whether to move to the next phase (System Development and  Demonstration).
System Development and Demonstration: Intended to develop a  system or a system increment and demonstrate through developer testing  that the system or system increment can function in its target  environment. A decision is made at the end of this phase (Milestone C  decision) regarding whether to move to the next phase (Production and  Deployment).
Production and Deployment: Intended to achieve an operational  capability that satisfies the mission needs, as verified through independent  operational test and evaluation, and ensures that the system is  implemented at all applicable locations.
Operations and Support: Intended to operationally sustain the system in  the most effective manner over its life cycle.
A key principle of DAS is that investments are assigned a category, where  programs of increasing dollar value and management interest are subject  to more stringent oversight. For example, Major Defense Acquisition  Programs and Major Automated Information Systems are large,  expensive programs subject to the most extensive statutory and regulatory  reporting requirements and, unless delegated, are reviewed by acquisition  boards at the DOD level. Smaller and less risky acquisitions are generally  reviewed at the component executive or lower levels. Another key  principle is that DAS requires acquisition management under the direction  of a milestone decision authority. The Milestone Decision Authority— with support from the Program Manager and advisory boards, such as the   Defense Acquisition Board and the IT Acquisition Board—determines  the project’s baseline cost, schedule, and performance commitments. The  Under Secretary of Defense for Acquisition, Technology, and Logistics has  primary responsibility for defining and implementing DAS.
DOD relies on its components to execute these investment management  policies and procedures. To implement DOD’s JCIDS process, the  Department of the Navy has developed service-level processes—the Naval  Capabilities Development Process and the Marine Corps Expeditionary  Force Development System—to support the requirements generation  process of JCIDS. To implement the PPBE process, department officials  stated that they use their budget guidance manual. Finally, to implement  the DAS process, the department has developed guidance that outlines a  systematic acquisition framework that mirrors the framework defined by  DOD and includes the same three event-based milestones and associated  five program life-cycle phases.
section_title: Business Investment Management System
paragraphs: The Business Investment Management System is a calendar-driven  approach that is described in terms of governance entities, tiered  accountability, and certification reviews and approvals. This system was  initiated in 2005, when DOD reassigned responsibility for providing  executive leadership for the direction, oversight, and execution of its  business systems modernization efforts to several entities. These entities  and their responsibilities include the following:    The Defense Business Systems Management Committee serves as the  highest-ranking governance body for business systems modernization  activities.
The Principal Staff Assistants serve as the certification authorities for  business system modernizations in their respective core business  missions.
The Investment Review Boards are chartered by the principal staff  assistants and are the review and decision-making bodies for business  system investments in their respective areas of responsibility. The boards  are also responsible for recommending certification for all business  system investments costing more than $1 million.
The component precertification authority is accountable for the  component’s business system investments and acts as the component’s  principal point of contact for communication with the Investment Review  Boards. The Department of the Navy has designated its CIO to be the  Precertification Authority.
The Business Transformation Agency is responsible for leading and  coordinating business transformation efforts across DOD. The agency is  organized into seven directorates, one of which is the Defense Business  Systems Acquisition Executive—the component acquisition executive for  DOD-wide business systems and initiatives. This directorate is responsible  for developing, coordinating, and integrating enterprise-level projects,  programs, systems, and initiatives—including managing resources such as  fiscal, personnel, and contracts for assigned systems and programs.  Figure 4 provides a simplified illustration of the relationships among these  entities.
According to DOD, in 2005 it also adopted a tiered accountability  approach to business transformation. Under this approach, responsibility  and accountability for business system investment management is  allocated among DOD (i.e., Office of the Secretary of Defense) and the  components, based on the amount of development/modernization funding  involved and the investment’s “tier.” DOD is responsible for ensuring that  all business systems with a development/modernization investment in  excess of $1 million are reviewed by the Investment Review Boards for  compliance with the business enterprise architecture, certified by the  principal staff assistants, and approved by the Defense Business Systems  Management Committee. Components are responsible for certifying  development/modernization investments with total costs of $1 million or  less. All DOD development and modernization efforts are assigned a tier  on the basis of the acquisition category or the size of the financial  investment, or both. According to DOD, a system is given a tier  designation when it passes through the certification process. Table 1  describes the investment tiers and identifies the associated reviewing and  approving entities for DOD and the Department of the Navy.
DOD’s business investment management system includes two types of  reviews for business systems: certification and annual reviews.  Certification reviews apply to new modernization projects with total costs  over $1 million. These reviews focus on program alignment with the  business enterprise architecture and must be completed before  components obligate funds for programs. The annual reviews apply to all  business programs and are intended to determine whether the system  development effort is meeting its milestones and addressing its Investment  Review Board certification conditions.
Certification reviews and approvals: Tier 1 through 3 business system  investments in development and modernization are certified at two  levels—components precertify and DOD certifies and approves these  system investments. At the component level, program managers prepare,  enter, maintain, and update information about their investments in their  data repository, such as regulatory compliance reporting, an architectural  profile, and requirements for investment certification and annual reviews.  The component precertification authority validates that the system  information is complete and accessible on the repository, reviews system  compliance with the business enterprise architecture and enterprise  transition plan, and verifies the economic viability analysis. This  information is then transferred to DOD’s IT Portfolio Repository. The  precertification authority asserts the status and validity of the investment  information by submitting a component precertification letter to the  appropriate Investment Review Board for its review.
Annual reviews: Tier 1 through 4 business system investments are  annually reviewed at the component and DOD-levels. At the component  level, program managers annually review and update information on all  tiers of system investments that are identified in their data repository. For  Tier 1 through 3 systems that are in development or being modernized,  information is updated on cost, milestones, and risk variances and actions  or issues related to certification conditions. The precertification authority  then verifies and submits the information for these business system  investments for the DOD Investment Review Board’s review in an annual  review assertion letter. The letter addresses system compliance with the  DOD business enterprise architecture and the enterprise transition plan  and includes investment cost, schedule, and performance information.
At the DOD level, the Investment Review Boards annually review  investments for certified Tier 1 through 3 business systems that are in  development or modernization. These reviews focus on program  compliance with the business enterprise architecture, program cost and  performance milestones, and progress in meeting certification conditions.  The Investment Review Boards can revoke an investment’s certification  when the system has significantly failed to achieve performance  commitments (i.e., capabilities and costs). When this occurs, the  component must address the Investment Review Board’s concerns and  resubmit the investment for certification.
section_title: Department of the Navy Precertification Process
paragraphs: As stated earlier, DOD relies on its components to execute investment  management policies and procedures. The Department of the Navy has  developed a precertification process for its business systems, which is  intended to ensure that new or existing systems that are being modernized  undergo proper scrutiny prior to being precertified by the department’s  Precertification Authority. The precertification process is initiated by the  Program Manager, who is responsible for completing all data elements  required for a specific tier, including entering data and attachments into  the department’s repository and entering funding information into the  DOD budgeting database.
After the precertification package has been completed by the Program  Manager, it is to be reviewed by both Functional Area Managers and the  Deputy CIOs for the Navy and Marine Corps. The Functional Area  Managers’ primary responsibilities are to functionally review data for each  defense business system for which they are the lead or stakeholder and  ensure that IT and business processes are aligned. The primary  responsibilities of the Deputy CIOs are to technically review each defense  business system within their service and verify that the system’s  architecture complies with the department’s enterprise architecture and  the DOD business enterprise architecture. The final task of the Deputy CIO  and the Functional Area Managers is to provide a recommendation to the  department Precertification Authority as to whether or not the business  system should be certified. The reviews of the Deputy CIOs and Functional  Area Managers may occur concurrently.
Following the Functional Area Manager and Deputy CIO reviews, a  business system is to be sent to the department’s CIO for final approval.  The CIO is responsible for reviewing Tier 1 through 4 submissions,  precertifying Tier 1 through 3 defense business system investments, and  certifying Tier 4 investments. The CIO is also responsible for monitoring  the activities of the Functional Area Managers and the Deputy CIOs, and  for ensuring that functional area manager coordination is effective and  sufficient for identifying redundant investments. Once a Tier 1 through 3  investment has been precertified, the CIO is to complete, among other  things, a precertification letter and send the certification package to DOD  for review by the applicable DOD Investment Review Board and Defense  Business Systems Management Committee.
Table 2 lists decision-making personnel involved in the department’s  investment management process and provides a description of their key  responsibilities.
Figure 5 shows a simplified overview of the process flow of  precertification reviews and approvals for the Department of the Navy.
section_title: Department of the Navy Has Not Yet Established the Management Structures Needed to Effectively Manage Business System Investments and Has Not Fully Defined Many of the Related Policies and Procedures
paragraphs: Although DOD relies on its components to execute investment  management policies and procedures, the Department of the Navy has  not yet established the management structures needed to effectively  manage its business system investments or fully developed many of the  related policies and procedures outlined in our ITIM framework. Relative  to its business system investments, the department has implemented two  of the nine key practices that call for project-level management structures,  policies, and procedures and none of the five key practices that call for  portfolio-level policies and procedures. Department officials stated that  they are currently working on guidance to address these weaknesses. For  example, the officials stated that they are drafting new portfolio-level  policies and procedures and are developing guidance that is intended to  assign IT management roles and responsibilities to new or existing boards.  The new policies and procedures and guidance are expected to be  approved by March 2008. According to our ITIM framework, adequately  documenting both the policies and the associated procedures that govern  how an organization manages IT projects and investment portfolios is  important because doing so provides the basis for having rigor, discipline,  and repeatability in how investments are selected and controlled across  the entire organization.
Until the department establishes the necessary management structure and  fully defines policies and procedures for both individual projects and the  portfolios of projects, it risks not being able to select and control these  business system investments in a consistent and complete manner, which  in turn reduces the chances that these investments will meet mission  needs in the most effective manner.
subsections: section_title: Department of the Navy Has Yet to Build a Foundation for Project- Level Investment Management
paragraphs: At ITIM Stage 2, an organization has attained a repeatable and successful  IT project-level investment control process and basic selection processes.  Through these processes, the organization can identify project expectation  gaps early and take the appropriate steps to address them. ITIM Stage 2  critical processes include (1) defining investment board operations, (2)  identifying the business needs for each investment, (3) developing a basic  process for selecting new proposals and reselecting ongoing investments,  (4) developing project-level investment control processes, and (5)  collecting information about existing investments to inform investment  management decisions.
Table 3 describes the purpose of each of these Stage 2 critical processes.
Within these five critical processes are nine key practices that call for  policies and procedures associated with effective project-level  management. The department has fully defined the policies and  procedures for two of these nine processes. Specifically, it has policies  and procedures for capturing investment information by submitting,  updating, and maintaining investment information in its repository and  loading information to the DOD repository. Further, the department has  assigned its CIO the responsibility of ensuring that information contained  in its repository is accurate and complete.
However, the management structures and policies and procedures  associated with the remaining seven project-level management practices  are missing critical elements needed to effectively carry out essential  investment management activities. For example:    The department has not yet established an Investment Review Board,  composed of senior executives from its IT and business units, to define  and implement the organization’s IT investment governance process.  Without an Investment Review Board, the department’s ability to ensure  that investment decisions are consistent and reflect the needs of the  organization is limited.
The department does not have a documented IT investment management  process that completely explains the agency’s selection, control, and  evaluation of IT investments. Without such an investment management  process, the department may not make consistent decisions regarding its  IT investments.
The department’s policies and procedures do not explain how ongoing IT  investments are periodically reviewed and verified relative to meeting the  business needs of its organization and users. Without documenting how  officials are to ensure that IT business system investments maintain  alignment with the organization’s strategic plans and business goals and  objectives, the department cannot ensure a consistent selection of  investments that best meet its needs and priorities.
The department’s procedures for selecting new investments do not specify  how the full range of cost, schedule, and benefit data are used by  department officials (CIO, Deputy CIOs, and Functional Area Managers) in  making selection decisions. Without documenting how these officials are  to consider factors such as cost, schedule, and benefits when making  selection decisions, the department cannot ensure that it can consistently  and objectively select system investments to best meet its needs and  priorities.
Policies and procedures do not specify how reselection decisions (i.e.,  annual review decisions) consider investments that are in operations and  maintenance. Without policies and procedures, its ability to make  informed and consistent reselection and termination decisions is limited.
Policies and procedures do not specify how funding decisions are  integrated into the process of selecting an investment. Without considering  its budget constraints and opportunities, the department risks making  investment decisions that do not effectively consider the relative merits of  various projects and systems when funding limitations exist.
Policies and procedures for providing oversight into the department’s  investment management activities do not specify the processes for  decision making during project oversight and do not describe how  corrective actions should be taken when the project deviates or varies  from the project management plan. Without such policies and procedures,  the department risks investing in systems that are duplicative, stovepiped,  nonintegrated, and unnecessarily costly to manage, maintain, and operate.
Table 4 summarizes our findings relative to the department’s execution of  the nine key practices for policies and procedures needed to manage IT  investments at the project level.
According to department officials, they are aware of the absence of  documented policies and procedures in certain areas of project-level  management, and plan to issue new policies and procedures addressing  these areas by March 2008. However, until the department has  documented IT investment management policies and procedures that  include fully defined Stage 2 activities, specify the linkages between the  various related processes, and describe how investments are to be  governed in the operations and maintenance phase, it risks not being able  to carry out investment management activities in a consistent and  disciplined manner. Moreover, the department risks selecting investments  that will not effectively meet its mission needs.
section_title: Department of the Navy Has Not Yet Defined the Policies and Procedures Associated with Effective Portfolio-Level Management
paragraphs: At Stage 3, an organization has defined the critical processes for managing  its investment as a portfolio or set of portfolios. Portfolio management is  a conscious, continuous, and proactive approach to allocating limited  resources among competing initiatives in light of the investments’ relative  benefits. Taking an agencywide perspective enables an organization to  consider its investments comprehensively, so that collectively the  investments optimally address the organization’s missions, strategic goals,  and objectives. Managing IT investments as portfolios also allows an  organization to determine its priorities and make decisions about which  projects to fund based on analyses of the relative organizational value and  risks of all projects, including projects that are proposed, under  development, and in operation. Although investments may initially be  organized into subordinate portfolios—based on, for example, business  lines or life-cycle stages—and managed by subordinate Investment Review  Boards, they should ultimately be aggregated into enterprise-level  portfolios.
According to ITIM, Stage 3 involves four critical processes (1) defining the  portfolio criteria; (2) creating the portfolio; (3) evaluating (i.e., overseeing)  the portfolio; and (4) conducting post-implementation reviews. Within  these critical processes are five key practices that call for policies and  procedures to ensure effective portfolio management. Table 5 summarizes  the purpose of each of these critical processes.
The department has not fully defined the policies and procedures needed  to effectively execute the five portfolio management practices.  Specifically, it does not have policies and procedures for defining the  portfolio criteria or assigning responsibility for managing the portfolio  criteria. In addition, the department does not have policies and procedures  for creating and evaluating the portfolio. Further, it does not have  component-level policies and procedures for conducting post- implementation reviews.
Table 6 summarizes the rating for each critical process required to manage  IT investments as a portfolio and summarizes the evidence that supports  these ratings.
Department officials agreed that portfolio management is primarily a  component responsibility and are aware that they are required to develop  and implement a portfolio management capability. Currently, they are  developing policy and associated procedures that are intended to address  these areas and plan to complete them by March 2008. In the absence of  policies and procedures for managing business system investment  portfolios, the department is at risk of not consistently selecting the mix of  investments that best supports the mission needs and not being able to  ensure that investment-related lessons learned are shared and applied  departmentwide.
section_title: Conclusions
paragraphs: Given the importance of business systems modernization to the  Department of the Navy’s mission, performance, and outcomes, it is vital  for the department to adopt and employ an effective institutional approach  to managing business system investments. However, although department  officials acknowledged shortcomings and the importance of addressing  them, the department has not yet established the management structures  needed to effectively manage its business system investments. The  department is also missing other important elements, such as specific  policies and procedures that are needed for project-level and portfolio- level investment management. In the absence of these essential elements,  the department lacks an institutional capability to ensure that it is  investing in business systems that best support its strategic needs and that  ongoing projects meet cost, schedule, and performance expectations. Until  the department develops this capability, it will be impaired in its ability to  optimize business mission area performance and accountability.
section_title: Recommendations for Executive Action
paragraphs: To strengthen the Department of the Navy’s business system investment  management capability and address the weaknesses discussed in this  report, we recommend that the Secretary of Defense direct the Secretary  of the Navy to ensure that well-defined and disciplined business system  investment management policies and procedures are developed and  issued. At a minimum, this should include instituting project-and portfolio- level policies and procedures that address seven key practices:    Establishing an enterprisewide IT Investment Review Board composed of  senior executives from IT and business units, including assigning the  investment board responsibility, authority, and accountability for  programs throughout the investment life cycle.
Documenting an investment management process that includes how it is  coordinated with JCIDS, PPBE, DAS, and the precertification process.
Ensuring that systems in operations and maintenance are aligned with  ongoing and future business needs.
Selecting new investments, including specifying how cost, schedule, and  benefit data are to be used in making decisions and specifying the criteria  and steps for prioritizing and selecting these investments.
Documenting an annual review process that includes the reselection of  ongoing IT investments.
Integrating funding with the process of selecting an investment, including  specifying how department officials are using funding information in  carrying out decisions.
Overseeing IT projects and systems, including specifying the processes for  the investment boards’ operations and decision making during project  oversight.
These well-defined and disciplined business system investment  management policies and procedures should also include portfolio-level  management policies and procedures that address the following five areas:    Creating and modifying IT portfolio selection criteria for business system  investments.
Defining the roles and responsibilities for managing the development and  modification of the IT portfolio selection criteria.
Analyzing, selecting, and maintaining business system investment  portfolios.
Reviewing, evaluating, and improving the performance of its portfolios by  using project indicators, such as cost, schedule, and risk.
Conducting post-implementation reviews for all investment tiers and  specifying how conclusions, lessons learned, and recommended  management actions are to be shared with executives and others.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, signed by the Deputy Under  Secretary of Defense (Business Transformation) and reprinted in appendix  II, DOD partially concurred with our recommendations. It stated that the  Department of the Navy has drafted Instruction 8115.02, Information  Technology Portfolio Management Implementation, which when finalized,  will address our recommendations. According to DOD, the instruction is  scheduled to be signed in March 2008. DOD added that it would provide  assistance, where appropriate, to the Navy to ensure alignment with  enterprise-level portfolio management policies and procedures as they are  matured. However, DOD also stated that, based on this pending document  from the Department of the Navy, it is the department’s position that a  Secretary of Defense directive on the matter will not be required. Our  recommendations did not state that DOD should develop a directive;  rather, we emphasized the need for the Department of the Navy to develop  policies and procedures.
We are sending copies of this report to interested congressional  committees; the Director, Office of Management and Budget; the Secretary  of Defense; the Deputy Secretary of Defense; the Secretary of the Navy;  the Department of the Navy Chief Information Officer; the Commandant of  Marine Corps; and the Under Secretary of Defense for Acquisition,  Technology, and Logistics. Copies of this report will be made available to  other interested parties on request. This report will also be made available  at no charge on our Web site at http://www.gao.gov.
Should you or your staffs have any questions on matters discussed in this  report, please contact me at (202) 512-6304 or melvinv@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix III.
section_title: Appendix I: Objective, Scope, and Methodology
paragraphs: Our objective was to determine whether the investment management  approach of the Department of the Navy (a major Department of Defense  (DOD) component) is consistent with leading investment management  best practices. Our analysis was based on the best practices contained in  GAO’s Information Technology Investment Management (ITIM)  framework and the framework’s associated evaluation methodology, and  focused on the department’s establishment of policies and procedures for  business system investments needed to assist organizations in complying  with the Clinger-Cohen Act of 1996 (Stages 2 and 3).
To address our objective, we asked the department to complete a self- assessment of its investment management process and provide the  supporting documentation. We then reviewed the results of the  department’s self-assessment of Stages 2 and 3 organizational commitment  practices—those practices related to structures, policies, and  procedures—and compared them against our ITIM framework. We  focused on Stages 2 and 3 because these stages represent the processes  needed to meet the standards of the Clinger-Cohen Act, and they establish  the foundation for effective acquisition management. We also validated  and updated the results of the self-assessment through document reviews  and interviews with officials, such as the Director of the Investment  Management Team and other staff in the department Chief Information  Officer’s office. In doing so, we reviewed written policies, procedures, and  guidance and other documentation providing evidence of executed  practices, including the Department of the Navy’s Business Information  Technology System Precertification Workflow Guidance, Secretary of  Navy Instruction 5000.2C, and the Budget Guidance Manual.
We compared the evidence collected from our document reviews and  interviews with the key practices in ITIM. We rated the key practices as  “executed” on the basis of whether the agency demonstrated (by providing  evidence of performance) that it had met all of the criteria of the key  practice. A key practice was rated as “not executed” when we did not find  sufficient evidence of all elements of a practice being fully performed or  when we determined that there were significant weaknesses in the  department’s execution of the key practice. In addition, we provided the  agency the opportunity to produce evidence for the key practices rated as  “not executed.”
We conducted our work at Department of the Navy offices in Arlington,  Virginia, from February 2007 through September 2007 in accordance with  generally accepted government auditing standards.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact person named above, key contributors to this  report were Tonia Johnson, Assistant Director; Jacqueline Bauer; Elena  Epps; Nancy Glover; and Jeanne Sung.